Where it fails37%
of paying journeys fail a check at reading the product facts.
29% fail at asking the shopper what they need. Building the cart, checking out and paying: under one percent each, on every protocol.
CartTrace-100k · an open dataset from General Machines
We sent an AI agent shopping 101,416 times.
It almost never failed at checkout. It failed earlier: reading the product page, and asking what the shopper meant.
- 1,044stand-in stores
- 25kinds of shop
- 5commerce protocols
- 68.8%ended right
An open dataset from General Machines, explored by Machine Commerce. Nothing was bought on a real store.
CartTrace-100k100 journeys, as they endedEnded in a sale we can confirm66
Lost at the till0.0
Per 100 everyday journeys. Every loss happens before the payment step.
Watch one
Every replay is a real journey from the dataset.
The shopper is a program. It holds some of its needs back and gives them only when asked. The agent has the store's tools and one instruction: buy the right thing.
Category
Replaying a journey
What we found
It fails before it pays.
Journeys are lost between finding a product and choosing it: reading the facts, and asking for what the shopper didn't say. Once the cart is right, checkout and payment go through on every protocol.
Stores54–82%
success, from the 5th to the 95th percentile store.
The stores where the agent does worst have one thing in common: search that misses the first query. A fitting product came up first in 81% of their journeys, against 85% on the best stores. So the agent stops more than it buys wrong.
worst quarter of storesbest quarter
Asking62% vs 12%
success when the shopper held two needs back, with and without the agent asking.
On the same task, asking adds 11.9 points (likely range 10.2 to 13.7) where a need is held back and almost nothing where none is. Yet the agent asks in 63% of journeys where nothing is hidden. The fix is aim, not more questions.
agent did not askagent asked
Giving up7.9%
of doable journeys end with the agent saying nothing fits.
On requests where a substitute would do, 27.4%. When a task really was impossible, it refused correctly 77.4% of the time, so only 9.4% of its refusals were right.
The store explains little1.9%
of the difference in outcomes comes from which store it was.
The task explains most of it, and plain chance much of the rest: of 24,549 tasks run three times or more, 29.3% both succeeded and failed. One test tells you little about a store.
When we broke it on purpose99.3%
of the time it told the shopper what really happened.
Declined cards, failed bank checks, price changes, requests nothing could fill. It paid when it shouldn't have in 6 of 3,572 journeys, and never paid for an impossible task.
The 25 categories
A shirt needs a size. A drill needs a voltage.
A gift needs a guess.
Each label shows how often the agent ended right in that category. Open one to see where it stalls, or watch a journey from it.
How it ran
Stand-in stores, real protocols, real payment steps.
No agent may pay on a real store, and a real store's data can't be published. So one agent shopped stores built to sell like real ones, all the way through checkout, with test tokens.
- 01 · Requests28,985
shopping tasks from 30 shopper personas: gifts, budgets, substitutes, exact sizes.
- 02 · Stores1,044
stand-ins in 25 categories, stocked from open product catalogs.
- 03 · Journeys101,416
from 17 to 22 October 2026, over UCP, ACP, AP2, MPP and WebMCP.
- 04 · Checks85
per journey: 71 settled by code from the store's records, 14 by a model.
- Agent
- Qwen3.8-27B
- Compute
- 971 H100 hours
- Paid orders
- 42,102
- Reference agent
- 82.2% ended right
The data, the paper and the code
Not a score.
A record of how agents shop.
For researchers and agent builders. Download every journey, read how they were run and judged, and re-run any journey yourself to check a number. The links open on launch day.
Packing listCartTrace-100k · v1
| Shopping journeys | 101,416 |
|---|---|
| Reference journeys | 10,132 |
| Stand-in stores | 1,044 |
| Categories | 25 |
| Tasks | 28,985 |
| Paid orders | 42,102 |
| Broken on purpose | 3,976 |
Per journey
- Conversationevery message between shopper and agent
- Tool callsname, arguments, result, errors, timing
- Protocol trafficeach request and reply on the wire
- Cart and checkoutthe state after every change
- Store eventsthe payment as the store saw it
- What the store showedwhich product facts each protocol exposed
- The setupstore, request, shopper, protocol, payment, faults
- The endingright or wrong, cart or order, and why
Stores are pseudonyms; prices and stock are synthetic. Data: CC BY 4.0 (product data keeps its source licence). Replay kit: Apache-2.0.
PriorityNo. CT-100K-01
DownloadThe dataEvery journey, as Parquet tables on Hugging Face
huggingface.co →
First classNo. CT-100K-02
ReadThe paperHow it was run and judged, and every number
Paper →
Handle with careNo. CT-100K-03
RunThe replay kitCode on GitHub to replay and re-judge any journey
github.com →
If you use CartTrace-100k in research, cite it as
@misc{carttrace100k,
title = {CartTrace-100k: How Shopping Agents Fare Across Stores,
Categories and Commerce Protocols},
author = {Bhatnagar, Pratham and Pal, Deepanshu},
year = {2026},
publisher = {General Machines},
url = {https://generalmachines.ai/datasets/carttrace-100k},
note = {Dataset: https://huggingface.co/datasets/generalmachines/CartTrace-100k}
}