Dataset Note: The 500-Store Panel

Composition, selection criteria, refresh cadence, and run conduct for the benchmark's store panel.

Composition

The panel consists of 500 Shopify-hosted stores drawn from six categories: apparel (120 stores), food and beverage (85), wellness (80), home goods (75), footwear (70), and accessories (70). Category allocations reflect approximate GMV distribution in the Shopify ecosystem as of Q2 2026.

Selection

Stores were selected to be representative, not exceptional. We excluded stores that had opted into Shopify's headless architecture (to ensure a testable baseline) and stores with fewer than 50 SKUs active at time of selection. No store was selected on the basis of known agent-readiness properties. Selection was blind to outcome.

Cadence

The full panel is re-run quarterly. Individual stores may be re-run between cycles if they report a significant structural change and request a re-score. Ad hoc re-runs are noted in the scorecard with a timestamp and a reason code.

Conduct

All benchmark runs use production store URLs. We do not use test modes, staging environments, or discount codes. Agents are given only the information a real customer would have access to. No store credentials or backend access are used. Purchase attempts use test payment credentials on stores that support Shopify's test mode; we do not complete real financial transactions.

Access

The full panel store list is not published to prevent gaming. Stores that have been scored may request their own detailed trace data. Aggregate statistics, category-level findings, and anonymized failure examples are published with each benchmark report.