Validation & Trust

How accurate is EchoTest? The benchmarks.

Name: EchoTest Simulation Engine Benchmarks
Creator: EchoTest

EchoTest validates its simulation engines by backtesting them against outcomes that are already known — published research norms, convergent measurement checks, and logical invariants the model must obey — rather than waiting months for live results. Cases are pre-registered and frozen before they are run, ground truth is cited, and runs are deterministic so anyone can reproduce the scorecard.

7/7

cases passing

engines validated

Frozen

pre-registered cases

2026-06-09

last updated

Why these numbers are trustworthy

Pre-registered & frozen

Every case is hashed into a lockfile before it is run. Editing a case after seeing results changes its hash and fails CI — you cannot quietly tune a case to pass.

Cited ground truth

Retrospective, convergent, and published-norm cases must carry a source citation, enforced at load time. The numbers come from external research, not from us.

Deterministic & reproducible

Runs fix temperature to 0 with seeded sampling, so the scorecard is reproducible by anyone with the repository.

Rank-first, and we publish the misses

We lead with whether the engine orders and directs outcomes correctly (Spearman, top/bottom choice) over fragile point-precision — and we report failures and not-yet-built coverage, not just passes.

Scorecard

Each case is scored on its own documented criterion. We lead with rank and directional accuracy over point-precision.

Commerce engine · standard

4 of 4 cases pass on their documented criteria

Run 2026-06-09 · gpt-4o-mini (debate) / gpt-4o (synthesis) · 3 replicates · 200 agents, United States

Tier	Case	What it tests	Result
T2	SaaS calibrated take-rate band	The calibrated purchase take-rate lands in the plausible SaaS band (0.8%–6%) — i.e. calibration is applied, not raw stated intent.0.99% within band · perfectly reproducible (CV 0.0 across 3 reps)	Pass
T3	Category take-rate ordering	Population take-rate orders the way real adoption does: everyday consumable > software subscription > big-ticket durable.Spearman 1.0 · food & beverage 4.17% > SaaS 0.99% > real estate 0.03%	Pass
T4	Price monotonicity	Across a decisive price span ($49 / $499 / $4999), conversion falls as price rises.Spearman 1.0 · 6.0% > 2.67% > 0%	Pass
T4	Budget threshold	An over-priced variant must be the worst converter and crater far below the affordable tiers.Worst-converter correctly flagged · $899 craters to 0.8% (worst/best 0.15)	Pass

Penetration engine · standard

Directional invariants confirmed

Run 2026-06-09 · gpt-4o-mini · 1 replicate · 200 agents, United States

Tier	Case	What it tests	Result
T4	Price → conversion	A higher price must not convert better — wave-4 conversion falls across $9 / $99 / $499.Spearman 1.0 · 17.0% > 15.7% > 10.2%	Pass
T4	Free tier lifts signup	A product with a free tier must not sign up worse than the same product without one.Spearman 1.0 · free tier 76% vs no free tier 56% signup	Pass

Growth engine · standard

Projection scales with the market

Run 2026-06-09 · gpt-4o-mini · 1 replicate · 150 agents, United States

Tier	Case	What it tests	Result
T4	Market size → projected subscribers	A larger addressable market must yield more projected subscribers in the recommended growth scenario.Larger market correctly projects more subscribers (bottom-choice confirmed)	Pass

The four validation tiers

T1 · Retrospective

Real products with known historical outcomes, run blind. (Strongest tier — in progress.)

T2 · Convergent validity

Agreement with established measurement methods and category benchmarks.

T3 · Published norm

Output matches a documented real-world distribution.

T4 · Synthetic stress

Logical invariants the engine must obey — the regression guard.

Honest limitations

Strongest tier (T1 retrospective, real products run blind) is still being assembled.
Per-category intent→action magnitudes are not yet fully provenanced; tests use robust orderings, not unsettled point values.
Coverage so far is United States and the standard engine variant.
Van Westendorp price-sensitivity ground truth is unmet (no defensible real-product figures found), so it is not used for numeric calibration.

Frequently asked questions

How accurate is EchoTest?

EchoTest is validated by backtesting its simulation engines against outcomes that are already known — published research norms, convergent measurement checks, and logical invariants — rather than waiting for live results. On the latest run the commerce engine passes all four of its documented cases, including correctly ordering category adoption and price sensitivity.

How are the benchmarks validated?

Each test case is pre-registered and frozen (hashed into a lockfile) before it is run, so cases cannot be tuned after seeing results. Ground-truth cases must cite an external source, and runs are deterministic (temperature 0 with seeded sampling) so the scorecard is reproducible.

Can I reproduce the results?

Yes. Cases are frozen and runs are deterministic, so the same cases produce the same scorecard. We lead with rank and directional accuracy (does the engine order and direct outcomes correctly) over fragile point-precision, and we publish the misses alongside the passes.

What do the benchmark tiers mean?

T1 retrospective backtests real products with known outcomes (strongest, in progress); T2 convergent validity checks agreement with established measurement methods; T3 published-norm checks output against documented real-world distributions; T4 synthetic stress asserts logical invariants the engine must obey.

See it on your own product

Read the full methodology, or test your content, pricing, and growth plan against a synthetic consumer panel before you spend a dollar.

Read the methodology Get started