Technical White Paper #2

Predicting vs. Reality

How EchoTest Holds Its Predictions Accountable — the validation protocol.

Executive summary

Any platform that predicts the future invites one fair, unavoidable question: prove it. The honest answer separates a serious tool from theater — and the honest answer is a protocol, not a slogan.

This paper documents exactly how EchoTest measures its own accuracy. It is deliberately published before we quote a headline number, because the protocol is what makes any future number trustworthy. A “92% accurate” claim with no described methodology is marketing; a measured result from a pre-registered, held-out protocol is evidence. The core discipline:

• Lock the prediction before the launch. A timestamped, immutable prediction is recorded before the real campaign runs.
• Score on outcomes the model never saw. Accuracy is measured against real results the simulation had no hand in producing.
• Publish the misses. A validation report that only shows wins isn't validation.

We do not yet have a held-out sample large enough to publish a stable accuracy figure. This paper tells you precisely what we measure and how, so that when the numbers come, you can judge them.

1. The accountability problem

Most “AI prediction” claims fail a basic test of evidence. They quote an accuracy percentage with:

• no description of what was measured,
• no separation between data the model learned from and data it was tested on,
• no disclosure of failures,
• and no way for an outsider to reproduce the result.

That's not a measurement; it's an assertion. The purpose of this protocol is to make EchoTest's accuracy a measured, reproducible quantity — one we are accountable to, and one you can interrogate.

2. The core idea: held-out validation

The only honest way to know whether a simulation predicts reality is to predict something it cannot already know, then wait and compare. A model graded on data it has already seen will always look good — it can memorize rather than generalize. So EchoTest's validation is held-out: the simulation makes a call on a launch whose real outcome does not yet exist, and is later scored against that outcome. There is no opportunity to fit the answer to the test, because at prediction time, the answer hasn't happened.

3. The protocol in detail

3.1 Lock the prediction

Before a customer's real campaign goes live, the simulation output is recorded as an immutable, timestamped prediction: predicted conversion rate, sentiment distribution, and per-segment breakdown, with their confidence bands. Once locked, it cannot be edited. This timestamp-before-launch rule is the backbone of the whole protocol — it guarantees the prediction wasn't shaped by hindsight.

3.2 Define the ground truth

For each prediction we specify, in advance, the real-world metric that will settle it — for example, the actual conversion rate over the campaign's first complete cycle. Defining the target before the outcome exists prevents the subtle cheat of choosing, after the fact, whichever metric happens to look favorable.

3.3 The waiting period

Real outcomes take time. The prediction sits locked until the campaign has run long enough for a stable, real result to exist. We would rather wait for a trustworthy number than score against noise.

3.4 Scoring

When the real outcome lands, the comparison engine computes error along several axes:

That last row matters as much as the headline: a well-calibrated model isn't one that's always right, it's one whose confidence bands tell the truth — when it says p10–p90, real outcomes should fall in that range about 80% of the time.

4. Guardrails against fooling ourselves

The easiest person to fool with a validation study is the person who built the thing. These rules make that hard:

• No peeking. The model does not see the real outcome before its prediction is locked.
• No tuning on the test set. Held-out launches used for scoring are never used to calibrate the model.
• Pre-defined metrics. The scoring metric for each prediction is fixed before the outcome exists.
• Failures are in-scope. Every locked prediction is scored and counted, including the embarrassing ones.

5. How calibration interacts with validation

EchoTest improves per customer through a feedback loop. Doesn't calibration contaminate the accuracy measurement? We handle it by reporting two numbers, not one:

• Cold accuracy — how well the model predicts a market it has never calibrated on (the honest baseline).
• Calibrated accuracy — how well it predicts after learning from that market's real outcomes.

Reporting only the second would flatter the system. Reporting both shows the true starting point andthe value of the feedback loop.

6. What we will publish

As the held-out sample grows, this paper becomes a living scorecard. We intend to publish:

• the number of held-out campaigns scored,
• cold vs. calibrated accuracy across the standard metrics,
• the distribution of errors — not just the average, but how often we miss badly,
• a candid section on the model's systematic blind spots as they emerge.

A small, honest scorecard that grows is worth more than a large, unverifiable claim.

7. Honest status (as of June 2026)

Appendix — Metric definitions

• Mean absolute error (sentiment): the average absolute difference between predicted and actual sentiment shares across categories.
• Relative error (conversion): the absolute difference between predicted and actual conversion rate, divided by the actual rate.
• Calibration coverage: the fraction of real outcomes that fall within the predicted p10–p90 band (~80% when well-calibrated).
• Cold vs. calibrated: “cold” = no prior feedback from that market applied; “calibrated” = the customer's past outcomes incorporated.

This paper describes a measurement protocol, not a finished result. It will be updated with held-out validation data as that data accumulates.