How to Test an AI Prediction Market Model
Test a prediction model on forecasts recorded before outcomes were known. Measure probability quality separately from simulated trading results, and include the prices, fees and fill assumptions available at the time.
A persuasive explanation is not a performance record. Neither is a screenshot of three winning picks. To assess an AI-assisted process, you need forecasts saved before the event, a clear scoring rule, and a way to separate prediction quality from execution assumptions.
This guide proposes a paper research protocol. It does not present Sports Index backtest results or claim that an AI model beats
Kalshi. Its worked numbers are invented to explain the method. Use the main AI workflow to create the research notes that enter the log.
Decide what you are testing
| Question | What you measure | What it cannot prove alone |
|---|---|---|
| Does the assistant read sources accurately? | Claim-level errors against a checked evidence packet | That its probabilities are useful |
| Are the probabilities informative? | Calibration, scoring rules and baseline comparisons | That available prices permit profitable trades |
| Does a trading rule work in simulation? | Net results under stated price, fee and fill assumptions | That real orders would fill the same way |
| Does the process survive new conditions? | Later, untouched observations under the same protocol | That future performance is guaranteed |
Choose the question first. Otherwise you can drift from a failed forecasting test into a story about the few profitable trades, or from an unrealistic trading simulation into a claim about forecasting skill. Each claim needs its own evidence.
Write a protocol before you collect results
Define the eligible markets, sport, time period, data cutoff, forecast frequency and model or prompt version. Choose a baseline such as a fixed probability or a contemporaneous market-derived probability, and explain how it is constructed. A midpoint baseline is a forecasting reference, not a claim that you could trade at the midpoint.
Set the decision rule before viewing outcomes. If you will simulate entries only when the model exceeds a price threshold after costs, record that rule and retain the forecasts that did not qualify. Otherwise you cannot tell whether later selection favored the winners.
Use a later time period as an untouched evaluation set. Develop the process on earlier data, freeze it, then evaluate forward. If you change the prompt, feature definitions or thresholds after seeing a result, start a new version and report the change. Do not splice only the best weeks from several versions into one track record.
Keep a forecast ledger that can be audited
| Ledger field | Purpose |
|---|---|
| Forecast ID and creation time | Proves which forecast existed before the result |
| Market ticker, side and rule snapshot | Defines exactly what was forecast |
| Information cutoff and source ledger | Detects later information leaking into the forecast |
| Model, prompt and calculation versions | Makes revisions visible |
| Probability and its basis | Separates an estimate from a verbal confidence score |
| Contemporaneous price observations | Supports the market comparison |
| Decision and reason for passing | Preserves the full eligible sample |
| Actual outcome and settlement value | Separates sports result from contract payoff |
| Simulated fills, fees and exclusions | Explains how the paper return was calculated |
Retain original forecasts. If an availability report changes before the event, add a second timestamped forecast rather than replacing the first. Decide whether the test scores the earliest forecast, the latest pre-cutoff forecast, or all forecasts under a specified rule. Multiple forecasts of the same event are not independent events.
Measure probabilities, not just correct picks
Calibration asks whether events assigned a given probability happen at roughly that rate over an adequate sample. A group forecast near 70% should not behave like a group with only 40% positive outcomes indefinitely. The scikit-learn calibration documentation explains reliability diagrams and why a model’s reported probabilities can require evaluation. scikit-learn: Probability calibration.
For binary outcomes, the Brier score is the average of (q − y)², where q is the predicted probability and y is 1 for Yes and 0 for No. Lower is better. It evaluates probabilistic error, not just whether the most likely side won. Do not use a fractional settlement payout as the binary label without explicitly changing the target and scoring method. scikit-learn: Brier score loss.
| Hypothetical event | Forecast q | Observed y | Squared error |
|---|---|---|---|
| A | 0.60 | 1 | 0.16 |
| B | 0.70 | 0 | 0.49 |
| C | 0.40 | 0 | 0.16 |
| D | 0.80 | 1 | 0.04 |
The mean is (0.16 + 0.49 + 0.16 + 0.04)/4 = 0.2125. A 50% forecast on all four events scores 0.25. The model’s score is lower in this toy example, but four events are far too little evidence for a claim of durable skill. Compare with the same-event market baseline too, using prices actually observed at the selected cutoff.
The model called three of four outcomes correctly if you classify probabilities above 50% as Yes. That 75% hit rate hides how wrong the 70% forecast was on event B. Probability scoring makes that error visible. Calibration plots, scores and sample counts complement one another; no single number answers every question.
Keep paper profit separate from forecasting quality
A well-calibrated model can still lack an advantage over the price available to buy. A poorly calibrated model can temporarily make money through luck. Evaluate both the forecast and a fully specified paper trading rule rather than treating one as proof of the other.
For a hypothetical cost example, suppose a simulated record reports $12 of gross profit across 20 completed positions and $15 of combined entry and exit fees. Net profit is −$3 before any other costs. Reporting only the gross result would reverse the conclusion. The fees worksheet explains which cash flows belong in the calculation.
A paper fill at a historical last trade does not establish that the quantity was available to you. You may lack the order-book depth, queue position or latency record needed to simulate execution. State that limit clearly. If execution cannot be reconstructed, describe the result as a price-based research simulation and show how worse entry prices change it.
Check for five common sources of false success
011. Information from the future
Final injury status, revised statistics and completed-game articles can leak into a historical prompt. A language model may also know an event’s outcome from training or retrieved pages. Supplying an old cutoff in the prompt does not prove the model has forgotten later information. Prefer forward paper collection; label retrospective tests as vulnerable to outcome knowledge.
022. Selective record keeping
Save every eligible forecast and every exclusion with its reason. Dropping missing data only when it produces a loss or showcasing one sport after testing many can make ordinary variation look like a discovery.
033. Unrealistic fills
Use the correct side and price type, account for available quantity where possible, and disclose what you cannot reconstruct. Refresh the price-data checklist before treating a historical series as an executable record.
044. Repeated tuning on the test period
Trying many prompt variants and retaining the one with the highest reported return uses the evaluation data to choose the model. Keep a separate later test or disclose the selection process. A strong-looking result after extensive tuning is a hypothesis for the next test.
055. Treating correlated outcomes as independent evidence
Several positions on one game share information and risks. Report both forecast count and unique event count. Group related observations when estimating uncertainty so that 20 markets on one game do not masquerade as 20 independent games.
Copy this evaluation prompt
Audit this paper prediction-market study. The protocol, forecast ledger, raw source times, outcomes and calculations are attached. Check information cutoff, missing forecasts, rule changes, model revisions, selection bias, correlated events, baseline comparability, price types, fills and fees. Recompute scoring and net results where data permits. Separate forecasting findings from execution assumptions. List unsupported claims and what additional evidence would be needed. Do not infer profitability from a short winning streak.
A good audit ends with what the evidence supports and what remains untested. For example: “Probabilities scored better than the stated baseline in this period; execution was not modeled from order-book data.” That is more informative than a single badge claiming the strategy works.
Common questions
01How many predictions are enough?
There is no universal count. The precision required, event dependence, probability distribution and size of the claimed effect all matter. Report uncertainty and sample composition. If the dataset is too small to distinguish your model from the baseline, say so.
02Should I use the exchange demo as proof of performance?
A practice environment can help test a workflow, but its activity does not establish that equivalent live fills would have occurred. Judge the record using its actual data, environment and execution assumptions. Do not relabel practice results as real trading results.
03What should the final report contain?
Include the protocol, full date range, eligible and excluded counts, unique event count, model versions, probability scores, baseline results, net paper returns, cost assumptions and known data limits. Link the settlement-rule record and dated evidence. Publish a limitation next to the result it qualifies, rather than hiding all limitations at the end.
Sources and freshness
Reviewed October 27, 2026. This is educational research, not a trade recommendation. All numerical trade examples are hypothetical. Recheck the exact contract, source dates, prices and fees before relying on an analysis. No model performance is claimed.
Official sources: scikit-learn: Probability calibration; scikit-learn: Brier score loss; Kalshi fee schedule, effective July 7, 2026; Kalshi API: Get Market.