Skip to content
Sports Index
PRO
Research 6 min read

How to Test an AI Prediction Market Model

Test a prediction model on forecasts recorded before outcomes were known. Measure probability quality separately from simulated trading results, and include the prices, fees and fill assumptions available at the time.

A persuasive explanation is not a performance record. Neither is a screenshot of three winning picks. To assess an AI-assisted process, you need forecasts saved before the event, a clear scoring rule, and a way to separate prediction quality from execution assumptions.

This guide proposes a paper research protocol. It does not present Sports Index backtest results or claim that an AI model beats Kalshi. Its worked numbers are invented to explain the method. Use the main AI workflow to create the research notes that enter the log.

Decide what you are testing

QuestionWhat you measureWhat it cannot prove alone
Does the assistant read sources accurately?Claim-level errors against a checked evidence packetThat its probabilities are useful
Are the probabilities informative?Calibration, scoring rules and baseline comparisonsThat available prices permit profitable trades
Does a trading rule work in simulation?Net results under stated price, fee and fill assumptionsThat real orders would fill the same way
Does the process survive new conditions?Later, untouched observations under the same protocolThat future performance is guaranteed

Choose the question first. Otherwise you can drift from a failed forecasting test into a story about the few profitable trades, or from an unrealistic trading simulation into a claim about forecasting skill. Each claim needs its own evidence.

Write a protocol before you collect results

Define the eligible markets, sport, time period, data cutoff, forecast frequency and model or prompt version. Choose a baseline such as a fixed probability or a contemporaneous market-derived probability, and explain how it is constructed. A midpoint baseline is a forecasting reference, not a claim that you could trade at the midpoint.

Set the decision rule before viewing outcomes. If you will simulate entries only when the model exceeds a price threshold after costs, record that rule and retain the forecasts that did not qualify. Otherwise you cannot tell whether later selection favored the winners.

Use a later time period as an untouched evaluation set. Develop the process on earlier data, freeze it, then evaluate forward. If you change the prompt, feature definitions or thresholds after seeing a result, start a new version and report the change. Do not splice only the best weeks from several versions into one track record.

Keep a forecast ledger that can be audited

Ledger fieldPurpose
Forecast ID and creation timeProves which forecast existed before the result
Market ticker, side and rule snapshotDefines exactly what was forecast
Information cutoff and source ledgerDetects later information leaking into the forecast
Model, prompt and calculation versionsMakes revisions visible
Probability and its basisSeparates an estimate from a verbal confidence score
Contemporaneous price observationsSupports the market comparison
Decision and reason for passingPreserves the full eligible sample
Actual outcome and settlement valueSeparates sports result from contract payoff
Simulated fills, fees and exclusionsExplains how the paper return was calculated

Retain original forecasts. If an availability report changes before the event, add a second timestamped forecast rather than replacing the first. Decide whether the test scores the earliest forecast, the latest pre-cutoff forecast, or all forecasts under a specified rule. Multiple forecasts of the same event are not independent events.

Measure probabilities, not just correct picks

Calibration asks whether events assigned a given probability happen at roughly that rate over an adequate sample. A group forecast near 70% should not behave like a group with only 40% positive outcomes indefinitely. The scikit-learn calibration documentation explains reliability diagrams and why a model’s reported probabilities can require evaluation. scikit-learn: Probability calibration.

For binary outcomes, the Brier score is the average of (q − y)², where q is the predicted probability and y is 1 for Yes and 0 for No. Lower is better. It evaluates probabilistic error, not just whether the most likely side won. Do not use a fractional settlement payout as the binary label without explicitly changing the target and scoring method. scikit-learn: Brier score loss.

Hypothetical eventForecast qObserved ySquared error
A0.6010.16
B0.7000.49
C0.4000.16
D0.8010.04

The mean is (0.16 + 0.49 + 0.16 + 0.04)/4 = 0.2125. A 50% forecast on all four events scores 0.25. The model’s score is lower in this toy example, but four events are far too little evidence for a claim of durable skill. Compare with the same-event market baseline too, using prices actually observed at the selected cutoff.

The model called three of four outcomes correctly if you classify probabilities above 50% as Yes. That 75% hit rate hides how wrong the 70% forecast was on event B. Probability scoring makes that error visible. Calibration plots, scores and sample counts complement one another; no single number answers every question.

Keep paper profit separate from forecasting quality

A well-calibrated model can still lack an advantage over the price available to buy. A poorly calibrated model can temporarily make money through luck. Evaluate both the forecast and a fully specified paper trading rule rather than treating one as proof of the other.

For a hypothetical cost example, suppose a simulated record reports $12 of gross profit across 20 completed positions and $15 of combined entry and exit fees. Net profit is −$3 before any other costs. Reporting only the gross result would reverse the conclusion. The fees worksheet explains which cash flows belong in the calculation.

A paper fill at a historical last trade does not establish that the quantity was available to you. You may lack the order-book depth, queue position or latency record needed to simulate execution. State that limit clearly. If execution cannot be reconstructed, describe the result as a price-based research simulation and show how worse entry prices change it.

Check for five common sources of false success

011. Information from the future

Final injury status, revised statistics and completed-game articles can leak into a historical prompt. A language model may also know an event’s outcome from training or retrieved pages. Supplying an old cutoff in the prompt does not prove the model has forgotten later information. Prefer forward paper collection; label retrospective tests as vulnerable to outcome knowledge.

022. Selective record keeping

Save every eligible forecast and every exclusion with its reason. Dropping missing data only when it produces a loss or showcasing one sport after testing many can make ordinary variation look like a discovery.

033. Unrealistic fills

Use the correct side and price type, account for available quantity where possible, and disclose what you cannot reconstruct. Refresh the price-data checklist before treating a historical series as an executable record.

044. Repeated tuning on the test period

Trying many prompt variants and retaining the one with the highest reported return uses the evaluation data to choose the model. Keep a separate later test or disclose the selection process. A strong-looking result after extensive tuning is a hypothesis for the next test.

055. Treating correlated outcomes as independent evidence

Several positions on one game share information and risks. Report both forecast count and unique event count. Group related observations when estimating uncertainty so that 20 markets on one game do not masquerade as 20 independent games.

Copy this evaluation prompt

Audit this paper prediction-market study. The protocol, forecast ledger, raw source times, outcomes and calculations are attached. Check information cutoff, missing forecasts, rule changes, model revisions, selection bias, correlated events, baseline comparability, price types, fills and fees. Recompute scoring and net results where data permits. Separate forecasting findings from execution assumptions. List unsupported claims and what additional evidence would be needed. Do not infer profitability from a short winning streak.

A good audit ends with what the evidence supports and what remains untested. For example: “Probabilities scored better than the stated baseline in this period; execution was not modeled from order-book data.” That is more informative than a single badge claiming the strategy works.

Common questions

01How many predictions are enough?

There is no universal count. The precision required, event dependence, probability distribution and size of the claimed effect all matter. Report uncertainty and sample composition. If the dataset is too small to distinguish your model from the baseline, say so.

02Should I use the exchange demo as proof of performance?

A practice environment can help test a workflow, but its activity does not establish that equivalent live fills would have occurred. Judge the record using its actual data, environment and execution assumptions. Do not relabel practice results as real trading results.

03What should the final report contain?

Include the protocol, full date range, eligible and excluded counts, unique event count, model versions, probability scores, baseline results, net paper returns, cost assumptions and known data limits. Link the settlement-rule record and dated evidence. Publish a limitation next to the result it qualifies, rather than hiding all limitations at the end.

Sources and freshness

Reviewed October 27, 2026. This is educational research, not a trade recommendation. All numerical trade examples are hypothetical. Recheck the exact contract, source dates, prices and fees before relying on an analysis. No model performance is claimed.

Official sources: scikit-learn: Probability calibration; scikit-learn: Brier score loss; Kalshi fee schedule, effective July 7, 2026; Kalshi API: Get Market.