A probability forecast contains more information than a prediction that an event will happen. Saying 20% and saying 80% should not be treated as equivalent confidence statements. NOAA’s technical literature describes the Brier score as a measure of squared probability error for a binary event, giving a reproducible way to assess the stated probability after its outcome is known. NOAA/NCEP technical note: Probabilistic forecast verification and Brier score
Follow the evidence
Trace how the event could reach markets, then inspect a competing explanation.
Compare explanations
Switch lenses to see what each account explains—and what remains uncertain.
WFN worked example: suppose ten hypothetical forecasts all assign probability 0.20 to the same clearly defined type of event. Two events occur and eight do not. The mean score is [2 × (0.20 − 1)² + 8 × (0.20 − 0)²] ÷ 10 = 0.16. Every input is invented for this arithmetic example. The observed event frequency in that tiny example is 20%, matching the stated probability. That coincidence is not statistical proof that a forecasting system will remain calibrated. A larger evaluation must examine uncertainty, the chosen observations and performance on data that were not selected to make the method look successful. Confidence affects the error even when the event occurs. With outcome one, a hypothetical probability of 0.80 produces squared error 0.04, while probability 0.51 produces 0.2401. Both forecasts are above one-half, yet their errors differ. Reducing the record to a win-or-loss count discards the probability information that the score evaluates.
WFN worked example: suppose ten hypothetical forecasts all assign probability 0.20 to the same clearly defined type of event. Two events occur and eight do not. The mean score is [2 × (0.20 − 1)² + 8 × (0.20 − 0)²] ÷ 10 = 0.16. Every input is invented for this arithmetic example. The observed event frequency in that tiny example is 20%, matching the stated probability. That coincidence is not statistical proof that a forecasting system will remain calibrated. A larger evaluation must examine uncertainty, the chosen observations and performance on data that were not selected to make the method look successful. Confidence affects the error even when the event occurs. With outcome one, a hypothetical probability of 0.80 produces squared error 0.04, while probability 0.51 produces 0.2401. Both forecasts are above one-half, yet their errors differ. Reducing the record to a win-or-loss count discards the probability information that the score evaluates.
A well-defined forecast still needs comparison with the actual contract price, payout and costs before any profitability calculation. As a separate invented example, a 60% event probability and a 100-unit payoff imply a 60-unit expected payout. Paying 70 units would leave an expected 10-unit shortfall before additional costs, even under the assumed probability. Keep the probability timestamp, event definition, outcome source and scoring convention together. If evaluating a tradable contract, add its execution and settlement terms separately. This framework helps assess predictions without presenting prices as pure probabilities or forecast accuracy as a guarantee of profit. The article reports no live odds, tested trading system or achievable return.
The probability and outcome must be defined first
For the binary form used here, the score averages (p − y) squared across observations, with p the forecast probability and y equal to one when the defined event occurs and zero otherwise. A lower score indicates a smaller mean squared probability error under that convention. NOAA’s published methodology supplies the statistical reference, not a financial-market return estimate. NOAA/NCEP technical note: Probabilistic forecast verification and Brier score
Calibration asks whether events assigned comparable forecast probabilities occur with corresponding frequencies across a suitable sample. It is different from checking whether a single prediction turned out correctly. A sample, event definition and evaluation method are needed before claiming persistent forecast quality; one winning contract cannot establish that claim.
Work through the score without inventing market odds
WFN worked example: suppose ten hypothetical forecasts all assign probability 0.20 to the same clearly defined type of event. Two events occur and eight do not. The mean score is [2 × (0.20 − 1)² + 8 × (0.20 − 0)²] ÷ 10 = 0.16. Every input is invented for this arithmetic example.
The observed event frequency in that tiny example is 20%, matching the stated probability. That coincidence is not statistical proof that a forecasting system will remain calibrated. A larger evaluation must examine uncertainty, the chosen observations and performance on data that were not selected to make the method look successful.
Confidence affects the error even when the event occurs. With outcome one, a hypothetical probability of 0.80 produces squared error 0.04, while probability 0.51 produces 0.2401. Both forecasts are above one-half, yet their errors differ. Reducing the record to a win-or-loss count discards the probability information that the score evaluates.
Forecast assessment is separate from contract economics
A well-defined forecast still needs comparison with the actual contract price, payout and costs before any profitability calculation. As a separate invented example, a 60% event probability and a 100-unit payoff imply a 60-unit expected payout. Paying 70 units would leave an expected 10-unit shortfall before additional costs, even under the assumed probability.
Keep the probability timestamp, event definition, outcome source and scoring convention together. If evaluating a tradable contract, add its execution and settlement terms separately. This framework helps assess predictions without presenting prices as pure probabilities or forecast accuracy as a guarantee of profit. The article reports no live odds, tested trading system or achievable return.