Methodology

How the calls are made.

The forecasting method behind Vatex’s calls — and how we hold it to account. Plain version: most forecasters predict what’s already knowable. Vatex goes after stochastic, high-aleatoric events (when a goal is scored, when a market turns), which can only honestly be forecast as a distribution, never a single guess.

The precise claim: well-calibrated, sharp predictive distributions over how and when random events unfold — scored by proper scoring rules, against named baselines, on pre-committed forecasts.

The method

  1. 01Compress broad real-world data down to the signal behind a stochastic process.
  2. 02Model how that randomness is distributed over time (a fitted statistical model).
  3. 03Distill the model into abstract storylines — the ways a random sequence can plausibly unfold.
  4. 04Simulate & score LLM agents run the storylines forward into a probability distribution over outcomes, then we score that distribution against what actually happened.

How it’s scored

Probabilistic forecasts are judged by proper scoring rules (a rule is proper when a forecaster maximizes their expected score only by reporting their true predictive distribution): CRPS for continuous “when” timing; log loss / Brier for binned questions (“event in the next N minutes”); ECE for calibration.

Calibration alone isn’t enough — a model can be perfectly calibrated yet useless (always say 50%). The target is sharpness subject to calibration, normalized by a dynamic baseline (an event can be highly uncertain a week out and near-deterministic hours before resolution) and paired with an economic / market-return metric.

Baselines we must beat

  • Poisson goal-timing modelHeuer et al. 2010

    In-match goals are, to a good approximation, a memoryless Poisson process — it pins the scoring rate/distribution, never the individual goal: the textbook bound on “when a goal is scored.”

  • Marked spatio-temporal point processNarayanan et al. 2023, JRSS-C

    A Hawkes-derived model that already forward-simulates probabilistic forecasts of goals/fouls in a time interval — the strongest prior art for the soccer use-case.

  • Market / bookmaker odds

    The standing reference for whether a forecast adds value at all.

  • Frontier-LLM ensembleProphet Arena, 2025

    Frontier LLMs already approach prediction-market accuracy on live forecasting. This is the bar.

The decisive test

The weakest, most-attacked link is the last two steps: does distilling storylines and interpreting them with LLMs beat directly fitting and forward-simulating the point-process model? The required experiment is an ablation — storyline+LLM pipeline vs. the raw statistical model — measured on CRPS and calibration. If the LLM layer doesn’t win, it isn’t the contribution.

Why it’s a real niche

Current AI-forecasting benchmarks (FutureX, ForecastBench, Metaculus AIB, Prophet Arena, Kalshibench) overwhelmingly score binary, fact-based questions. Almost none score distributional “when / how-many” timing of stochastic events — that gap is the defensible niche.

On pre-commitment

Hash-chained, tamper-evident timelines (the receipts) are a delivery mechanism for an honest public record — a complement to standard pre-registration and Brier-scored track records, not a substitute for rigorous scoring. Tamper-evidence is not rigor.

References

  • Halawi, Zhang, Yueh-Han & Steinhardt — Approaching Human-Level Forecasting with Language Models (2024) — arXiv:2402.18563
  • Hsieh, Fu & Chen — Reasoning and Tools for Human-Level Forecasting (RTF) — arXiv:2408.12036
  • Yang et al. — LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena (2025) — arXiv:2510.17638
  • Nel — Kalshibench — arXiv:2512.16030
  • Uncertainty-quantification & calibration survey — arXiv:2503.15850
  • Quantifying aleatoric & epistemic uncertainty with proper scoring rules — arXiv:2404.12215
  • Proper-scoring-rule UQ decomposition — arXiv:2505.22538
  • Heuer, Müller & Rubner — Soccer: Is scoring goals a predictable Poissonian process? (2010) — arXiv:1002.0797
  • Narayanan, Kosmidis & Dellaportas — Marked point processes for football (JRSS-C, 2023) — arXiv:2103.04647

Caveat: frontier benchmark standings are a 2025 snapshot and date quickly; the soccer / Poisson claims rest on single authoritative sources.

← Back to Vatex