Training Llms To Predict World Events

Mantic, a team that builds AI forecasting systems, reports in a guest post on the Thinking Machines Lab site that reinforcement learning on forecasting questions improved the open-weight model gpt-oss-120b. Thinking Machines supplied the Tinker training API they used. The team trained the model on about 10,000 questions of the form "Will an event occur before a date?", rewarding it for putting more probability on the real outcome. Their forecasting setup first uses research agents to collect background context, then has the model output a probability.

On unseen questions from the Q2 2025 Metaculus AI Benchmark Tournament, the post reports the model's mean score rising from 38.6 to 45.8 baseline points per question (a log-score scale where 0 matches a uniform 50% guess), moving from below the frontier models it compared to marginally above them. A version trained without the research phase and mixture-model tools gained about 3 points instead of 7, which the authors read as support for combining gathered context with task-specific training. This is a retrospective evaluation on a past tournament by the team that built the system, not a prospective or trading result.