Your Prediction Agent Is Learning From Its Own Past Guesses — Before the Answer Even Arrives

A system that improves its forecasting process by comparing its own earlier and later predictions on the same unresolved question — harvesting useful supervision before the outcome is ever known.

~6 min read · Zhongguancun Academy / USTC / Tsinghua · April 21, 2026 · Agentic

TL;DR Most prediction agents only learn once an outcome is revealed — which is too late to fix how they gathered evidence or tracked key factors. Milkyway introduces "internal feedback": by revisiting the same unresolved question at multiple time points, it extracts lessons from the contrast between earlier and later predictions and writes them into a persistent "harness" that guides future runs. On two live forecasting benchmarks, it achieves the best overall scores among compared methods.

There's a fundamental awkwardness at the heart of AI forecasting: the signal you most want — "was I right?" — arrives only after the question is already settled. By then, it's too late to fix the fact that you were tracking the wrong indicators, reading the wrong sources, or collapsing to a confident answer when the evidence still warranted caution. Final outcomes tell you that you were wrong. They don't tell you where in the process things went sideways.

Researchers from Zhongguancun Academy, USTC, and Tsinghua have a different idea. They observe that when you revisit the same unresolved question over multiple weeks or days, something interesting happens: your later predictions — informed by richer evidence — implicitly reveal what your earlier ones missed. That contrast itself is a signal.

They call it internal feedback, and they've built a system called Milkyway around it.

💡 The Core Idea Instead of waiting for the answer to improve your process, compare your earlier self to your later self on the same question. If your Day 4 prediction tracked labor market data that your Day 1 prediction ignored, that omission is diagnosable right now — no outcome required. Write that lesson into a reusable guide, and your next prediction (and future questions like it) will start ahead.

What's a Harness?

Milkyway keeps its base language model entirely frozen. There's no fine-tuning, no gradient updates. Instead, it maintains a future prediction harness: an editable external document that organizes three aspects of the prediction process.

• F (Factor tracking): Which factors and indicators should be monitored for this class of question?
• E (Evidence gathering and interpretation): Which sources, search queries, and verification routines reliably surface useful information? Which signals tend to mislead?
• U (Uncertainty handling): When should the agent commit to a prediction? When should it hold competing interpretations open?

Think of the harness as a living checklist written by a seasoned analyst who has worked through many similar questions. Except here, the analyst is the system's accumulated experience — updated in real time.

How the System Evolves

sequenceDiagram
    participant Q as Unresolved Question
    participant BA as BaseAgent
    participant CN as Checkpoint Note
    participant HE as Harness Editor
    participant H as Harness (F, E, U)

    Q->>BA: Predict at checkpoint τ₁ (early, sparse evidence)
    BA->>CN: Write compact note: evidence, judgment, risks
    Q->>BA: Predict at checkpoint τ₂ (more evidence available)
    BA->>CN: Write checkpoint note τ₂
    CN->>HE: Compare notes: what changed, what was missed?
    HE->>H: Extract internal feedback → update harness
    Note over H: Later predictions on this question use updated harness
    Q-->>HE: Question resolves → retrospective check
    HE->>H: Validate & carry forward to next related question

At each checkpoint, the BaseAgent (a ReAct-style tool-using agent with web search, browser, and code execution) makes a prediction under the current harness and records a compact checkpoint note — not a raw log dump, but a structured summary covering the question state, key evidence surfaced, the main line of judgment, and still-unresolved risks.

The Harness Editor then compares notes across checkpoints. The contrast — what changed, what appeared only later, where confidence was misplaced — becomes structured diagnostic guidance. That guidance gets written back into the harness for immediate use on the current question's remaining checkpoints, and eventually carried forward to future questions after a post-resolution retrospective check validates the provisional revisions.

This two-timescale design matters: updates within a question are provisional and fast; updates across questions only happen after the outcome confirms (or corrects) them.

The Numbers

On both benchmarks, Milkyway achieves the highest overall score among the compared methods.

The gain over the direct GPT-5.4 baseline is 16.83 points on FutureX and 15.74 points on FutureWorld. The gain over the next strongest method in each table is 3.40 and 5.99 points, respectively.

The rolling daily evaluation on FutureWorld is particularly telling: Milkyway ranks first on all five consecutive daily windows, with margins of 3.52–8.89 points over the strongest competing method on each individual day. Crucially, it already leads on Day 1 — before any within-window retrospective update is available — which means the advantage isn't purely from accumulating updates inside the evaluation window. The harness itself, evolved from earlier questions, is doing real work from the first prediction.

Isolating the Harness Effect

The paper includes a clean ablation on a fixed cohort of FutureWorld questions, re-running predictions daily from 5 days before resolution to 1 day before resolution. Three settings are compared:

The agent scaffold alone (no harness read/write) adds 3–6 points over the base model. The full harness adds a further 5–11 points on top of that. And the harness advantage grows as the question approaches resolution — consistent with the core mechanism: the closer you are to an answer, the more informative the temporal contrasts become, and the more the harness has been sharpened by earlier internal feedback on that question.

Why It Matters

Most self-improving agent frameworks — Reflexion, ExpeL, Voyager, MemEvolve — accumulate reusable lessons across completed tasks. That's valuable, but it leaves a gap in settings where the relevant supervision is delayed. Future prediction is the clearest example: a question about whether the Fed will cut rates at a March meeting might span weeks of evolving evidence, and the only clean signal arrives after the meeting ends.

Milkyway's insight is that you don't have to wait. The unresolved question's own evolving trajectory — visited repeatedly before resolution — already contains diagnostic information about process failures. Harvesting that signal and writing it into a structured, reusable harness turns repeated uncertainty into structured learning.

The approach is also practically grounded: it works without retraining the base model, keeps question-specific facts out of the persistent harness (avoiding drift), and validates provisional revisions against realized outcomes before carrying anything forward. The harness doesn't accumulate noise; it accumulates tested procedure.

⚠️ Watch out for

• The method requires questions to be revisited at multiple checkpoints before resolution — settings with only a single prediction opportunity get no internal feedback benefit.
• The textual harness may accumulate redundancy or drift over very long question horizons; the paper acknowledges this and leaves mitigation to future work.
• The evaluation covers two benchmarks and a limited window of live dates — the authors are appropriately cautious about generalizing temporal variance claims.

Go Deeper

• FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction — the primary benchmark used for evaluation; introduces the weekly-cadence live forecasting setting.
• Outcome-based Reinforcement Learning to Predict the Future — a complementary direction: learning from realized outcomes after resolution, rather than from pre-resolution internal feedback.
• Reflexion: Language Agents with Verbal Reinforcement Learning — foundational prior work on converting agent experience into reusable natural-language lessons; Milkyway adapts this paradigm to the delayed-supervision setting.

Source: The World Leaks the Future: Harness Evolution for Future Prediction Agents Authors: Chuyang Wei, Maohang Gao, Zhixin Han, Kefei Chen, Yu Zhuang, Haoxiang Guan, Yanzhi Zhang, Yilin Cheng, Jiyan He, Huanhuan Chen, Jian Li, Yu Shi, Yitong Duan, Shuxin Zheng Published: 2026-04-21 PDF: https://arxiv.org/pdf/2604.15719