← Paper Digest

Counterfactual World Models: JEPAs That Reason About "What If"

Randall Balestriero, Yann LeCun, Kerda Varaku

November 2026

TL;DR The authors prove that a world model trained only on factual transitions cannot say what another action would have done under the same hidden disturbance. They fix this by adding the observed prediction error to the model's prediction for the other action, which raises counterfactual R² from 38-46% to 86-96% on four simulators with the same frozen JEPA world models and no retraining.

What they do

A world model predicts where an action leads on average over hidden disturbances, such as a current pushing a boat. Once an outcome is observed, the question becomes where another action would have led under the same disturbance. The authors prove a world model trained on factual transitions cannot answer this alone, and show that for many physical systems a simple correction can.

How it works

  • Residual reuse. The counterfactual outcome is the observed outcome plus the world model’s predicted difference between the alternative and the actual action: two predictor calls, no counterfactual data, extra parameters or retraining.
  • Why it holds. When action and disturbance enter as separate forces, their interaction is second-order in the transition duration h, so the error is O(h²); a matching minimax lower bound shows this rate is the best possible from unpaired factual data.
  • Setup. JEPA predictors on seven frozen image encoders (DINOv2, MAE and others), trained on factual transitions in rendered simulators such as HandReach, FetchReach and the BlueROV2 underwater vehicle.

Results

  • On HandReach, Ball-in-Cup, FetchReach and PassWater, counterfactual R² rises from 38.1-45.6% for the same predictors to 85.6-95.7%, above the 49.1-51.1% of an oracle that knows the simulator state but not the outcome. Correcting the image features directly, without state labels, gives 73.4-94.1%.
  • BlueROV2 needs more data: 90.1-93.5% with 16x the factual episodes.
  • Hindsight planning succeeds in 35-69% of events, against 9-25% for the trained predictor and 14-23% for the oracle planner.
  • The predicted failures appear: with Push-T contacts and full gain faults, the correction does worse than ignoring the outcome.

Limitations

All environments are simulated, with controlled actuator faults. The rule needs coordinates in which action and disturbance add (raw pixels and cos(pitch) do not), fails with contacts and command-scaling faults, and covers one transition at a time. Whether it is safe on a system can only be checked with paired outcomes, not factual logs.

Figures

From Figure 4: The seven rendered simulators, each with a hidden disturbance: Racetrack, HandReach, Ball-in-Cup, Fetc...
From Figure 4: The seven rendered simulators, each with a hidden disturbance: Racetrack, HandReach, Ball-in-Cup, FetchReach, Push-T, the BlueROV2 underwater vehicle and liquid transport (PassWater).
From Figure 3: The action-disturbance interaction ratio against transition duration, from simulator dynamics only. Le...
From Figure 3: The action-disturbance interaction ratio against transition duration, from simulator dynamics only. Left: a damped pendulum with an additive force, a gain fault, saturation or a damping fault. Right: native HandReach servos, where the offset ratio falls as the servos settle while gain faults stay near 2.
From Figure 5: Physical counterfactual R² per environment and encoder. Bars: residual correction after the readout. B...
From Figure 5: Physical counterfactual R² per environment and encoder. Bars: residual correction after the readout. Black ticks: the same predictor without the observed outcome. Dashed lines: an oracle predictor with the full simulator state but no outcome. Push-T is diagnostic only.
From Figure 10: Hindsight-planning success, scored by replaying the observed event with the chosen command, for the t...
From Figure 10: Hindsight-planning success, scored by replaying the observed event with the chosen command, for the trained predictor, an oracle planner that knows the simulator but not the outcome, and the counterfactual world model.
Abstract

World models operate in a world they cannot fully observe. When a hidden disturbance, such as a current pushing a boat, changes the outcome of an action, a counterfactual question follows: what would have happened under a different action, subject to the same disturbance? Its answer separates bad decisions from bad luck. We prove that world models trained on factual transitions cannot answer it on their own, whatever their data or capacity. For many physical systems, the observed outcome and a simple property of the dynamics suffice: natura non facit saltus, nature makes no leaps. When actions and disturbances act as separate forces, their interaction is second-order in time, so the counterfactual outcome is, to first order, the observed outcome plus the difference between the world model's predictions under the alternative and factual actions. This yields a counterfactual world model with no counterfactual data, extra parameters, or retraining. For transitions of duration $h$, we prove an $O(h^2)$ bound on the counterfactual error relative to the disturbance variance, establish a matching minimax lower bound from unpaired factual data, and characterize failures at discontinuities, coordinate folds, and disturbances that scale the command. We validate these predictions from pixels with JEPA world models on frozen DINOv2 and masked autoencoder (MAE) encoders. On four rendered simulators, our approach improves counterfactual $R^2$ from $38$--$46\%$ to $86$--$96\%$ with the same world models; on a fifth, an underwater vehicle, it reaches $90$--$94\%$ with sixteen times more factual data. Four further frozen encoders, DINOv3, SigLIP~2, CLIP and EVA-02, give $81$--$95\%$ on the same simulators. In hindsight planning, it finds the action that would have reached the goal in $35$--$69\%$ of events, against $14$--$23\%$ for an oracle planner that knows the simulator but not the outcome.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.