After 600 experiments, here's what I discovered about JEPA world models

Haddy Alchaer's months of experiments on LeWM, a JEPA world model released this year by a team that includes Yann LeCun, point to the planner rather than the model's predictions as what holds it back on long tasks. A JEPA (joint-embedding predictive architecture) world model does not generate future video frames. An encoder turns each camera frame into a compact vector describing the scene, a predictor estimates the next vector given an action, and a planner tries hundreds of action sequences inside the predictor and keeps the one whose predicted end state lands closest to the goal's vector. LeWM's main test is Push-T, a simulation in which a robot's round pusher must slide a T-shaped block onto a target shown in a goal image (Alchaer's write-up; LeWM paper).

The central experiment separates prediction error from planning. Used as published, LeWM reaches the goal in 22% of tasks where it is 75 steps away, but 89.7% of the time when it is 25 steps away. If small prediction errors were compounding over the long plan, swapping in the simulator's exact physics should fix that; on the same 250 tasks it raised success only to 42%. Keeping LeWM's own predictor and giving it a subgoal every 25 steps, taken from an expert demonstration, raised success to 80%, and with those subgoals exact physics did no better than the learned predictor. Alchaer's reading is that LeWM plans accurately over about 25 steps, so chaining searches of that length matters more than better predictions.

Expert subgoals are not available to a deployed robot, so Alchaer replaced them with a graph built from LeWM's own embeddings of about 460,000 demonstration states, searching it for a path of subgoals. Charging a cost for every jump between different demonstrations stopped the search from stitching together paths the block could not follow, and raised success from 62% to 85%, at least as high as the 80% reached with expert subgoals. Alchaer then shows this benchmark flatters the method: Push-T demonstrations all end with the block in about the same place, and for 84% of standard test goals the graph already held a nearly identical state. On a harder test, with start and goal drawn from different demonstrations, the memory planner fell from 75% to 45%, and the best version reached 56%, failing mostly on goals that run against the direction of the demonstrations. None of these methods retrains the model. The results come from one researcher's simulation experiments on a single task, and the expectation that they carry over to larger JEPA world models is Alchaer's projection.