AI2 built a toy robot world, and its own models still missed half the tests
ToyBlocks is a simulated tabletop of wooden blocks that Ai2 built to study why robots fail. The first results show its own pretrained models still can't reliably handle a new block layout.
Ai2 wants to know why robots fail, so it built them a smaller world to fail in. On November 6, the Allen Institute for AI introduced ToyBlocks, a simulation of a robot arm pushing, flipping, and stacking wooden blocks on a tabletop.
It’s the first of what Ai2 calls FoundationWorlds, and the pitch is almost biological: the way scientists study a complex system by starting with a petri dish, Ai2 wants a controlled environment small enough that researchers can change one variable, rerun the test, and actually trace why a robot’s model got something wrong.
The blocks have letters and pictures printed on their faces, so a task isn’t just “stack three blocks.” It can be “spell Ai2,” which forces the robot to flip a block over to check a hidden face before deciding where it goes. Simple on paper. In Ai2’s own early results, it wasn’t simple for the models.
The finding that’s the real story here
Vision-language-action models, the kind that map a camera image and a text instruction straight to robot motion, are supposed to generalize from pretraining on large datasets. Whether they actually do has been an open argument: some papers say these models are brittle to new object positions and lighting, others say they transfer fine. Ai2’s read is that most of that disagreement comes from real-world datasets that can’t vary spatial layout cleanly enough to test the question.
ToyBlocks can. Ai2 ran three models, ManiFlow, π0.5, and MolmoAct 2, on tasks that varied where blocks spawned and how many blocks were in a stack, splitting results into interpolation (new setups inside the training range) and extrapolation (setups outside it). The result, in Ai2’s words: “despite large-scale pretraining, these popular action models generalize poorly in the initial results.” Not one outlier model. All three.
What held up, and what didn’t
Ai2 also tried transferring its own models, a Molmo 2 high-level planner paired with a ManiFlow low-level controller, from pure simulation onto a real Franka FR3 arm, with zero training on real-world examples.
| Measure | Simulation | Real robot |
|---|---|---|
| ManiFlow, single skill (pick, place, flip, rotate) | 80.0% | 81.1% |
| Molmo 2 + ManiFlow, full spelling task (3 blocks) | 55.5% (111/200) | 50.0% (15/30) |
The single-skill number held up almost exactly between sim and reality, which is the part Ai2 can point to as evidence its physics-matching work paid off. The full task, chaining several skills together with a planner deciding what to do next, dropped to roughly a coin flip in both settings.
Ai2 flags the real-world count itself: 30 trials isn’t enough to call 50.0% a solid number, only a direction.
Claude plays with blocks, and mostly gets it right
The more interesting experiment, to me, wasn’t the trained models. It was Ai2 wiring ToyBlocks’ skills into the Model Context Protocol and letting Claude Opus 5.5 plan and act directly, calling pick, flip, and rotate like tools. Given the same block-stacking tasks that stumped frontier models a few years ago, Ai2 says Claude solved all 30, both from text descriptions and from a closed loop of repeated camera observations.
The harder version, “search and spell,” asked the agent to find and stack blocks spelling a three-letter word among as many as eight blocks, and to correctly say “impossible” when the needed letters weren’t actually in the scene (true for half the scenes). Claude solved or correctly flagged 63% of those. Most of the misses were letter confusion or an upside-down letter it didn’t catch, not planning failures. Ai2 also found a cheap fix: letting the agent write its own skill descriptions from its own failed attempts raised its success rate from 24% to 63%, beating the 36% it got with descriptions a human expert wrote. That’s a small result on a toy task, but it’s a real one, and it says something about where the actual bottleneck in robot agents sits right now: not the planning, but whether the agent understands its own tools.
My read: the headline fact here isn’t that a toy-block world is cute. It’s that a lab with real resources ran its own pretrained, purpose-built models through a clean test and got a coin-flip on the full task. That’s a more honest number than most robotics demos put in a blog post, and it’s worth more than a flashier result would be.
What’s actually released
Ai2 put out the simulation code, the automatic task and demonstration generators, two trained models (one built on Molmo 2, one on ManiFlow), training code, and an initial dataset of 85,336 long-horizon demonstrations, covering about 1,600 hours of simulated robot execution and 144 million frames. A larger second release, with more skills and broader benchmarking, is due in a few weeks, alongside integrated reinforcement-learning training that isn’t in this first package.
It’s a narrower release than Physicl’s $10 million bet on synthetic training scenes for robots, announced the same day: Physicl sells generated environments at scale, while ToyBlocks is a research tool meant to isolate why a model fails, not a production data pipeline. Different problem, same underlying bet that robotics still needs better simulated ground truth before it needs more real-world video.
Note on sources: every figure, quote, and benchmark number above comes from Ai2’s own ToyBlocks announcement on Hugging Face; no independent outlet had covered the release separately as of publication.