What is LIBERO?

LIBERO is a benchmark rather than a data product. It runs in simulation on robosuite and MuJoCo with a Franka arm at a tabletop, and it is organized into four suites that each vary one thing: LIBERO-Spatial moves object positions, LIBERO-Object swaps the objects themselves, LIBERO-Goal changes what the instruction asks for, and LIBERO-Long chains several steps into a single instruction. Around 130 tasks in total, with human demonstrations collected by teleoperation alongside scripted ones.

The design is the contribution. By changing one axis at a time, the suites separate "the model cannot see the object" from "the model cannot follow the instruction" from "the model cannot hold a plan together": distinctions that a single aggregate success rate hides completely. The Long suite is where most policies fall over, because a multi-step task fails if any step does and the failure rate compounds across them.

For a buyer, LIBERO is most useful as a shared reference point. Because the suites are fixed and public, a success rate on them is comparable between two teams in a way an in-house benchmark never is, which makes it a defensible acceptance criterion for a policy deliverable, provided the brief names which suite and which split. It is not a substitute for real-robot evaluation: a policy that scores well on LIBERO-Long can still fail on the first contact-rich step in the physical world.

What the suites isolate

The four suites are a controlled experiment rather than four difficulty levels. Each holds the robot and the task family steady and changes one axis: object positions in one suite, the objects themselves in another, the wording of the goal in a third, and the number of steps in the fourth. A policy's scores across the suites separate failures that one average would blur together.

That design is what makes the benchmark useful for diagnosis. A policy that copes when positions move but fails when objects change has a perception problem, not a planning one. A policy that handles every single step but fails the chained suite is losing to error compounding, where one wrong move ends the whole episode.

Each suite also ships demonstrations for training and held-out conditions for evaluation. The held-out split is the part that measures generalization, because it tests conditions the training demonstrations did not contain. Which split a number was measured on changes what the number means.

How it is used as an acceptance test

The main commercial use of the benchmark is as a shared reference point. Because the suites are fixed and public, a success rate on them can be compared between two teams, which an in-house benchmark never allows. That makes it a defensible acceptance criterion for a policy deliverable, provided the brief names the suite, the split, and what counts as a completed evaluation run.

The limits belong in the same breath. It runs in simulation, on one robot at one tabletop, so it says nothing about contact-rich manipulation, deformable objects or a different platform. A policy that scores well on the hardest suite can still fail on the first physical attempt, and a buyer who treats the score as proof of a working robot will be disappointed.

Used properly it is a filter, not a finish line: a cheap, comparable way to eliminate weak policies before anyone books robot time. It is also not a data product. The demonstrations exist to make the benchmark runnable, and a purchase decision around them is really a decision about simulation.