Inhabitant

Toward Long-horizon Human-scene Interaction

arXivGitHubDataset

Scroll to step inside

It starts with one natural-language instruction for a household task.

An LLM agent generates a motion sequence of whole-body scene interaction.

One episode

A sentence goes in.
A continuous motion comes out.

segment 1

Click a segment to jump to it. Every block is one motion that the agent committed after trying, measuring and revising it.

How it works

Try. Measure. Revise.

Inhabitant connects a language model to a pretrained motion generator and a scene simulator through two interfaces: spatial control over the motion, and quantitative feedback on its execution. Nothing is trained.

The framework

1try(c, e)spatial control motion 2measure(q)distances, positions, contacts rendered frames and a success check, after every try 4commitor roll back to an earlier segment INPUTInstructionand the scene state, as text Agenta language model, not trained Claude Opus 5.5 or any other language model3revise Motion generatorwhole-body motion from textand spatial constraints Kimodo, by NVIDIA Scene simulatorfurnished scenes, 31 layouts furniture stops the bodyobjects move when touched,thrown or kicked half-physics Interaction sequenceone block per committed segment

Hover or tap a part of the diagram. Technical details are in the paper.

One revision, step by step

try 1

That was one task

Three goals, 29 seconds of motion, 10 segments.

There are 75.

The benchmark, Inhabit

75 long-horizon tasks, 31 layouts.

Each task is one instruction with up to three goals, carried out by the whole body in one of 31 furniture layouts. A task takes 15 to 45 seconds of continuous motion, built from 4 to 12 segments.

Results

74 of 75 tasks completed.

With the same interfaces, the success of Inhabitant follows the language model that acts as the agent. No model is trained.

Success rate on Inhabit

† existing agents adapted to Inhabit, with Claude Opus 5.5. Each task is run once.

The film

Inhabitant in 72 seconds.

74 / 75tasks completed
0models trained
3goals in one instruction, at most
31furniture layouts

Paper

Inhabitant: An Agent Framework for Long-Horizon Human-Scene Interaction with Spatial Control and Quantitative Feedback