TAVIS: A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning

A benchmark for egocentric active-vision imitation learning and anticipatory gaze on humanoid torsos. TAVIS ships eight simulated manipulation tasks across two robots (Fourier GR1T2 and Pollen Reachy 2), 2,200 VR-teleoperated demonstrations, pretrained π₀ baselines, and a proprioceptive metric -- GALT (Gaze-Action Lead Time) -- for quantifying anticipatory gaze in the learned policies.

---

👀 For reviewers:

1. Data can be viewed from the browser (zero download).

• Viewer: https://huggingface.co/spaces/lerobot/visualize_dataset -> paste, e.g., tavis-benchmark/tavis-head-gr1t2
• Sample datasets: tavis-head-sample-gr1t2 · tavis-head-sample-reachy2

2. Run a pretrained policy across a whole suite (no training). Download a multi-task π₀ checkpoint and roll it out:

huggingface-cli download tavis-benchmark/pi0-tavis-head-gr1t2-headcam \
    --local-dir checkpoints/pi0-tavis-head-gr1t2-headcam

python scripts/eval_benchmark.py \
    --checkpoint checkpoints/pi0-tavis-head-gr1t2-headcam \
    --suite tavis-head --robot gr1t2 \
    --episodes 96 --num-envs 1

python scripts/print_benchmark_results.py results/pi0-tavis-head-gr1t2-headcam

This evaluates all 5 TAVIS-HEAD tasks × {id, ood_spatial} × 96 episodes and prints success rates with Wilson 95% CIs plus per-episode GALT. (Requires the code + an RTX GPU -- see Installation. Several hours on a single 4090.)

---

📦 Datasets

All released as LeRobotDataset v3.0. The four multi-task suites total 2,200 VR-teleoperated demonstrations (~3 h of interaction).

Also released: tavis-head-sample-gr1t2 and tavis-head-sample-reachy2 (small previews) · tavis-assets (robot / task USD assets).

🤖 Pretrained checkpoints (π₀)

The headcam / fixedcam pair on TAVIS-HEAD is the core active-vision vs. fixed-camera comparison.

---

🎯 Tasks

Eight tasks in two suites. The robot, camera mode (headcam / fixedcam), and eval mode are orthogonal axes set at evaluation time.

TAVIS-HEAD -- global visual search & clutter

TAVIS-HANDS -- local occlusion & peeking

Both robots expose a canonical 19-D action layout (left/right arm IK targets, 3-DoF neck, two gripper scalars), so policies share one action space across embodiments.

📐 Evaluation protocol

Each (robot × task × eval-mode) cell is evaluated over 96 stochastic episodes; success rates are reported with Wilson 95% confidence intervals. Three eval modes (see docs/ood_modes.md):

GALT (Gaze-Action Lead Time) -- t_hand_arrival - t_head_arrival in seconds -- measures how far in advance the head settles on the target before the hand arrives. It is proprioceptive: computed purely from the 19-D commanded-action trajectory (no eye-tracking, no scene state), so it ports to any robot exposing those channels. Every benchmark rollout emits a GALT estimate alongside its success flag. Details: docs/galt.md.

---

Installation

Download the code from the anonymous mirror, unzip, then from inside the extracted folder:

bash install.sh
conda activate tavis

install.sh bootstraps the full pinned stack (via uv, applying the dependency overrides in pyproject.toml) into a fresh tavis conda env on Python 3.11.

Hardware. Simulation, data collection, and evaluation need an RTX GPU. Training: diffusion-policy and π₀-LoRA fit on a single 24 GB card; full π₀ training fits on an H100.

Reproduction recipes

A) Run a pretrained multi-task policy across a whole suite. Fastest path; no training. See above ("For Reviewers").

B) Train + evaluate a single-task diffusion policy from scratch (~12 h on a 4090). The released datasets are multi-task suites; --task filters to one task class:

huggingface-cli download tavis-benchmark/tavis-head-gr1t2 \
    --repo-type dataset --local-dir datasets/tavis-head-gr1t2

python scripts/train_policy.py \
    --dataset datasets/tavis-head-gr1t2 \
    --task conditional_pick --model diffusion --camera headcam \
    --steps 200000

python scripts/eval_benchmark.py \
    --checkpoint checkpoints/tavis-head-gr1t2__conditional_pick_diffusion_headcam \
    --robot gr1t2 --tasks conditional_pick \
    --eval-modes id ood_spatial --episodes 96 --num-envs 1

Multi-task training is π₀-only (it uses the per-episode language instruction). The paper's multi-task π₀ runs used H100-days on an external orchestrator, so for reviewers the released π₀ checkpoints (recipe A) are the practical reproduction path. Full recipes: see the mirror's README.md.

Documentation (in the code mirror)

License

• Code: MIT.
• Datasets: CC-BY-4.0.
• Robot models, YCB objects, task assets: under their respective upstream licenses (NVIDIA IsaacLab, the YCB project, and the original robot-model authors); per-asset attribution ships with tavis-assets.

Citation

If you use TAVIS -- benchmark, datasets, or checkpoints -- please cite the TAVIS paper. The formal BibTeX will be posted here once the anonymous review period concludes; until then, please cite it by name as "TAVIS: A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning".

Contact

Questions and bug reports (anonymous-review-safe): tavis.benchmark@gmail.com.

Acknowledgments

Built on IsaacLab (NVIDIA), the LeRobot framework (Hugging Face), and IsaacLab-Arena. Robot models from the upstream Fourier GR1T2 and Pollen Reachy 2 USD distributions; objects from the YCB project; VR teleoperation via Meta Quest.