Enhancing Task Generalization in Robotics with World Action Models
Research explores video and action generalization for improved robot learning.
Achieving broad task generalization in robotics is a significant challenge. This study investigates World Action Models (WAMs) by focusing on video generalization, which predicts visual outcomes for new tasks, and action generalization, which converts these predictions into actions.
By testing various video and action supervision methods on RoboTwin 2.0, the research reveals that using in-distribution video-action supervision results in a 25.07% success rate for out-of-distribution tasks. When videos of tasks are introduced without action labels, the success rate increases to 55.90%. Adding just one action-labeled demonstration per task further boosts the success rate to 70.70%.
The findings indicate that broader video coverage enhances task transfer by utilizing existing inverse-dynamics capabilities, while sparse action supervision strengthens these capabilities. The study introduces WorldBridge, a WAM that integrates video and action experts through structured attention, achieving state-of-the-art performance across multiple benchmarks.