TAPESTRY: A Single Backbone for Video Generation and Robot Control
Pretrained video generation models encode rich priors about physical interactions, but leveraging them for robot control typically requires either resource-intensive video generation at test time or action heads trained from scratch that do not fully utilize the pretrained backbone. We introduce Tokenizing Actions as Video Patches in a Shared Transformer Policy (TAPESTRY), a world action model…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
