DataHelix: A CPU-GPU Co-Scheduling Data Plane for Multimodal Model Training
Multimodal model training breaks the loose coupling between data loading and model execution assumed by conventional training pipelines. Large per-sample variation in decoding cost, token/latent count, and tensor size causes both CPU preprocessing time and GPU compute time to vary substantially across data-parallel ranks (training processes). In synchronous training, these two sources of…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
