Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL

AsyncGRPOTrainer can now train a LoRA adapter and sync only that adapter to vLLM, eliminating the need for NCCL communication. This allows a rank-1 adapter for a 1.5B model, which is just a few megabytes in size, to be transferred via a storage bucket mounted in every HF Job instead of across machines. A small proxy in front of the vLLM replicas handles authentication, routing, and broadcast of adapter loads to each replica.

Five runs took the same recipe from 3 hours and 27 minutes to 53 minutes for 500 steps. LoRA support was recently added to TRL's AsyncGRPOTrainer, and it ships with TRL v1.14. Training and inference can now be run on separate machines, rather than sharing a machine.