Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets
Modern Intel AI PCs ship capable integrated GPUs and NPUs with 16+ GB of unified memory, and they spend considerable time idle. That is not enough memory to fit a large model such as a 70B-parameter LLM. We show that a handful of AIPCs, working together over an ordinary network, can serve models beyond the capability of any single one. We use pipeline parallelism: a model is split by layer into…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
