Homelabber uses second-gen Optane memory sticks to run a 2.8-trillion-parameter AI model on CPU alone
A Reddit homelab user ran Moonshot AI’s 2.8-trillion-parameter Kimi K3 CPU-only on Optane PMem 200 at 0.85 tokens/s with a 1-bit quant.
A Reddit user constructed a homelab setup utilizing Intel's second-generation Optane Persistent Memory (PMem 200) to execute the Moonshot AI's 2.8-trillion-parameter Kimi K3 AI model solely on a CPU, without employing a GPU. The 40-core 3rd-gen Xeon Scalable CPU system generated considerable heat, necessitating mandatory active DIMM cooling with dual 12W blower fans to maintain optimal temperatures.
The project, a follow-up to a first-gen Optane build, operated slowly at 0.85 tokens per second, even with 1-bit quantization. However, the heat generated by the Optane DIMMs was so intense that it could potentially damage neighboring DRAM modules. Optane PMem 200 Series memory, launched in 2020, offers up to 32% more memory bandwidth than the first-generation PMem 100 when used on two-socket Xeon platforms.
Despite its performance drawbacks, the second-gen Optane PMem server running Kimi K3 on CPU alone serves as a cost-effective alternative to data center inference engines for home users facing skyrocketing DRAM prices.
Written by urgent.news from Tom's Hardware's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
