D-Matrix bets stacked DRAM can solve inference’s memory bandwidth problem
As artificial intelligence workloads shift to inference, memory bandwidth decides whether hardware generates tokens quickly or sits idle. D-Matrix Corp. is answering with Raptor, a second-generation chip stacking 3D DRAM beneath its compute logic. AI infrastructure has become a systems game, and winners combine memory, interconnect and software into cohesive designs, according to Satyam…
D-Matrix Corp. has devised a solution to address AI workloads' increasing memory bandwidth requirements during the inference stage. The company's Raptor chip, which is set to arrive next year, features stacked 3D DRAM beneath its compute logic to optimize performance. According to Satyam Srivastava, a fellow at D-Matrix, memory bandwidth is now a critical advantage that must be tackled because the AI stack encompasses memory, interconnect, and software.
Srivastava explained that the new technology is positioned between SRAM and high-bandwidth memory (HBM) in terms of capacity, offering 10 times greater energy efficiency and bandwidth compared to HBM. By integrating the Corsair-based server with Nvidia's existing Hopper GPU fleet, utilization can soar from around 20% to up to 80%, resulting in more tokens per dollar.
D-Matrix has also joined Nvidia's NVLink Fusion rackscale infrastructure to connect their accelerators to Nvidia's AI factories, thereby tapping into a highly established ecosystem. This strategy aligns with heterogeneous computing, where software coordinates GPUs, CPUs, and accelerators like Raptor as a single system, allowing users to choose the appropriate hardware unit to meet the needs of their workload.
Written by urgent.news from SiliconANGLE's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.