The Physical Nightmare of On-Device AI: Why Your LLM Fails on a Custom PCB

In the year 2026, On-Device AI has become the latest craze in the tech industry. Companies like Apple, Google and Microsoft are all racing to integrate large language models (LLMs) and vision models into portable devices such as smartphones and laptops. However, when developers move from testing models on powerful workstations to deploying them on custom PCBAs (Printed Circuit Board Assemblies) they design themselves, they often encounter significant physical challenges.

The first major issue developers face is the difference between peak compute and sustained compute. Chip manufacturers often advertise TOPS (Trillions of Operations Per Second) as a measure of performance, but in reality, chips cannot sustain their peak performance indefinitely. When an LLM inference model is executed at full load, it generates an immense amount of heat, which can cause the chip's temperature to rise to over 100°C within minutes.

If the PCB design lacks adequate thermal management, such as large copper pours, dense thermal via arrays, or external heatsinks, the chip can overheat, triggering thermal throttling. This means the model may run smoothly for a few seconds before slowing down or crashing due to protection mechanisms.

Another problem developers encounter is voltage droop, or instantaneous current draw leading to an abrupt loss of supply voltage. When the AI chip executes dense tensor operations, it draws tens of amps in microseconds from a low-power sleep state. If the PCB's power delivery network (PDN) cannot respond quickly enough, the voltage can collapse, causing the chip to crash and reboot.

This "false OOM" cannot be fixed by adjusting code, as it is a hardware issue related to the power supply. To mitigate this problem, PDN design must prioritize decoupling capacitors placed close to the chip pins and use thinner dielectric layers to increase plane capacitance.

The third obstacle is the "invisible ceiling" of memory bandwidth and signal integrity (SI) issues. On traditional PCs or servers, memory bandwidth is rarely a concern, but on edge devices, it can be a fatal bottleneck. LLM inference is a memory-bound task, meaning the speed at which data can be transferred between memory and the main chip determines performance.

If the memory cannot keep up with the chip's demands, everything else becomes irrelevant. Additionally, data exchange relies heavily on PCB traces, and any issues with length matching, impedance control, or phase skew can result in data corruption or model failure.

Finally, the limited physical space in AI devices poses challenges for power and ground routing. In software, resource competition can be managed by the operating system, but in hardware, layout space is physically fixed. In compact AI devices, high-speed signals, high-current power lines, and sensitive analog sensors must be squeezed into a few square centimeters.

Poor layout can cause switching noise from high current to couple onto analog signal lines, leading to sensor data corruption. To address this issue, PCB designers must strictly partition analog ground (AGND) and digital ground (DGND), adding shielding and isolation between power and ground planes. Manufacturing processes like HDI (High-Density Interconnect) with laser-drilled microvias can help achieve more interconnections in limited space and minimize signal interference.

In conclusion, deploying on-device AI is not just a software challenge; it is a complex interplay between algorithms, thermodynamics, electronics, and materials science. AnyPCBA, a manufacturer specializing in small-to-medium volume PCB and PCBA production, frequently assists edge AI teams with Design for Manufacturing (DFM) reviews. By understanding these physical limitations, developers can better bridge the boundary between algorithm and physics, ensuring their models run stably on custom PCBAs in users' pockets.