Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem

One of the most cost-effective methods to enhance large language models' performance is through whole transformer block removal. This technique not only reduces memory usage but also provides predictable speedups in inference while stacking well with other compression methods. The challenge lies in determining which blocks to eliminate.

Removing the wrong blocks can cause the model to fail, and the impact of removing a single block varies based on other blocks being removed simultaneously, making it a combinatorial problem rather than a ranking problem. The article's latest research, "LLM Compression by Block Removal with Constrained Binary Optimization," draws a direct parallel between this problem and an Ising glass, a disordered spin system with all-to-all interactions and a set number of 'up' spins.

By converting block selection into a constrained binary optimization (CBO) problem, the researchers map it onto the Ising glass, a problem that the physics of spin systems is inherently designed to address. Each transformer block is assigned a binary variable (0 for keeping it, 1 for removing it), similar to a spin that can point either up or down.

This reformulation allows for the direct application of classical and quantum-inspired solvers to find the optimal set of blocks to remove. The research highlights that the energy of the spin system serves as a strong, inexpensive proxy for the pruned model's performance on benchmarks. This enables the ranking of numerous candidate configurations without the need for actual benchmarking.

The method's efficiency is demonstrated by achieving nearly 23 percentage points improvement in MMLU over the best competing block-removal method at 50% compression of Llama-3.3-70B-Instruct. The approach is particularly effective for deep-compression scenarios, where brute force enumeration of configurations is impractical. By leveraging solvers tailored for Hamiltonian problems, the method can scale to large models, solving the hardest cases in days using open-source tabu solvers.