Dynamic Abliteration: Non-Destructive Refusal Suppression via Engram Steering

The article discusses a new approach called Dynamic Abliteration for controlling refusal behavior in open-weight language models like Qwen, without permanently altering the model's weights. Unlike traditional weight abliteration techniques which modify parameter weights and can degrade performance on non-refusal tasks, Dynamic Abliteration intercepts intermediate residual streams at runtime using PyTorch forward hooks.

This allows for non-destructive suppression of refusal behavior while keeping the base model weights frozen.

The article explains how traditional abliteration methods work by capturing hidden states from a single layer, computing a refusal difference vector, and subtracting it during decoding. However, this approach can still allow downstream layers to reconstruct the refusal behavior, leading to inconsistent results. To address this, the authors propose using Multi-Layer Steering with Engram, a conditional memory architecture introduced by DeepSeek.

Instead of injecting static vectors, Engram evaluates the current layer hidden state against local N-gram memory when processing refusal triggers or hedging headers. This allows for adaptive, context-aware steering that only intervenes when necessary. Engram hashes sliding token windows across 4 prime-modulo tables, enabling O(1) constant time recognition of sequence triggers such as ChatML headers or prompt keyphrases.

To implement this approach, the authors initialize shared memory weights and attach PyTorch forward hooks across specific layers (12, 14, 16, 18, and 20) of the Qwen3-4B model. They train the Engram steering head using 2,000 clean samples from PKU-Alignment/PKU-SafeRLHF data, filtering out unsafe samples and disabling Qwen3's default reasoning tag injection. The base model's backbone is frozen, and only the parameters of the MultiLayerEngramModule are optimized using AdamW and a cosine warmup scheduler.

The results demonstrate that Dynamic Abliteration using Multi-Layer Engram Steering provides a modular, non-destructive alternative to traditional weight abliteration and fine-tuning techniques. It effectively suppresses refusal behavior while maintaining the base model's performance on standard tasks.