Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning

Quantization-aware distillation (QAD) restores much of the short-form question-answering performance lost to sub-3-bit quantization, yet leaves mathematical and code reasoning substantially impaired. Long generations often degenerate into repetitive loops, exhausting the decoding budget without completing a solution. We trace this gap to quantization-amplified exposure bias: QAD trains on fixed…

arxiv.org ·11h ago

We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.

Read the original at arxiv.org →

Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning

More in AI

I replaced Gemini with a local AI model on my phone; battery drain was surprisingly low

I Sent an AI Agent to Hunt Open-Source Bounties for Three Days: It Earned $0

The missing layer in AI tooling: sharing what your assistant already knows

TraceVIC: Causal Reasoning over Code Evolution for Identifying Vulnerability-Inducing Commits

Writing Rust code that's faster than state-of-the-art libraries by asking agents to make the code faster

Heidi Lands $340 Million to Push Healthcare AI Beyond Notetaking to Action