Urgent.News

What's breaking now, across thousands of outlets.

Tech

How do you tune speculative decoding in a production serving stack?

Tune speculative decoding against your own traffic, not a benchmark. Choose a method that fits your workload, set draft length from measured acceptance rate, and re-test at production concurrency, because the speedup shrinks as batches grow. Key takeaways : Watch acceptance rate—draft length is the main knob that moves it. Concurrency decides whether speculation helps or hurts, so sweep load with…

When tuning speculative decoding in a production serving stack, you should base your method on your own traffic rather than benchmark data. Choose a method that suits your workload—n-gram and suffix decoding offer modest speedups without increasing workload at peak traffic, while model-based methods like EAGLE and MTP provide the best latency reduction.

If your model ships native MTP support, start with that; otherwise, EAGLE is a strong general-purpose default. Adjust the draft length based on the acceptance rate, as longer drafts promise more speedup but have a higher rejection rate. Test various values and monitor per-request acceptance metrics before settling on a number. Once extra tokens are mostly rejected, stop increasing the draft length.

Testing should be done at your real concurrency levels because draft tokens compete with real tokens for compute at high concurrency, and rejected drafts waste resources. Run load tests at low, medium, and peak concurrency, and also include a run with speculation disabled. If your traffic fluctuates, consider vLLM's adaptive verification, which sizes verification per request.

With the standard rejection sampler, vLLM describes speculative decoding as lossless up to hardware numerical precision, so use lossy options like FLy only in non-quality-sensitive paths. Finally, if configuring vLLM yourself is not feasible, consider using DigitalOcean Dedicated Inference, which runs vLLM and llm-d on dedicated GPUs and handles orchestration, endpoints, and scaling for you.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

5 Security Mistakes I See in Flask + React Apps (and How to Fix Them)

Flask and React are a great pairing: fast to build, flexible, and easy to deploy. But that same speed means security often gets pushed to "later," and later usually means after something breaks.

  • Validate user input against schema using Pydantic
  • Store JWTs in HttpOnly, Secure, SameSite cookies
  • Include token in Authorization header on React frontend

What happens when an offline POS sale syncs twice?

A cashier completes a sale while the connection is down. Later, the server saves it, but the response never reaches the device. The app retries.

  • Unique client sale ID assigned to prevent duplicate transactions.
  • Server handles retries safely with unique database constraint.
  • Testing essential to verify receipt number and inventory accuracy.

Running untrusted pull request code from agents and contributors

Originally published on sandscale.dev . A pull request is a request to run someone else's code on your machine. CI has always worked this way: check out the branch, install dependencies, run the…

  • Pull requests run untrusted code, posing risks like code escaping or data exfiltration
  • Sandscale's sandbox creates a separate kernel and network environment for each PR
  • After tests, sandbox should be stopped and discarded to prevent lingering malicious code

More from Friday 20 November →