Inference-Time Scaling on a Budget: Making Small Local LLMs Reason Better with Self-Consistency and Verifiers
Tuần này khóa học video "build reasoning models from scratch" của Sebastian Raschka đang lên top Hacker News. Nhiều bạn xem xong sẽ hỏi: "Hay đấy, nhưng mình không có cụm GPU để train RL thì làm được gì?". Câu trả lời là có. Trong các kỹ thuật làm model "suy luận tốt hơn", có một nhánh không cần train lại gì cả: inference-time scaling , tức là dùng thêm compute lúc chạy để đổi lấy độ chính xác.…
The article discusses a technique called inference-time scaling, which involves using more compute power during model inference to improve accuracy without altering the model itself. The author, Sebastian Raschka, applied this technique to an 8B model running locally on an RTX 3090. The main methods mentioned are self-consistency, verifier loop, and when not to use them.
Self-consistency involves asking the model N times with a temperature of 0 and taking a majority vote of the different reasoning paths to filter out noise. A more advanced technique is best-of-N with a verifier, where a separate external checker (like unit tests or EXPLAIN for SQL) is used to select the correct answer. The author notes that the cost increases linearly with N, but batching on an inference server allows for near-linear scalability.
The setup involves using llama-server from llama.cpp, configured with parallel slots and continuous batching. The author provides code examples for implementing self-consistency for tasks with discrete answers and mentions that the cost increases linearly with N, but batching allows for near-linear scalability.
Brief written by urgent.news from Dev.to's own syndicated text. Machine-written — may contain errors; check the original before relying on it.
