Why Go's encoding/csv Burns 540MB on 5M Rows (and How I Fixed It)
TL;DR: Go's built-in encoding/csv.Read() allocates string slices and strings for every single row, generating 10M+ allocations and burning 540 MB on a 5M-row file. I built go-zerocsv to replace this with in-place typed scanning ( Record.Scan ) and a compacting 4 KB buffer, keeping memory flat at ~5 KB with zero allocations per record on the hot path. If you parse or write large CSV files in Go,…
Go's built-in encoding/csv package allocates memory for a new []string slice and newly allocated heap strings for every single row in a CSV file. This leads to a significant number of allocations, burning up to 540MB of memory when processing a 5M-row file. To address this issue, I created go-zerocsv, a CSV parser and writer that replaces the standard library's in-place typed scanning and a compacting 4 KB buffer. This implementation keeps memory flat at around 5 KB and has zero allocations per record on the hot path.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
