Python Generators and Iterators: Process Large Data Without Blowing Up Memory

Python's generators and iterators allow you to process large data sets without consuming massive amounts of memory. When scripts begin using hundreds of megabytes of RAM, the problem is often that the entire dataset is loaded into memory at once. Generators provide a lazy evaluation solution that lets you process data one item at a time.

Consider reading a large log file and counting how many lines contain the word "error". Using the straightforward approach loads all lines into memory at once with readlines(), which will cause issues with very large files. The fix is to iterate directly over the file object, yielding one line at a time. This approach keeps memory usage flat regardless of file size.

Generators are created by generator functions or generator expressions, which use the yield keyword. They return a generator object that can be iterated over. Generators use constant memory, while list comprehensions build and return a full list, which can scale with file size. Generators are single-use, so if you need to iterate multiple times, you must recreate them or store the results in a list.

Practical applications include chunked processing with islice to feed records into a database, and streaming aggregations to pipeline multiple transformations without ever materializing intermediate lists. This allows you to work with large data sets efficiently, keeping memory usage predictable and manageable.