Como aprendi Apache Spark — Parte 7: Memória — Executors, Driver, Cache, Spill e OutOfMemory

Memory plays a crucial role in Apache Spark, touching every stage of the application's execution. Initially, the focus was primarily on OutOfMemory errors and increasing RAM, but this approach oversimplifies the issue. A memory problem can stem from various components such as the Driver, Executors, Partitions, Shuffle, Broadcast, Cache, Serialization, Garbage Collection, Python workers, Off-heap/native memory, and Memory Overhead.

The key question is no longer "How much memory should I add?" but rather "Which components are consuming memory, why, and where in the architecture?"

1. Driver Memory: The Driver orchestrates the application, managing information related to the application, jobs, stages, tasks, scheduling, metadata, planning, scheduling, metadata, and execution results. Its memory is used to store planning, scheduling, metadata, and execution control.

2. The Dangers of collect(): Using collect() to bring all data to the Driver can lead to significant memory issues. For instance, if a dataset contains 500 GB and the Driver has only 8 GB, there's a predictable problem. The key is to understand the volume of data that needs to be brought to the Driver. Collect() should be used only when the expected result size is small.

3. Executor Memory: Executors execute tasks, and memory is required for processing, aggregations, joins, sorting, shuffle, intermediate structures, serialization, local code execution, and off-heap/native memory. A task responsible for a partition with a large amount of data may experience memory pressure, while other tasks may not.

4. Partition Size and Memory: The partition size and distribution significantly influence memory pressure on tasks. Even though not all data from a partition needs to be loaded into memory simultaneously, understanding this relationship is essential. Simply increasing spark.executor.memory may mask the real cause of the memory issue.