Urgent.News

What's breaking now, across thousands of outlets.

Tech

Como aprendi Apache Spark: revisitando uma jornada pela Engenharia de Dados

Quando comecei a estudar Apache Spark, o ecossistema de Big Data era muito diferente do que conhecemos hoje. Hadoop e MapReduce dominavam boa parte das conversas sobre processamento distribuído. Trabalhar com grandes volumes de dados normalmente significava pensar em clusters, HDFS, processamento em lote e muitas operações de leitura e escrita em disco. > Foi nesse cenário que o Spark começou a…

Original Portuguese Read in English

In my journey learning Apache Spark, the big data ecosystem was vastly different from today's landscape. Hadoop and MapReduce dominated conversations around distributed processing. Working with large datasets typically involved thinking about clusters, HDFS, batch processing, and numerous disk read and write operations. It was within this context that Spark began to capture my attention.

This series emerges from the idea of revisiting my old study materials on Apache Spark and reconstructing them with the perspective I now have as a Data Engineering and AI professional. I aim not to simply republish old notes, but rather to understand what has aged, what remains highly relevant, and most importantly, which concepts have traversed different generations of Spark and continue to be present in modern data engineering.

Why revisit Apache Spark? Some technologies change so dramatically that studying their older versions might seem like archaeology. Spark is different. APIs have evolved. The ecosystem has shifted. RDDs are no longer the primary abstraction used in many applications. DataFrames, Spark SQL, Catalyst, Structured Streaming, and various optimizations now occupy a central role.

Yet, several fundamentals remain essential. Concepts such as distributed processing, partitioning, transformations and actions, lazy evaluation, lineage, DAGs, shuffle, distributed execution, fault tolerance, serialization, and memory management continue to explain why a Spark job works—or why it becomes terribly slow. It is precisely in these fundamentals that studying the old materials still holds value.

A crucial note about this series: It will not merely be a reproduction of my old materials. Some of them utilize older versions of Spark and APIs that have been replaced or significantly evolved. Whenever this occurs, I will separate three things: how it functioned, the behavior or API used during that stage of Spark's evolution, and the underlying concept that remains important regardless of the version.

By thinking about it this way today, we will sometimes see old code. And that is intentional. The legacy code also tells the story of a technology—from Hadoop MapReduce to Spark.

One idea that made Spark especially interesting was reducing the dependence on successive disk read and write operations during certain types of processing. In iterative workloads and analytical queries, this made a tremendous difference. But it would be simplistic to reduce Spark to: Spark is fast because it processes everything in memory.

The architecture is far more interesting. Spark introduced a distributed computing model based on abstractions capable of maintaining information about how data was produced. One of these abstractions became fundamental to understanding the early years of the project: RDD — Resilient Distributed Dataset. A RDD represents a distributed collection of data that can be processed in parallel.

But the real interest lies not just in the fact that the data is distributed. It is the combination of partitioning, immutability, functional operations, lazy evaluation, lineage, recomputation, and fault tolerance that have created a powerful model for distributed computing. In 2012, the work on Resilient Distributed Datasets received the prize for the best paper at NSDI.

Years later, many of the high-level abstractions used in Spark would be built upon foundations established during that period. Then came DataFrames, Spark SQL, and much more.

The story didn't stop with RDDs. The ecosystem evolved. Spark SQL made processing structured data much more accessible. DataFrames raised the abstraction level. The Catalyst Optimizer began to play a crucial role in query optimization. The Tungsten project brought important advancements in execution and memory management. Structured Streaming unified batch and streaming processing under a more uniform model.

And Spark became one of the leading technologies in modern data engineering. It is this evolution that I want to explore throughout this series. What we will study In the upcoming articles, we will cover topics such as Apache Spark's architecture; RDDs; transformations and actions; lazy evaluation; SparkContext; distributed execution; Client Mode and Cluster Mode; partitioning; joins and shuffles; accumulators; broadcast variables; JSON; DataFrames; Spark SQL; schemas; data sources; window functions; Spark Streaming; testing; memory management; memoryOverhead; optimization, and troubleshooting.

We will also compare some historical APIs with how we would solve the same problems today. The goal is not to memorize Spark. After working with Data Engineering for some time, it becomes increasingly clear that knowing how to write Spark code does not necessarily mean knowing Spark. It is relatively easy to write something like: df . groupBy ( customer_id ). count () What is truly important is being able to ask: How many partitions exist?

Will there be a shuffle? What volume will be moved across the network? Is the dataset adequately partitioned? Is the dataset fit within partitions? Is the problem on the driver or executors? Is the bottleneck CPU, memory, disk, or network? Does this transformation really need to exist? That type of reasoning transforms knowledge of an API into knowledge of distributed data engineering.

Next article In Part 1, we will begin with the fundamentals: What is Apache Spark and why was its architecture so important? We will understand the problem Spark sought to solve; its historical relationship with Hadoop; the role of RDDs; in-memory processing; fault tolerance; and how these decisions influenced the Spark we use today. This series will be a journey through the versions of Spark.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Other reports of this story

Every outlet that reported this, and when →

Read the original at dev.to →

More in Tech

Managing Notion Databases as Code with notionctl

If you've ever tried to automate Notion databases — setting up relations, managing IDs across workspaces, or keeping databases consistent across environments — you know the pain.

  • notionctl simplifies managing Notion databases as code
  • YAML config defines databases, relations, properties
  • notionctl handles relation resolution, drift detection

More from Tuesday 6 October →