Why are you still debugging Spark on your host machine?
Last October, a "minor" dependency bump in a PySpark job cost us four hours of downtime and roughly $12,000 in cloud compute credits. A developer had tested a new UDF locally using an older version of delta-spark than what we ran on our EMR cluster. Locally, the serialization worked fine. In production, the different Scala/Java versions in the underlying runtime caused a…
Spark debugging has become a headache for a team after a minor dependency update caused four hours of downtime and $12,000 in cloud costs. Developers were able to test new UDFs locally, but the different Scala/Java versions in production led to java.io.NotSerializableExceptions. The developer realized that having a local environment not matching the cluster caused issues and decided to either stay in a problematic situation or containerize their local development.
The two options are installing Java, Python, Spark, and Delta Spark on your machine or creating a Docker container that mirrors the production environment. The second option, although requiring initial effort, provides a consistent environment, easier debugging, and faster local development. The author recommends the containerized approach, especially for those working on critical data, as it ensures the code runs consistently across different environments, saving time and money.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
