5.2 Spark: Lineage, DAG, DataFrame, and Data Skew

Spark organizes multiple transformations into a Directed Acyclic Graph (DAG), which is only materialized when an action is triggered. Performance hinges not on whether data resides in memory, but on minimizing scans, shuffles, serialization, and unnecessary materialization, while ensuring that failed partitions can be recomputed.

RDD's Three Core Properties

• partition: the unit of parallelism and data locality;
• dependency/lineage: which parent partitions are used to compute each partition;
• function: how the current data is derived from its parent data.

RDDs are immutable. If a partition is lost, it can be recomputed along the lineage without needing to persist every intermediate result.

Long lineage chains or unavailable source data can trigger checkpointing, which writes the state to a durable storage and truncates the dependency chain. cache/persist is a performance optimization that allows recomputation even after an executor failure; checkpointing is more focused on recovery boundaries and has a different semantic meaning.

Transformation and Action

map, filter, select, join, etc., are built first as transformation plans; count, collect, and "write" actions trigger execution.

Repeating actions without persistence may result in recomputing the entire lineage. Persistence isn't more beneficial the more it's used: caches consume memory, may spill to disk, and require eviction and serialization.

collect() pulls all results into the driver, making it suitable only for very small datasets. Production code must have hard limits, decisions based solely on development samples are unacceptable.

Narrow Dependencies and Wide Dependencies

In narrow dependencies, a single parent partition is used by only a few child partitions (for example, map/filter) allowing it to be processed sequentially within the same stage.

In wide dependencies, data from multiple parent partitions flows into multiple child partitions (for example, groupByKey/repartition/join) requiring a shuffle operation and creating a stage boundary.

It's preferable to perform map-side combine using reduceByKey/aggregateByKey rather than first groupByKey transporting all values across the network. However, custom aggregations must still satisfy the correct associative semantics.

DataFrame Makes the Engine See Structure

In RDDs, any function is a black box to the optimizer; DataFrames provide a schema and expressions that enable the optimizer to perform:

• predicate pushdown;
• column pruning;
• constant folding;
• join reordering and strategy selection;
• more compact memory and code generation paths.

UDFs can block certain optimizations, prefer built-in expressions instead. When examining execution plans, always review both the logical plan, physical plan, and runtime statistics in parallel; don't rely solely on the linear chain of source code.

Join Strategies and Skew

• Broadcast join: Broadcast the smaller side to the executors to avoid large table shuffles;
• Sort-merge join: Partition and sort both sides by key, ideal for large-scale data;
• Shuffle hash, and other strategies: applicability depends on the engine and runtime statistics.

The broadcast threshold must account for both the serialized size and the number of executors. Skewed join keys can be split, handled separately, or optimized using adaptive skew handling, but nulls or default keys are often indicators of data quality issues and should not be masked merely by adding more resources.

Small Files and Partition Count

Too many small files increase metadata overhead, task scheduling, and file open costs; too few large partitions reduce parallelism and expand the scope of failure and reprocessing.

The number of output partitions should be designed based on data volume, downstream read patterns, and target file size. coalesce(1) will route the entire output into a single task, which is typically only suitable for very small result sets.

Validate Work Rather Than Just "Success"

• Input, filtered, and output record counts;
• Primary key uniqueness, null rate, and range constraints;
• Partition size distribution and ratio of maximum to median;
• Shuffle, spill, GC, and executor loss;
• Whether the plan utilizes expected pushdown or join;
• Whether rerunning produces the same business outcome;
• Whether output publication is atomic and whether failure retries result in duplication.

References

• Zaharia et al., Resilient Distributed Datasets
• Apache Spark, RDD Programming Guide
• Apache Spark, SQL Performance Tuning