5.2 Spark: Lineage, DAG, DataFrame, and Data Skew

Spark organizes transformations into a lazily evaluated Directed Acyclic Graph (DAG). An action triggers execution and may involve several jobs. Performance hinges not on whether data resides in memory, but on minimizing scans, shuffles, serialization, and unnecessary materialization, while ensuring that failed partitions can be recomputed.

RDD's Three Core Properties

• partition: the unit of parallelism and data locality;
• dependency/lineage: which parent partitions are used to compute each partition;
• function: how the current data is derived from its parent data.

RDDs are immutable. If a partition is lost, it can be recomputed along the lineage without needing to persist every intermediate result.

A reliable checkpoint writes results to reliable storage while they are still computable, truncating lineage only after completion. This suits long dependency chains or jobs needing an independent recovery point. Calling checkpoint after source data has already disappeared cannot recover missing partitions. RDD checkpoint() first marks the intent; a later action materializes it.

cache/persist retains recomputable results, with disk use determined by StorageLevel. Rebuilding lost cache still requires source data and deterministic lineage. localCheckpoint instead truncates dependencies using executor-local caching and lacks the fault-tolerance guarantee of a reliable-storage checkpoint. The saved persistence cost may return as a recovery problem when an executor disappears.

Transformation and Action

map, filter, select, join, etc., are built first as transformation plans; count, collect, and "write" actions trigger execution.

Repeating actions without persistence may result in recomputing the entire lineage. Persistence isn't more beneficial the more it's used: caches consume memory, may spill to disk, and require eviction and serialization.

collect() pulls all results into the driver, making it suitable only for very small datasets. Production code must have hard limits, decisions based solely on development samples are unacceptable.

Narrow Dependencies and Wide Dependencies

Narrow dependencies such as map/filter compute child partitions directly from the corresponding parents. Several transformations can be pipelined within one stage while partition tasks still run in parallel.

Wide dependencies that redistribute data across partitions require a shuffle and create a stage boundary. A join or keyed aggregation does not necessarily shuffle every time: compatible existing partitioners, reusable distribution, or a broadcast join can change the plan. Inspect actual dependencies and plans rather than classifying solely by operator name.

It's preferable to perform map-side combine using reduceByKey/aggregateByKey rather than first groupByKey transporting all values across the network. However, custom aggregations must still satisfy the correct associative semantics.

DataFrame Makes the Engine See Structure

In RDDs, any function is a black box to the optimizer; DataFrames provide a schema and expressions that enable the optimizer to perform:

• predicate pushdown where the data source supports it;
• column pruning;
• constant folding;
• join reordering and strategy selection;
• more compact memory and code generation paths.

UDFs can block some optimizations, so prefer built-in expressions. When examining execution plans, review the logical plan, physical plan, and runtime statistics together; don't rely solely on the linear chain of source code.

Join Strategies and Skew

• Broadcast join: Broadcast the smaller side to the executors to avoid large table shuffles;
• Sort-merge join: Partition and sort both sides by key, ideal for large-scale data;
• Shuffle hash, and other strategies: applicability depends on the engine and runtime statistics.

Before broadcasting, account for serialized transfer size, construction cost on the driver, and memory for the complete decoded hash table on each executor. Compressed file size is not the size of that table. Executor count also multiplies aggregate transfer and storage cost. Skewed join keys can be split, handled separately, or optimized using adaptive skew handling, but nulls or default keys are often indicators of data quality issues and should not be masked merely by adding more resources.

Small Files and Partition Count

Too many small files increase metadata overhead, task scheduling, and file open costs; too few large partitions reduce parallelism and expand the scope of failure and reprocessing.

The number of output partitions should be designed based on data volume, downstream read patterns, and target file size. coalesce(1) pushes all output through one task and normally suits only small results. It may also reduce parallelism in preceding narrow-dependency work. Getting one output file is easy; getting the final task to finish promptly may be harder.

Validate Work Rather Than Just "Success"

• Input, filtered, and output record counts;
• Primary key uniqueness, null rate, and range constraints;
• Partition size distribution and ratio of maximum to median;
• Shuffle, spill, GC, and executor loss;
• Whether the plan utilizes expected pushdown or join;
• Whether rerunning produces the same business outcome;
• Whether output publication is atomic and whether failure retries result in duplication.

References

• Apache Spark, RDD.checkpoint

• Apache Spark, RDD.localCheckpoint

• Zaharia et al., Resilient Distributed Datasets

• Apache Spark, RDD Programming Guide

• Apache Spark, SQL Performance Tuning