Debugging · Advanced · 30 min
A Spark join has one task that never finishes
Use task evidence to distinguish skew from insufficient compute.
Task
A Spark job joins 2 TB of events to a 5 GB account table. Almost every task finishes in two minutes, but one runs for 50 minutes and spills heavily. Forty percent of event rows have account_id = UNKNOWN. The account table has three rows for that key.
Requirements
- Explain what evidence you inspect before changing cluster size.
- State why the join can multiply records and how to resolve the UNKNOWN key correctly.
- Compare filtering, separate processing, AQE skew handling, salting, and broadcasting.
- Propose checks that prove both correctness and improved performance.
Spark plansSkewJoin cardinalityProduction diagnosis
Loading exercise…