Debugging · Advanced · 35 min
A Spark job reads thousands of tiny files
Distinguish file listing and task overhead from a hot key.
Task
A Spark scan reads 80 GB split across 400,000 small files. The job spends most of its time listing files and starting tasks. One old 20 GB file is also present. There is no join or group-by in the slow stage.
Requirements
- Use the scan plan, file sizes, task duration distribution, and input format to explain the likely bottlenecks.
- Do not assume one file always equals one Spark task: splitting and file grouping depend on the reader and configuration.
- Compare upstream compaction, partition sizing, coalesce/repartition, and a different file format.
- Explain how you verify schema, row counts, and reader visibility when replacing files.
Spark scansSmall filesCompaction
Loading exercise…