Distributed Processing with PySpark

Introduction to Distributed Processing with PySpark

A delivery aggregation runs on a cluster, but one task takes much longer than the rest. Adding workers does not necessarily help: the slow task may still hold most of the data.

Spark divides work across partitions. A shuffle moves records between executors so matching keys can be processed together. Understanding where records move is the starting point for explaining cost.

Loading course…