Debugging · Advanced · 35 min
A successful backfill writes today’s partition
Separate the processing interval from the wall clock.
Task
A daily pipeline is backfilled for January 10, but the task chooses its partition using date.today(). It writes today’s folder and still reports success. A downstream dbt model publishes stale January metrics.
Requirements
- The scheduled run has a declared half-open data interval and must be reproducible on retry.
- Explain the wrong-partition defect and the contract between extraction, transformation, and publication.
- Describe the role of a readiness sensor and why a callback alone is not a data-quality gate.
- Include a retry-safe output path, an interval-specific validation step, and a way to detect a missing expected partition.
OrchestrationBackfillsData quality
Loading exercise…