Batch Workflows and Publication

Batch Ingestion Considerations

Batch ingestion collects data into discrete units of work. A batch may be defined by a time interval or by the amount of data accumulated.

A time-based batch collects an interval’s data. A size-based batch closes after enough data has accumulated, which can be useful when writing incoming events into files.

Smaller batches can deliver fresher data but increase scheduling and file-management overhead. Larger batches reduce that overhead while making consumers wait longer.

Choose a boundary that balances freshness and processing cost. Record what each batch includes so failures and retries can address the same input.