What is batch processing?
Batch processing refers to the execution of a sequence of programs or jobs over a large set of data in batches, as opposed to processing continuously or interactively. Jobs are scheduled and managed as groups.
Batch processing workloads are tuned to optimize throughput, utilization and efficiency when processing very high volumes of data systematically and repetitively.
Batch processing excels at throughput but has higher latency than real-time systems. Big data architectures like lambda architecture combine batch and real-time processing while kappa architecture and unified processing replace batch with fast scalable streams.
What does it do? How does it work?
In batch processing, data is collected over a period and then processed in bulk at scheduled intervals as batches. Jobs run without manual intervention based on pre-defined specifications.
Batch systems orchestrate workflows, load balancing, failure handling, and output delivery while optimizing overall throughput. Batch jobs tend to have high compute needs but predictable patterns.
Why is it important? Where is it used?
Batch processing is critical for data and compute-intensive workloads involving periodic, systematic processing of very large datasets. Use cases include ETL pipelines, report generation, machine learning model training, scientific simulations, image/video processing, transaction processing, and data warehouse loading.
Batch processing provides efficiency, reliability, and economies of scale for massive, repeatable workloads with expected patterns. It underpins many big data architectures.
FAQ
How does batch processing differ from stream processing?
Batch operates on finite stored datasets vs continuous never-ending streams. Latency is higher, throughput and efficiency optimized.
What are some key batch processing tools?
Popular batch processing tools include Hadoop, Apache Spark, Apache Beam, Kubernetes CronJobs, Amazon EMR and AWS Batch.
What are the main challenges with batch processing?
Challenges include dealing with failures, coordinating dependencies, handling peak processing loads, optimizing throughput and lowering latency to near real-time.
When should you choose batch over stream processing?
Batch excels for workloads involving:
- Large structured datasets
- Complex ETL and transformations
- High throughput requirements
- Schedulable and repeatable jobs
References:
- [Book] Big Data Processing with Apache Spark by Packt Publishing
- [Article] Tackling the widespread and critical impact of batch effects in high-throughput data
- [Post] General-purpose Stream Joins via Pruning Symmetric Hash Joins
- [Post] Getting Started with Batch Processing : Key Concepts and Fundamentals
- [Post] Big Data Battle : Batch Processing vs Stream Processing
