TL;DR: Module 15 covered how events get durably captured and moved around (queues, logs, Kafka). This module covers what happens to that data next: large-scale batch processing (MapReduce, Spark) for bounded datasets processed on a schedule, and stream processing (Flink, Kafka Streams) for unbounded data processed continuously as it arrives — plus two patterns that connect storage to this processing world: Change Data Capture (finally formalized, after being referenced since Module 05) and Event Sourcing, an architectural pattern that takes Module 15's event log and makes it the actual source of truth, not just a transport mechanism.
- Batch processing: operates on a bounded, finite dataset known in advance (e.g., "all of yesterday's log files," "the entire orders table as of midnight") — runs to completion, produces a result, and stops. Optimizes for throughput over latency: it's fine for a batch job to take hours, as long as it correctly processes an enormous volume of data efficiently.
- Stream processing: operates on unbounded, continuously-arriving data (Module 15's event logs/topics) — there's no defined "end," and results are produced continuously and incrementally as new data arrives. Optimizes for latency over raw throughput: results should reflect recent events within seconds or less, not after an overnight batch run.
This is, again, Module 00's latency-vs-throughput trade-off, now determining an entire category of system architecture rather than a single tuning knob — and it's why real systems increasingly need both: a batch layer for comprehensive, accurate historical analysis, and a stream layer for near-real-time freshness, a tension formalized in Section 4.
MapReduce (Google's original paper, later popularized by open-source Hadoop) processes huge datasets across many machines via three phases:
- Map: apply a function independently to each record/chunk of input data, in parallel across many machines, producing intermediate key-value pairs. Since each map task operates independently on its own slice of data, this step is embarrassingly parallel — no coordination needed between map tasks.
- Shuffle: intermediate results are grouped by key and redistributed across the cluster, so all values for a given key end up together, on the machine that will handle that key's reduce step — this is the expensive, network-and-disk-intensive phase, and much of MapReduce's historical performance tuning effort centered on minimizing shuffle cost.
- Reduce: for each key, aggregate/combine all its associated values into a final result (e.g., summing counts, per Module 00's classic word-count example).
- Data locality: MapReduce famously schedules map tasks to run on (or near) the machine that already physically holds the relevant data chunk (typically stored in a distributed file system like HDFS, Module 36's precursor concept), rather than moving huge volumes of data across the network to a compute node — minimizing network I/O by moving computation to the data instead of the reverse, a principle that reappears throughout large-scale data systems.
- Fault tolerance via re-execution: if a map or reduce task fails partway (a machine crashes), MapReduce simply re-runs that specific task on a different machine, rather than needing any complex recovery protocol — this works cleanly because map/reduce tasks are (by design) meant to be deterministic and side-effect-free given the same input, so re-running one is always safe and produces the identical result.
⚠️ Trap: MapReduce's fault-tolerance-via-re-execution strategy quietly depends on tasks being idempotent/deterministic — this is exactly Module 00 and Module 13's idempotency principle again, now as a load-bearing assumption of an entire distributed processing paradigm, not just an API design nicety.
Apache Spark improved substantially on MapReduce's central weakness: MapReduce writes intermediate results to disk between every stage (map output goes to disk before shuffle, shuffle output goes to disk before reduce) — safe and simple, but slow for any multi-step pipeline (Module 01's disk-vs-RAM latency gap, paid repeatedly at every stage boundary).
- Spark instead keeps intermediate data in memory across stages wherever possible (using an abstraction called RDDs — Resilient Distributed Datasets, or the higher-level DataFrame API), only spilling to disk when memory genuinely runs out — dramatically speeding up iterative or multi-stage pipelines (common in ML training pipelines and complex ETL) that would otherwise pay disk I/O at every single step under classic MapReduce.
- Spark builds a DAG (Directed Acyclic Graph) of the full computation before executing it, allowing it to optimize the entire pipeline holistically (e.g., combining/reordering operations, deciding what to cache in memory) rather than executing each MapReduce job as an isolated, disconnected step.
- Fault tolerance is still achieved similarly to MapReduce's philosophy — Spark tracks the lineage (the sequence of transformations that produced a given RDD partition) so that if a partition is lost (a machine crashes), Spark can recompute just that lost partition by re-running its recorded lineage, rather than needing to replicate every intermediate result for safety (a lighter-weight, computation-based alternative to Module 09's data-replication-based fault tolerance).
📐 Numbers: For a multi-stage pipeline with, say, 5 sequential transformation steps, classic MapReduce (writing to disk between every step, at roughly Module 01's ~ms-scale disk write cost) versus Spark (keeping data in RAM between steps, at roughly Module 01's ~100ns-scale memory access cost) illustrates why Spark can be reported as running iterative workloads many times faster in practice — the gap is a direct, compounding instance of Module 01's memory hierarchy, paid once per stage boundary rather than once total.
Stream processing frameworks (Apache Flink, Kafka Streams) continuously consume from Module 15's event logs/topics and compute results incrementally — but unbounded, continuous data introduces problems batch processing never has to face.
You can't compute "the total count" over a stream that never ends — so stream processing groups events into windows:
- Tumbling windows: fixed-size, non-overlapping time buckets (e.g., "every 5-minute interval") — each event belongs to exactly one window.
- Sliding windows: fixed-size, but overlapping (e.g., "the last 5 minutes, recomputed every 1 minute") — an event can belong to multiple overlapping windows.
- Session windows: dynamically sized based on actual activity gaps (e.g., "group events into a session, closing the session after 30 minutes of inactivity") — window boundaries depend on the data itself, not a fixed clock schedule.
This is the single most important, and most commonly mishandled, concept in stream processing:
- Event time: when the event actually occurred, in the real world (e.g., a timestamp embedded in the event payload itself, generated by the device/service that produced it).
- Processing time: when the stream processing system actually receives and processes the event.
These are not the same, and the gap between them can be significant and variable — a mobile device might buffer events while offline and send a batch of hours-old events all at once; network delays (Module 02) can arbitrarily delay individual events; Module 15's consumer lag can mean a whole partition's worth of events arrive "late" relative to when they actually happened. Windowing based on processing time is simple but produces incorrect results relative to what actually happened in the real world whenever events arrive out of order or delayed — the correct approach for most genuine business logic is to window based on event time, using the timestamp embedded in the event itself, not when the system happened to see it.
If you window by event time, you face an unavoidable question: when can you consider a window "done" and emit its result, given that a late-arriving event for that window might still show up? A watermark is the stream processor's explicit, tunable declaration: "I believe no more events with an event-time earlier than this watermark value will arrive" — once the watermark passes a window's end time, the window is closed and its result is emitted, accepting the small risk that a genuinely later-arriving straggler event will be dropped or handled via a separate "late data" path.
⚠️ Trap: This is Module 12's FLP impossibility problem in a new costume: you cannot know with absolute certainty that no more events for a given time window will ever arrive (a device could be offline for days and then reconnect) — a watermark is fundamentally a practical, probabilistic estimate, trading off "wait longer for stragglers" (more complete/accurate results, higher latency) against "close windows sooner" (lower latency, risk of incomplete results) — exactly Module 11's PACELC latency-vs-correctness trade-off, now expressed as a stream-processing-specific tuning parameter.
🤖 AI/ML Callout: Real-time feature computation for ML serving (Module 24, Module 25) is one of the most direct practical applications of everything in Section 3: a feature like "number of clicks in the last 10 minutes" is a sliding window aggregation over event time, and getting the event-time-vs-processing-time distinction wrong (e.g., accidentally windowing by processing time) can silently produce incorrect feature values whenever there's any ingestion delay or out-of-order arrival — a subtle, hard-to-detect source of exactly the training/serving skew problem flagged repeatedly since Module 05, since training data is typically computed correctly using true historical event time, while a naively-implemented real-time feature pipeline can easily and silently use processing time instead.
- Lambda architecture: run both a batch layer (comprehensive, accurate, but high-latency — reprocesses everything periodically) and a separate stream layer (fast, approximate, low-latency) in parallel, merging their outputs at query time — the batch layer's periodic, authoritative recomputation eventually "corrects" anything the faster-but-less-rigorous stream layer got approximately right in the moment. The clear downside: maintaining two separate codebases/ pipelines implementing conceptually the same logic, which is a real, ongoing engineering and consistency-risk cost.
- Kappa architecture: a simpler, more modern alternative — treat everything as a stream, including reprocessing historical data (by simply replaying the event log, Module 15's event log replay capability, from the beginning) rather than maintaining a wholly separate batch pipeline. This directly depends on Module 15's Kafka-style log retention and replay capability — Kappa architecture is essentially "Lambda, simplified, once your event log itself is durable and replayable enough to serve as the single source for both real-time and full-historical reprocessing."
🎤 Interview Angle: "How do you keep a real-time dashboard accurate while also having correct historical reports?" is a Lambda-vs-Kappa question. A strong answer names the specific trade-off (Lambda's dual- codebase maintenance burden vs. Kappa's dependency on the event log being genuinely durable/replayable enough to substitute for a separate batch layer) rather than treating "we'll do both batch and streaming" as a self-evidently correct default without engaging with this cost.
CDC has been referenced since Module 05 (denormalization sync) and used directly in Module 13's Outbox pattern — this section gives it a proper, complete treatment.
CDC captures every change (insert/update/delete) made to a database and publishes it as a stream of events, so other systems can react to data changes without the source database needing to know or care who's listening.
The best CDC implementations (e.g., Debezium) don't add application-level "publish an event on every write" code at all — instead, they read the database's own Write-Ahead Log (Module 04's WAL, yet again) directly, extracting the same change records the database already produces internally for its own crash-recovery and replication purposes (Module 09's WAL-shipping replication is, in fact, structurally very similar to what log-based CDC does, just delivering the extracted changes to external consumers instead of only to database replicas).
- Why this is superior to application-level "publish an event on every write": it requires zero changes to existing application code, and it cannot miss a write — since it reads the database's own authoritative log of everything that happened, rather than depending on every code path that writes to the database to remember to also publish an event (a common, easy-to-miss source of the dual-write problem from Module 13, which log-based CDC sidesteps structurally rather than requiring application discipline).
- Module 08 (Caching): CDC-driven cache invalidation — a change to the source database triggers an event that invalidates/updates affected cache entries, decoupled from the application's write path.
- Module 13 (Outbox pattern): the background process reading unpublished outbox rows is very often implemented as log-based CDC watching the outbox table specifically.
- Search index synchronization: keeping a search index (e.g., Elasticsearch) up to date with a primary database's changes, without the application needing to remember to update both.
- Feeding a stream processing pipeline (this module): CDC is a very common source of events for Flink/Kafka Streams pipelines — turning "changes to a database" into "a stream that can be windowed, aggregated, and joined with other streams."
Event Sourcing is an architectural pattern (distinct from, though
related to, CDC and Module 15's event logs) where, instead of storing
only the current state of an entity (the normal approach — a row
representing "this order's current status"), the system stores the
complete, ordered sequence of events that led to that state (e.g.,
OrderCreated, ItemAdded, PaymentReceived, OrderShipped) as the
authoritative source of truth — current state is always a derived
view, computed by replaying (folding over) the event history, not stored
independently as the primary record.
- CDC captures changes to an already-existing, state-based system's data, as a side effect/export of normal database operations — the database's current-state table is still the primary source of truth; CDC is a derived, secondary stream.
- Event Sourcing makes the event sequence itself the primary, authoritative data — the current-state view (if one exists at all, often materialized separately for efficient querying) is the derived artifact, not the source of truth. This is a fundamentally different design decision made at the application/domain modeling level, not just an infrastructure choice layered on top of an existing schema.
- Benefits: a complete, perfect audit trail for free (every change that ever happened is preserved, by construction); the ability to reconstruct state as of any past point in time by replaying events up to that point; new "read models" (materialized, denormalized views optimized for specific queries — directly connecting to Module 05's denormalization discussion) can be added later by simply replaying the full event history through new logic, without having ever needed to anticipate that query pattern when the events were originally recorded.
- Costs: querying "current state" directly is no longer trivial — it requires either replaying events on every read (expensive at scale) or maintaining separate, continuously-updated materialized views (the CQRS — Command Query Responsibility Segregation pattern, commonly paired with Event Sourcing, where writes go through the event log and reads are served from a separately maintained, query-optimized projection); schema evolution of event formats over time is a genuinely hard, ongoing engineering concern (old events, recorded years ago in an old format, must still be replayable and interpretable by current logic).
🤖 AI/ML Callout: Event Sourcing's "current state is a derived, replayable view over a durable event history" principle is architecturally very close to how a well-designed feature store's offline/training data layer should work (Module 24): rather than only keeping the current value of each feature, retaining the full historical event stream that produced those feature values is exactly what enables correct point-in-time-correct feature retrieval for model training (Module 06's MVCC-snapshot analogy, revisited) — you can only correctly answer "what did this feature look like at this past training example's timestamp" if the underlying system was designed, Event-Sourcing-style, to retain the full history rather than only ever overwriting a single current value in place.
- Batch (bounded data, throughput-optimized, e.g., MapReduce/Spark) vs. stream (unbounded data, latency-optimized, e.g., Flink/Kafka Streams) — Module 00's latency-vs-throughput trade-off, now architecture-scale.
- MapReduce: map (parallel, independent) → shuffle (expensive, network/disk-heavy) → reduce (aggregate per key). Data locality (compute moves to data) and fault tolerance via re-running deterministic tasks are its two foundational design principles.
- Spark: keeps intermediate data in memory across pipeline stages (vs. MapReduce's disk-write-per-stage), tracks lineage for recompute-based fault tolerance instead of data replication — dramatic speedup for multi-stage/iterative pipelines.
- Windowing (tumbling/sliding/session) makes aggregation over unbounded streams meaningful. Event time (when it really happened) vs. processing time (when the system saw it) is the most commonly mishandled distinction — window by event time for correctness.
- Watermarks: a practical, tunable estimate of "no more late events are coming" for a window — an FLP-flavored impossibility (can't know for certain) resolved via a PACELC-style latency-vs-completeness trade-off.
- Lambda architecture: separate batch + stream layers merged at query time (accurate but dual-codebase maintenance cost). Kappa architecture: everything as a stream, reprocessing via full log replay (Module 15) — simpler, but depends on the log being durable/replayable enough to substitute for batch entirely.
- CDC: capture DB changes as an event stream. Log-based CDC (reading the WAL directly, e.g., Debezium) is superior to application-level "publish on write" — zero app code changes, structurally can't miss a write. Powers cache invalidation, the Outbox pattern's publish step, search index sync, and stream pipeline ingestion.
- Event Sourcing: the event sequence is the source of truth; current state is a derived, replayable view — genuinely different from CDC (which exports changes from an already-state-based system). Enables perfect audit trail and point-in-time reconstruction, at the cost of needing materialized read models (CQRS) for efficient current-state queries and hard long-term event-schema evolution.
- Explain why MapReduce's fault tolerance strategy (simply re-running a failed task) depends on a specific property of map/reduce tasks — name that property and connect it to a concept from an earlier module.
- Explain precisely why Spark is faster than classic MapReduce for a multi-stage pipeline, tying your answer to Module 01's memory hierarchy.
- A stream processing job computes "average order value per 5-minute window" using the timestamp of when each order event is received by the server, rather than when the order was actually placed. Explain a concrete scenario where this produces an incorrect result, and name the fix.
- Explain what a watermark actually is, and why it's fundamentally a trade-off rather than a way to achieve perfect correctness — connect this to a specific concept from Module 12.
- Contrast Lambda and Kappa architecture in terms of what each requires you to maintain, and explain specifically what property of Module 15's event logs makes Kappa architecture viable at all.
- Explain why log-based CDC (reading the database's WAL directly) is structurally immune to the dual-write problem that Module 13's Outbox pattern was designed to solve for application-level "publish an event on write" code.
- A colleague says "Event Sourcing is basically the same thing as CDC, just with more steps." Explain precisely what's wrong with this statement — what's actually different about which data is considered the source of truth in each?
Answers — click to expand
-
MapReduce's "just re-run the failed task on another machine" strategy depends on map and reduce tasks being deterministic and side-effect-free — given the same input, a task must always produce exactly the same output, with no external state mutated as a side effect that a second run would duplicate or conflict with. This is precisely idempotency, the same property first introduced in Module 00 (and central to Module 13's distributed transaction handling) — if a task's output depended on something other than its input (e.g., incrementing an external counter as a side effect, or depending on wall-clock time), re-running it after a failure could produce a different or duplicated result, corrupting the overall job's correctness. MapReduce's entire fault-tolerance model is only safe because tasks are required, by design, to be idempotent with respect to re-execution.
-
Classic MapReduce writes intermediate results to disk between every stage of a multi-step pipeline (map output to disk before shuffle, shuffle output to disk before reduce) — each of these writes/reads pays Module 01's disk-access cost (roughly millisecond-scale for SSD, worse for HDD) at every single stage boundary. Spark instead keeps intermediate data in memory (RAM, roughly ~100ns access, Module 01's memory hierarchy) across stages wherever it fits, only spilling to disk when memory is genuinely exhausted — so for a pipeline with several sequential stages, Spark avoids paying the (many-orders-of- magnitude-larger) disk access cost repeatedly at each boundary, which compounds into a dramatic overall speedup for exactly the kind of multi-stage, iterative pipelines (common in complex ETL and ML training workflows) where this cost would otherwise be paid over and over.
-
Concrete scenario: a mobile app buffers order events locally while the device is offline, then sends a batch of several hours' worth of orders all at once when connectivity is restored. If the stream processor windows by processing time (when the server received each event), all of those genuinely-hours-old orders get counted together in whatever 5-minute window happens to be active at the moment they finally arrive — inflating that window's average order value with orders that didn't actually occur during that real-world time period at all, while the windows corresponding to when those orders actually happened show an artificially low (or missing) count, since those orders hadn't arrived yet when those windows were originally computed. The fix: window by event time — the timestamp embedded in the order event itself, reflecting when the order was actually placed — so each order is correctly attributed to the 5-minute window it genuinely occurred in, regardless of when the server happened to receive it.
-
A watermark is the stream processor's explicit declaration of "I believe no more events with an event-time earlier than this value will arrive," used to decide when a given event-time window can be considered complete and its result emitted. It's fundamentally a trade-off, not a path to perfect correctness, because there is no way to know with absolute certainty that a straggler event won't still arrive later (a device could remain offline for an arbitrarily long time and then send an old event) — this mirrors Module 12's FLP impossibility point exactly: you cannot achieve certainty about "nothing more is coming" over an asynchronous system where delays are unbounded, so a watermark is necessarily a practical, tunable estimate rather than a guarantee, trading "wait longer for more completeness" against "close the window sooner for lower latency" — the same correctness-vs-latency trade-off shape Module 11 formalized as PACELC, now expressed as a specific, concrete stream-processing configuration parameter rather than an abstract system property.
-
Lambda architecture requires maintaining two separate pipelines/codebases implementing conceptually the same processing logic — a batch layer (for comprehensive, accurate, periodically recomputed results) and a stream layer (for fast, approximate, low-latency results) — with the ongoing engineering burden and consistency risk of keeping both correctly aligned in their logic over time. Kappa architecture requires maintaining only one pipeline (a stream processing pipeline), treating historical reprocessing as simply replaying the event log from the beginning through that same single pipeline, rather than needing a wholly separate batch codebase. What makes Kappa viable at all is specifically Module 15's event log retention and replay capability — Kafka-style logs retain messages durably and let any consumer replay from any point (including the very beginning), which is exactly the property needed to substitute "replay the log through the stream pipeline" for what would otherwise require a dedicated batch pipeline reading from a separate, comprehensive historical data store.
-
Log-based CDC reads directly from the database's own Write-Ahead Log — the same authoritative, complete record of every change the database itself already produces internally, for its own crash recovery and replication purposes (Module 04, Module 09). Because it reads from this single, guaranteed-complete source, it is structurally impossible for it to "miss" a write — every write that successfully commits to the database, by definition, has already been recorded in the WAL, so the CDC process reading that WAL will eventually see and capture it. This is fundamentally different from application-level "publish an event on write" code, where the dual-write problem (Module 13) arises because there are two separate operations (write to the database, and separately, call code to publish an event) that can fail independently of each other (a crash between them, or simply a code path that forgot to include the publish call at all) — log-based CDC eliminates this risk entirely by not depending on any second, separate application-level action ever happening correctly; it just reads what the database already, unconditionally, records for its own purposes.
-
The statement is wrong because it conflates which data is the actual source of truth in each pattern, which is the fundamental distinguishing design decision, not an implementation detail. In CDC, the database's current-state representation is still the primary source of truth — CDC is simply a derived, secondary export mechanism that captures and republishes changes to that already-authoritative state, for other systems to consume; if you deleted the CDC pipeline entirely, the actual data (and its correctness) would be completely unaffected, since it lives independently in the database's own tables. In Event Sourcing, the sequence of events itself is the primary, authoritative source of truth — there may be no independently-stored "current state" table at all beyond a derived, replayable materialized view built from the event sequence; if you deleted a materialized current-state view built on top of Event Sourcing, you could always perfectly reconstruct it by replaying the event history, because that history was always the real record, not a byproduct. This is a difference in fundamental application/domain design (what does the system actually consider "true" and irreplaceable), not merely "more steps" added on top of the same underlying idea.
Module 17 — Rate Limiting, Quotas & Admission Control shifts from processing data already inside the system to protecting the system's edges — deciding how much incoming traffic to accept in the first place, directly connecting to this module's backpressure discussion (Module 15) and setting up Module 18's broader resiliency patterns.