Stream Graph¶
A stream graph is the directed operator topology that turns source records into results; its edges determine partitioning, serialization, and failure boundaries as much as its nodes determine business logic.
flowchart LR
J[Junior: pipeline as a graph] --> M[Middle: operators and partitions]
M --> S[Senior: chaining and rescaling] --> P[Professional: runtime execution]
flowchart LR
S[Kafka source] --> P[Parse]
P --> K[keyBy account_id]
K --> W[Window aggregate]
W --> O[Lakehouse sink]
K -.network shuffle.-> W
Choose a level¶
| Level | Guide | You are done when |
|---|---|---|
| Junior | From steps to a graph | You can identify sources, transformations, and sinks in a streaming DAG. |
| Middle | Operators and partitioning | You can explain operator parallelism, chaining, and shuffles. |
| Senior | Safe topology evolution | You can reason about skew, rescaling, and state compatibility. |
| Professional | Execution internals | You can compare Flink, Kafka Streams, and Beam execution models. |
Practice rule¶
Draw every repartition edge. A hidden shuffle is often the largest latency, network, state-movement, and recovery cost in the graph.