Skip to content

README.md

Latest commit

 

History

History
140 lines (109 loc) · 10.2 KB

README.md

File metadata and controls

140 lines (109 loc) · 10.2 KB
Raw

Kates — The Definitive Guide

Kafka Advanced Testing & Engineering Suite

A comprehensive guide to performance testing, chaos engineering, and operational resilience for Apache Kafka — from theory to practice.

Table of Contents

Reading order is defined by _quarto.yml — filenames are stable identifiers, not ordering. Chapter numbers are assigned automatically by the rendered book. Each Part opens with a Part page that says what the Part is for and lists its chapters; the appendices are introduced at the end of the Part VI page.

Part I — Foundations

Title Description
Introduction What Kates is, why it exists, and the problems it solves
Architecture & Design Platform architecture, component design, data model, and technology choices
The Cluster Under Test Understanding krafter, the Kafka cluster under test, and its node layout

Part II — Performance Testing

Title Description
Performance Theory Measuring performance: latency, throughput, percentiles, and statistics
Test Types Deep Dive Every test type explained with methodology and use cases
Scenario Files & SLA Gates YAML scenario format, spec fields, and automated SLA enforcement
Lab — Interactive Performance Tuning The interactive TUI workbench for iterative tuning and result comparison

Part III — Chaos & Integrity

Title Description
Chaos Engineering Theory Principles, practices, and the Game Day methodology
Chaos Engineering in Practice Disruption types, playbooks, safety guardrails, and SLA grading
Data Integrity Verification Ensuring zero message loss under fault conditions

Part IV — Observability

Title Description
Observability & Monitoring Metrics, dashboards, heatmaps, and trend analysis

Part V — Deployment & Operations

Title Description
Deploying the Strimzi Operator The operator as its own release: the strimzi-operator wrapper chart, its CRD-upgrade hook, and namespace scope
Installing Kafka with the kafka-cluster Helm Chart Step-by-step Kafka deployment with prerequisites and verification
Kafka Deployment Engineering The engineering rationale: Strimzi, KRaft, broker tuning, and operations
Deployment Guide Deploying the Kates stack: topology decisions, sizing, and cloud guidance
Compliance Checks A reference architecture as a file, and checks that hold the live cluster to it
Security & Compliance Authentication, authorization, certificates, network policies, and Kyverno
Multi-Tenancy Topic naming, service onboarding, quotas, and tenant isolation
Upgrade Playbook Step-by-step procedures for upgrading Kafka, Strimzi, and Kates
Kafka Connect & CDC Pipelines Connect concepts: architecture, Debezium CDC, transforms, and delivery semantics
Operating Kafka Connect Day-2 operations: scaling, tuning, security rotation, upgrades, and DR
Cross-Cluster Replication and Migration MirrorMaker 2 topologies for disaster recovery and cluster migration
Migrating a Legacy Kafka Source Moving off a 2.x/3.x cluster onto the current Kafka line
Recipes & Patterns Ready-to-use workflows for upgrades, nightly regressions, and tuning

Part VI — Reference

Title Description
CLI Reference Complete Kates CLI command reference with all subcommands and aliases
REST API Reference The Kates API's REST endpoints and data models
gRPC API Reference Protobuf service definitions, message types, and usage examples

Appendices

Title Description
Glossary Quick reference for all terms and abbreviations
Troubleshooting Index Consolidated troubleshooting procedures from across the book
CI/CD Pipeline GitHub Actions workflows, build validation, and release automation
Version & Compatibility Matrix Every pinned version, generated from the repo's own pins
Compliance Language Reference Every field, value, rule and lint finding of a KafkaReference file
IDE Extensions & Language Server JSON Schemas, offline linters, kates lsp, and the VS Code and Visual Studio 2022 plugins
References The works the book cites, numbered alphabetically by first author

Tutorials

Hands-on step-by-step guides for specific workflows. The tutorials are for practice and the book is for explanation; each Part page links the tutorials that go with its chapters.

# Tutorial Description
1 Getting Started with Kates Deploy the stack, run a first test, and read and export its results
2 Running Every Test Type Run each test type, from flags or from the built-in templates
3 Chaos Engineering with Kates Disruption plans, a built-in playbook, and a resilience run
4 Data Integrity Under Fire Data integrity verification under fault conditions
5 Heatmaps, Trends, and Exports Latency heatmaps, report comparison, trends, and export formats
6 CI/CD Integration Performance, integrity, and chaos gates in a pipeline, and scheduled runs
7 Kyverno & Security Policy enforcement, security auditing, and compliance checks
8 Deploy, Detect & Clean Cluster analysis with kates detect, the interactive deployment wizard, and teardown with kates clean
9 Kafka Connect Working Examples (CDC + JDBC) CDC + JDBC connector setup with Debezium
– Kafka Connect Source/Sink Quick Runbook Minimal source-to-sink runbook
10 Installing and Setting Up MirrorMaker 2 A loopback mirror on krafter, installed and proven to move records
11 Migrating Kafka 2.x to 4.x with MirrorMaker 2 A 2.x cluster migrated onto 4.x, with a rehearsed cutover, on Kind
12 Migrating Kafka 3.x to 4.x with MirrorMaker 2 The same migration from a 3.x cluster
13 Using the Grafana Dashboards Install the Grafana boards, read them, and diagnose an empty panel
14 Using Kates from an AI Agent Connect an AI agent to the lab through the read-only kates mcp server
15 Authoring & Debugging YAML Suites in VS Code Install vscode-kates, use kates lsp quick-fixes and inlay hints, refactor scenarios into chaos tests, and run live cluster pre-flight

Who This Book Is For

  • Platform engineers who need to validate Kafka cluster resilience before production
  • SREs who want automated chaos testing with SLA enforcement
  • Performance engineers who need rigorous benchmarking beyond kafka-producer-perf-test
  • Developers building event-driven systems who want confidence in their Kafka infrastructure

How to Read This Book

Don't read this book cover-to-cover. Pick a reading path based on what you need:

🎯 "I want to understand Kafka performance"

  1. Performance Theory — why averages lie, percentiles, coordinated omission (~10 min)
  2. Test Types Deep Dive — all 8 test types explained (~15 min)
  3. Observability & Monitoring — reading dashboards, heatmaps, trend analysis (~15 min)

🚀 "I want to deploy Kates"

  1. Installing Kafka with the kafka-cluster Helm Chart — full step-by-step with prerequisites (~30 min)
  2. Deployment Guide — architecture decisions, resource sizing, cloud deployment (~15 min)
  3. The Cluster Under Test — understanding what you just deployed (~10 min)

💥 "I want to run chaos experiments"

  1. Chaos Engineering Theory — principles and methodology (~10 min)
  2. Chaos Engineering in Practice — disruption types and playbooks (~20 min)
  3. Data Integrity Verification — proving zero message loss (~10 min)

🔒 "I want to harden security"

  1. Security & Compliance — threat model, ACLs, network policies, Kyverno (~20 min)
  2. Kafka Deployment Engineering — broker security, certificates (~15 min)
  3. Tutorial 7: Kyverno & Security — hands-on policy enforcement (~20 min)

📋 "I just need a reference"