Legacy Agent Migration (LAM) Fluentd E2E Validation

A framework for validating exact log parity between google-fluentd and the new oss-fluentd-plugin when migrating legacy agents.

Terminology

• Baseline: Logs generated by the legacy agent (google-fluentd).
• Upstream: Logs generated by the new agent (OSS Fluentd + required plugin).

Directory Structure

• scenarios/: YAML templates defining specific log ingestion scenarios and what fields to validate.
• scripts/: Python tools for validating, comparing, reporting, and extracting log outputs.
• outputs/: Ignored directory where generated json mock data, raw logs, and validation reports are saved.

Quick Start

1. Configure and Run a Comparison

To compare a set of exported logs against a scenario schema:

python3 scripts/compare_logs.py \
  --scenario scenarios/golang_slog_json.yaml \
  --baseline outputs/baseline/golang.json \
  --upstream outputs/upstream/golang.json

This script will validate:

• Timestamp presence and InsertId uniqueness.
• Resource Types and Log Names.
• JSON vs Text payload validation.
• Missing or extra logs between agents.
• Deep field-level parity on the defined comparison_keys.

It saves a JSON validation result under outputs/reports/.

2. Generate a Markdown Summary

Once you have generated one or more JSON reports, you can compile them into an easy-to-read markdown file.

python3 scripts/report.py

This produces a outputs/reports/summary.md detailing the pass/fail status and matched discrepancies of all test runs.

3. Export Logs

You can use the exporter to pull logs from GCP, ensuring you do not hardcode your Project IDs or credentials. It relies on your local gcloud CLI authentication.

python3 scripts/export_logs.py \
  --project my-gcp-project-id \
  --log-name ravi-golang-app \
  --freshness 1h \
  --output outputs/baseline/golang.json

4. Direct GCP Memory Comparison

If you want to skip exporting files entirely, you can query and compare GCP logs directly in memory. This is especially useful if your baseline and upstream logs were generated at completely different times, as you can specify exact time boundaries for each:

python3 scripts/compare_gcp_logs.py \
  --project my-gcp-project-id \
  --baseline-log-name ravi-golang-app \
  --upstream-log-name ravi-golang-app-upstream \
  --scenario scenarios/golang_slog_json.yaml \
  --baseline-start "2026-06-04T20:04:28.047931762Z" \
  --baseline-end "2026-06-04T20:04:31.050529758Z" \
  --upstream-start "2026-06-08T21:29:19.747187282Z" \
  --upstream-end "2026-06-08T21:29:22.749277045Z"

Adding New Scenarios

To add a new scenario, create a .yaml file under scenarios/. Use the existing templates as a schema reference. Ensure you don't hardcode any secrets inside the YAML! Keep metadata flexible.