Switch pod-scheduler configs from JSON to YAML; add JSON Schema

README.md

@@ -19,6 +19,6 @@ Microsoft employees can approve and merge contributions by joining [this team](h
 # Continuous benchmarking pipelines
 
 The Azure DevOps pipelines that drive the daily benchmark sweep are generated
-from JSON configs by the [pod-scheduler](scripts/pod-scheduler/README.md).
+from YAML configs by the [pod-scheduler](scripts/pod-scheduler/README.md).
 See [`build/README.md`](build/README.md) for the mapping between configs and
 generated YAML files, and how to regenerate them after editing.

build/README.md

@@ -7,35 +7,41 @@ configs that generate them.
 
 The CI pipelines that run the daily benchmark sweep are generated by the
 [pod-scheduler](../scripts/pod-scheduler/README.md). Each pipeline maps to a
-JSON config that lists the available pods (machine groups) and scenarios.
+YAML config that lists the available pods (machine groups) and scenarios.
 
 | Generated YAML | Source config | Notes |
 |---|---|---|
-| `benchmarks-ci-01.yml`, `benchmarks-ci-02.yml` | `benchmarks_ci_pods.json` | On-prem fleet (citrine + mono); split across two YAMLs to spread load |
-| `benchmarks-ci-azure.yml` | `benchmarks_ci_azure_pods.json` | Azure cloud pods (includes the merged eastus2 cobalt cloud pods) |
-| `benchmarks-ci-cobalt.yml` | `benchmarks_ci_cobalt_pods.json` | Cobalt-hosted pods |
+| `benchmarks-ci-01.yml`, `benchmarks-ci-02.yml` | `benchmarks_ci_pods.yml` | On-prem fleet (citrine + mono); split across two YAMLs to spread load |
+| `benchmarks-ci-azure.yml` | `benchmarks_ci_azure_pods.yml` | Azure cloud pods (includes the merged eastus2 cobalt cloud pods) |
+| `benchmarks-ci-cobalt.yml` | `benchmarks_ci_cobalt_pods.yml` | Cobalt-hosted pods |
 
 To regenerate, from the repo root:
 
 ```bash
 python ./scripts/pod-scheduler/main.py \
-    --config ./build/benchmarks_ci_pods.json \
+    --config ./build/benchmarks_ci_pods.yml \
     --yaml-output ./build
 
 python ./scripts/pod-scheduler/main.py \
-    --config ./build/benchmarks_ci_azure_pods.json \
+    --config ./build/benchmarks_ci_azure_pods.yml \
     --base-name benchmarks-ci-azure \
     --yaml-output ./build
 
 python ./scripts/pod-scheduler/main.py \
-    --config ./build/benchmarks_ci_cobalt_pods.json \
+    --config ./build/benchmarks_ci_cobalt_pods.yml \
     --base-name benchmarks-ci-cobalt \
     --yaml-output ./build
 ```
 
 The header of each generated YAML embeds the exact regen command for that
 file.
 
+The configs ship with a `# yaml-language-server: $schema=...` directive at
+the top so VS Code / Cursor (with the Red Hat YAML extension or the
+`yaml-language-server` LSP) provide autocomplete, hover docs, and inline
+validation while editing. The schema lives at
+[`scripts/pod-scheduler/pod-config.schema.json`](../scripts/pod-scheduler/pod-config.schema.json).
+
 ## Hand-maintained scenario templates
 
 Everything else in this directory is hand-maintained. The most important
@@ -44,28 +50,32 @@ groups:
 - `*-scenarios.yml` — AzDO job templates that the generated pipelines invoke
   (e.g. `proxies-scenarios.yml`, `grpc-scenarios.yml`). One per benchmark
   family. The pod-scheduler picks which template to run via the
-  `template` field on each scenario in the JSON config.
+  `template` field on each scenario in the config.
 - `prbenchmarks.*.config.yml` — PR-trigger pipelines for individual PR runs.
 - `job-variables.yml` — shared variable definitions consumed by the scenario
   templates.
 
 When adding a new benchmark family, drop a `*-scenarios.yml` template in this
-directory and reference it from the relevant `benchmarks_ci_*_pods.json`.
+directory and reference it from the relevant `benchmarks_ci_*_pods.yml`.
 
 ## Editing the schedule
 
-Common edits, all done in JSON:
+Common edits, all done in the `benchmarks_ci_*_pods.yml` configs:
 
-- **Add a scenario**: append to `scenarios[]` in the appropriate
-  `benchmarks_ci_*_pods.json`.
-- **Add a pod / machine class**: append to `pods[]` and reference its name
+- **Add a scenario**: append to `scenarios:` in the appropriate config.
+- **Add a pod / machine class**: append to `pods:` and reference its name
   from any scenario you want to target it.
 - **Change cadence or splits**: edit `metadata.schedule` (cron),
   `metadata.yaml_generation.target_yaml_count`, or
   `metadata.yaml_generation.schedule_offset_hours`.
-- **Override a job's timeout**: add `"timeout": <minutes>` to the scenario.
+- **Override a job's timeout**: add `timeout: <minutes>` to the scenario.
 
-After any edit, regenerate the YAML and commit both the JSON and the YAML
-together. The snapshot tests in
+After any edit, regenerate the YAML and commit both the config and the
+generated YAML together. The snapshot tests in
 `scripts/pod-scheduler/tests/test_snapshots.py` will fail in CI if the
 generated YAML drifts from the configs.
+
+> Legacy `.json` configs are still accepted by the loader for back-compat,
+> but new edits should use the `.yml` form so configs can carry comments and
+> stay in sync with the schema-driven editor tooling.
+

build/benchmarks-ci-01.yml

@@ -1,7 +1,7 @@
 # Do not change this file, it is generated by the pod-scheduler.
 # Source of truth: see ../scripts/pod-scheduler/README.md
 # To regenerate, run from the repo root:
-#   python ./scripts/pod-scheduler/main.py --config ./build/benchmarks_ci_pods.json --base-name benchmarks-ci --yaml-output ./build
+#   python ./scripts/pod-scheduler/main.py --config ./build/benchmarks_ci_pods.yml --base-name benchmarks-ci --yaml-output ./build
 
 trigger: none
 pr: none

build/benchmarks-ci-02.yml

@@ -1,7 +1,7 @@
 # Do not change this file, it is generated by the pod-scheduler.
 # Source of truth: see ../scripts/pod-scheduler/README.md
 # To regenerate, run from the repo root:
-#   python ./scripts/pod-scheduler/main.py --config ./build/benchmarks_ci_pods.json --base-name benchmarks-ci --yaml-output ./build
+#   python ./scripts/pod-scheduler/main.py --config ./build/benchmarks_ci_pods.yml --base-name benchmarks-ci --yaml-output ./build
 
 trigger: none
 pr: none

build/benchmarks-ci-azure.yml

@@ -1,7 +1,7 @@
 # Do not change this file, it is generated by the pod-scheduler.
 # Source of truth: see ../scripts/pod-scheduler/README.md
 # To regenerate, run from the repo root:
-#   python ./scripts/pod-scheduler/main.py --config ./build/benchmarks_ci_azure_pods.json --base-name benchmarks-ci-azure --yaml-output ./build
+#   python ./scripts/pod-scheduler/main.py --config ./build/benchmarks_ci_azure_pods.yml --base-name benchmarks-ci-azure --yaml-output ./build
 
 trigger: none
 pr: none

build/benchmarks-ci-cobalt.yml

@@ -1,7 +1,7 @@
 # Do not change this file, it is generated by the pod-scheduler.
 # Source of truth: see ../scripts/pod-scheduler/README.md
 # To regenerate, run from the repo root:
-#   python ./scripts/pod-scheduler/main.py --config ./build/benchmarks_ci_cobalt_pods.json --base-name benchmarks-ci-cobalt --yaml-output ./build
+#   python ./scripts/pod-scheduler/main.py --config ./build/benchmarks_ci_cobalt_pods.yml --base-name benchmarks-ci-cobalt --yaml-output ./build
 
 trigger: none
 pr: none

build/benchmarks_ci_azure_pods.yml

@@ -0,0 +1,104 @@
+# yaml-language-server: $schema=../scripts/pod-scheduler/pod-config.schema.json
+
+# Azure CI pod-scheduler config.
+#
+# Covers Azure cloud pods plus the merged eastus2 cobalt-cloud pods (folded
+# in here when their dedicated pipeline was retired). Generates a single
+# ./benchmarks-ci-azure.yml.
+#
+# Edit this file then regenerate from the repo root:
+#   python ./scripts/pod-scheduler/main.py \
+#       --config ./build/benchmarks_ci_azure_pods.yml \
+#       --base-name benchmarks-ci-azure --yaml-output ./build
+metadata:
+  name: Azure CI Benchmarks Configuration
+  description: Pod-based scheduling for Azure CI benchmarks (includes cobalt cloud from eastus2)
+  version: "2.1"
+  schedule: "0 2/12 * * *"
+  queues:
+    - azure
+    - azurearm64
+  yaml_generation:
+    target_yaml_count: 1
+    schedule_offset_hours: 0
+
+pods:
+  - name: azure-arm64
+    machines: { sut: azure-server-arm64,  load: azure-client,  db: azure-db }
+    profiles: { sut: azure-server-arm64-app, load: azure-client-load, db: azure-db-db }
+
+  - name: azure2-amd64
+    machines: { sut: azure2-server-amd64, load: azure2-client, db: azure2-db }
+    profiles: { sut: azure2-server-amd64-app, load: azure2-client-load, db: azure2-db-db }
+
+  # eastus2 cobalt-cloud pods, merged in when the dedicated pipeline was retired.
+  - name: cobalt-cloud-lin
+    machines: { sut: cobalt-cloud-lin-server, load: cobalt-cloud-lin-client, db: cobalt-cloud-lin-db }
+    profiles: { sut: cobalt-cloud-lin-server-app, load: cobalt-cloud-lin-client-load, db: cobalt-cloud-lin-db-db }
+
+  - name: cobalt-cloud-lin-azl3
+    # Same load + db machines as cobalt-cloud-lin but a different SUT image
+    # (Azure Linux 3). Cannot share a stage with cobalt-cloud-lin triples.
+    machines: { sut: cobalt-cloud-lin-server-azl3, load: cobalt-cloud-lin-client, db: cobalt-cloud-lin-db }
+    profiles: { sut: cobalt-cloud-lin-server-azure-linux3-app, load: cobalt-cloud-lin-client-load, db: cobalt-cloud-lin-db-db }
+
+  - name: cobalt-cloud-lin-azl3-dual
+    # Dual-only variant: re-uses the db box as the load generator so we can
+    # run dual scenarios without needing a separate load machine.
+    machines: { sut: cobalt-cloud-lin-server-azl3, load: cobalt-cloud-lin-db }
+    profiles: { sut: cobalt-cloud-lin-server-azure-linux3-app, load: cobalt-cloud-lin-db-load }
+
+  - name: idna-amd-lin
+    # idna-amd-lin and idna-intel-lin are mutually each other's load box.
+    machines: { sut: idna-amd-lin, load: idna-intel-lin }
+    profiles: { sut: idna-amd-lin-app, load: idna-intel-lin-load }
+
+  - name: idna-amd-win
+    # Borrows azure2-db as the load box for cross-cluster idna jobs.
+    machines: { sut: idna-amd-win, load: azure2-db }
+    profiles: { sut: idna-amd-win-app, load: azure2-db-load }
+
+  - name: idna-intel-lin
+    machines: { sut: idna-intel-lin, load: idna-amd-lin }
+    profiles: { sut: idna-intel-lin-app, load: idna-amd-lin-load }
+
+  - name: idna-intel-win
+    machines: { sut: idna-intel-win, load: azure2-db }
+    profiles: { sut: idna-intel-win-app, load: azure2-db-load }
+
+scenarios:
+  - name: Containers
+    template: containers-scenarios.yml
+    type: triple
+    pods: [azure-arm64, azure2-amd64, cobalt-cloud-lin, cobalt-cloud-lin-azl3]
+    estimated_runtime: 90
+
+  - name: Baselines Database
+    template: baselines-database-scenarios.yml
+    type: triple
+    pods: [azure-arm64, azure2-amd64, cobalt-cloud-lin, cobalt-cloud-lin-azl3]
+    estimated_runtime: 45
+
+  - name: Baselines
+    template: baselines-scenarios.yml
+    type: dual
+    pods: [azure-arm64, azure2-amd64, cobalt-cloud-lin, cobalt-cloud-lin-azl3-dual]
+    estimated_runtime: 30
+
+  - name: MVC
+    template: mvc-scenarios.yml
+    type: triple
+    pods: [cobalt-cloud-lin, cobalt-cloud-lin-azl3]
+    estimated_runtime: 20
+
+  - name: Trends
+    template: trend-scenarios.yml
+    type: dual
+    pods: [azure-arm64, azure2-amd64, cobalt-cloud-lin, cobalt-cloud-lin-azl3-dual, idna-amd-lin, idna-amd-win, idna-intel-lin, idna-intel-win]
+    estimated_runtime: 20
+
+  - name: Trends Database
+    template: trend-database-scenarios.yml
+    type: triple
+    pods: [azure-arm64, azure2-amd64, cobalt-cloud-lin, cobalt-cloud-lin-azl3]
+    estimated_runtime: 15