chore: replace gardener promtail with alloy

common/roles/defaults/defaults/main.yaml

@@ -1,10 +1,10 @@
 ---
 metal_registry_auth_enabled: false
 metal_registry_auth_user: ""
 metal_registry_auth_password: ""
 metal_registry_url: "https://index.docker.io/v1/"
 
 metal_stack_release:
   mapping:
     # binaries
     metal_hammer_image_tag: "binaries.metal-stack.metal-hammer.version"
@@ -165,6 +165,8 @@
     metal_helm_chart_tag: "helm-charts.metal-stack.metal-control-plane.tag"
     logging_chart_version: "helm-charts.logging.loki.version"
     logging_chart_repo: "helm-charts.logging.loki.repository"
+    gardener_logging_alloy_chart_version: "helm-charts.logging.alloy.version"
+    gardener_logging_alloy_chart_repo: "helm-charts.logging.alloy.repository"
     logging_alloy_chart_version: "helm-charts.logging.alloy.version"
     logging_alloy_chart_repo: "helm-charts.logging.alloy.repository"
     logging_promtail_chart_version: "helm-charts.logging.promtail.version"

control-plane/roles/gardener-logging/README.md

@@ -1,22 +1,80 @@
 # gardener-logging
 
-This role deploys a promtail into a Gardener shooted seed. It is expected that the [logging role](../logging/) was deployed into the metal-stack control plane before executing this role.
+Deploys [Grafana Alloy](https://grafana.com/docs/alloy/latest/) into Gardener shooted seeds and optionally into the garden cluster itself. Alloy reads pod logs from the node filesystem and forwards them to the Loki instance in the metal-stack control plane.
+
+Expects the [logging role](../logging/) to have been deployed first.
+
+Alloy configuration, labels, meta-monitoring, and migration guidance are documented in [logging-common](../logging-common/).
+
+> **Promtail is deprecated.** Setting `gardener_logging_promtail_enabled: true` emits a deprecation warning on every run. See the [migration guide](../logging-common/README.md#migration-from-promtail).
+
+## Background and Architecture
+
+### Why this role exists
+
+Gardener ships with a built-in logging stack — gardenlet can deploy [Vali](https://github.com/credativ/plutono) (a Loki fork) and fluent-bit into each seed cluster. The metal-stack deployment disables this stack intentionally.
+
+Instead, this role deploys Alloy as a DaemonSet into the Garden cluster and Shooted Seeds to ship **all** pod logs to the central Loki instance in the metal-stack control plane. This gives platform operators a single place to query infrastructure logs across all Gardener clusters — including the Garden cluster itself, seed system components, and extension controllers.
+
+This role targets the **Garden cluster** (`gardener_logging_deploy_to_garden_cluster: true`) and **Shooted Seeds** (`gardener_logging_shooted_seeds`).
+
+### Why there are two task files
+
+- **`tasks/main.yaml`** — handles the Garden cluster. The Garden cluster is a normal Kubernetes cluster reachable with the current kubeconfig. Alloy is deployed directly.
+- **`tasks/gardener-shooted-seed.yaml`** — handles one Shooted Seed per loop iteration. Because Shooted Seeds are managed as Shoots, their kubeconfig is not directly available. The task fetches it via the virtual garden API (`virtual_garden_kubeconfig` + `shoot_admin_kubeconfig` filter), then deploys Alloy using that kubeconfig.
+
+### What gets collected
+
+Alloy runs as a Kubernetes DaemonSet in the `monitoring` namespace. It reads pod logs from the node filesystem (`/var/log/pods`, `loki.source.file`) — one DaemonSet pod per node, collecting only the logs for pods scheduled on that node. Kubernetes events are also collected natively via `loki.source.kubernetes_events` with clustering-based leader election to avoid duplication.
+
+### Prometheus metrics
+
+Alloy can optionally push its own self-metrics to a remote Prometheus endpoint (see [logging-common meta-monitoring](../logging-common/README.md#meta-monitoring)). It does **not** collect metrics from other workloads in the cluster.
+
+Seed workload metrics are Gardener's responsibility: gardenlet manages Cache, Seed, and Aggregate Prometheus instances on each seed. Those Prometheus instances use annotation-based discovery restricted to known namespaces and do not reach the `monitoring` namespace where Alloy runs — pull-based metric collection from Alloy is not supported. Use push via `logging_alloy_prometheus_write_endpoints` instead.
 
 ## Variables
 
 You can look up all the default values of this role [here](defaults/main.yaml).
 
-The following variables can be set to configure the role:
-
 ### General
 
-| Name                                              | Mandatory | Description                                                      |
-| ------------------------------------------------- | --------- | ---------------------------------------------------------------- |
-| gardener_logging_promtail_chart_version           | yes       | Helm chart version for promtail specified under release vector   |
-| gardener_logging_promtail_chart_repo              | yes       | Repository for promtail specified under release vector           |
-| gardener_logging_namespace                        |           | The deployment's target namespace                                |
-| gardener_logging_ingress_dns                      |           | DNS for loki ingress                                             |
-| gardener_logging_ingress_loki_basic_auth_password |           | The basic auth password for the external loki ingress            |
-| gardener_logging_ingress_loki_basic_auth_user     |           | The basic auth user for the external loki ingress                |
-| gardener_logging_deploy_to_garden_cluster         |           | Deploys promtail also into the garden cluster                    |
-| gardener_logging_shooted_seeds                    |           | Shooted seed names on which to deploy promtails that log to loki |
+| Name                                              | Mandatory | Default                                      | Description                                                                                                                                                                    |
+| ------------------------------------------------- | --------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| gardener_logging_alloy_enabled                    |           | `true`                                       | Deploy Alloy. Requires `gardener_logging_alloy_chart_version` and `gardener_logging_alloy_chart_repo`.                                                                         |
+| gardener_logging_alloy_chart_version              |           |                                              | Helm chart version for alloy (release vector)                                                                                                                                  |
+| gardener_logging_alloy_chart_repo                 |           |                                              | Repository for alloy (release vector)                                                                                                                                          |
+| gardener_logging_promtail_enabled                 |           | `false`                                      | Deploy Promtail (**deprecated**). When `false`, any existing Promtail release is removed automatically from the garden cluster and all seeds. Requires chart vars when `true`. |
+| gardener_logging_promtail_chart_version           |           |                                              | Helm chart version for promtail (release vector)                                                                                                                               |
+| gardener_logging_promtail_chart_repo              |           |                                              | Repository for promtail (release vector)                                                                                                                                       |
+| gardener_logging_namespace                        |           | `monitoring`                                 | Target namespace                                                                                                                                                               |
+| gardener_logging_garden_name                      |           | `{{ gardener_defaults_garden_name }}`        | Name of the garden cluster (used as `cluster=` label for the garden cluster Alloy deployment)                                                                                  |
+| gardener_logging_deploy_to_garden_cluster         |           | `true`                                       | Deploy Alloy also into the garden cluster                                                                                                                                      |
+| gardener_logging_shooted_seeds                    |           | `[]`                                         | List of shooted seeds to deploy Alloy into. Each entry: `{name: <seed-name>}`                                                                                                  |
+| gardener_logging_ingress_dns                      |           | `loki.{{ metal_control_plane_ingress_dns }}` | DNS for the loki ingress (used in `gardener_logging_alloy_loki_write_endpoints` examples and in the deprecated Promtail template)                                              |
+| gardener_logging_ingress_loki_basic_auth_user     |           | `promtail`                                   | Basic auth user for the external loki ingress (used in `gardener_logging_alloy_loki_write_endpoints` examples and in the deprecated Promtail template)                         |
+| gardener_logging_ingress_loki_basic_auth_password |           |                                              | Basic auth password for the external loki ingress. Used by the deprecated Promtail template and for configuring `gardener_logging_alloy_loki_write_endpoints`.                 |
+
+### Alloy
+
+| Name                                                     | Mandatory | Default | Description                                                                                                                                                                        |
+| -------------------------------------------------------- | --------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| gardener_logging_alloy_loki_write_endpoints              | yes\*     | `[]`    | List of Loki push endpoints. Required when `gardener_logging_alloy_enabled: true`. Each entry: `{url, remote_timeout?: duration, basic_auth?: {username, password}}`               |
+| gardener_logging_alloy_prometheus_write_endpoints        |           | `[]`    | Prometheus remote_write endpoints for Alloy self-metrics. When empty, self-metrics are disabled. Each entry: `{url, remote_timeout?: duration, basic_auth?: {username, password}}` |
+| gardener_logging_alloy_prometheus_wal_truncate_frequency |           | `2h`    | How often the WAL is compacted                                                                                                                                                     |
+| gardener_logging_alloy_prometheus_wal_max_keepalive_time |           | `8h`    | Maximum time undelivered samples are kept in the WAL before being dropped                                                                                                          |
+| gardener_logging_alloy_config_raw                        |           |         | Full Alloy River config override for all targets. Bypasses all structured vars and the template.                                                                                   |
+
+All `gardener_logging_alloy_*` variables are explicitly mapped to their `logging_common_alloy_*` counterparts when calling [logging-common](../logging-common/). This is necessary because the two roles use different variable prefixes — the [logging role](../logging/) similarly maps its `logging_alloy_*` variables to `logging_common_alloy_*`.
+
+## Migration from Promtail
+
+See the [logging-common migration guide](../logging-common/README.md#migration-from-promtail) for general guidance.
+
+Alloy is deployed by default. Existing Promtail installations are automatically removed from the garden cluster and all shooted seeds when the role runs.
+
+| Scenario                   | `gardener_logging_alloy_enabled` | `gardener_logging_promtail_enabled` | Notes                                                                                                  |
+| -------------------------- | -------------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------ |
+| **Default**                | `true`                           | `false`                             | Alloy is deployed; existing Promtail releases are removed automatically from all targets.              |
+| **Parallel run**           | `true`                           | `true`                              | Both collectors ship logs. Loki receives duplicate entries. Deprecated — emits a warning on every run. |
+| **Promtail only** (legacy) | `false`                          | `true`                              | Deprecated — emits a warning on every run.                                                             |

control-plane/roles/gardener-logging/defaults/main.yaml

@@ -1,11 +1,47 @@
 ---
 gardener_logging_namespace: monitoring
-
-gardener_logging_ingress_dns: "loki.{{ metal_control_plane_ingress_dns }}"
-
 gardener_logging_garden_name: "{{ gardener_defaults_garden_name }}"
-gardener_logging_ingress_loki_basic_auth_user: promtail
-gardener_logging_ingress_loki_basic_auth_password:
 gardener_logging_deploy_to_garden_cluster: true
 gardener_logging_shooted_seeds: []
   # - name: my-shooted-seed
+
+# Alloy deployment.
+# Requires gardener_logging_alloy_chart_version and gardener_logging_alloy_chart_repo when enabled.
+gardener_logging_alloy_enabled: true
+
+gardener_logging_ingress_dns: "loki.{{ metal_control_plane_ingress_dns }}"
+# is not renamed to alloy for backward compatibility with existing installs.
+# if updated, the shooted seed`s alloy/promtail need their loki endpoints updated to match.
+gardener_logging_ingress_loki_basic_auth_user: promtail
+gardener_logging_ingress_loki_basic_auth_password:
+
+# Loki push endpoints.
+gardener_logging_alloy_loki_write_endpoints: []
+  # - url: "https://{{ gardener_logging_ingress_dns }}/loki/api/v1/push"
+  #     remote_timeout: 60s
+  #     basic_auth:
+  #       username: "{{ gardener_logging_ingress_loki_basic_auth_user }}"
+  #       password: "{{ gardener_logging_ingress_loki_basic_auth_password }}"
+
+# Prometheus remote_write endpoints for Alloy self-metrics (prometheus.exporter.self).
+# Alloy scrapes its own /metrics and pushes them via remote_write.
+# Each entry: {url, remote_timeout?: duration, basic_auth?: {username, password}}
+gardener_logging_alloy_prometheus_write_endpoints: []
+# - url: "https://{{ monitoring_thanos_receive_ingress_dns }}/api/v1/receive"
+#     remote_timeout: 60s
+#     basic_auth:
+#       username: "{{ monitoring_thanos_receive_ingress_basic_auth_user }}"
+#       password: "{{ monitoring_thanos_receive_ingress_basic_auth_password }}"
+
+# WAL retention for prometheus.remote_write.
+gardener_logging_alloy_prometheus_wal_truncate_frequency: "2h"
+gardener_logging_alloy_prometheus_wal_max_keepalive_time: "8h"
+
+# Full Alloy River config override. When set, bypasses all structured vars and the template.
+# gardener_logging_alloy_config_raw: |
+
+# Promtail deployment (deprecated). Set true only if still on Promtail.
+# Requires gardener_logging_promtail_chart_version and gardener_logging_promtail_chart_repo when enabled.
+# When false, any existing Promtail release is automatically uninstalled.
+# See the Migration from Promtail section in README.md.
+gardener_logging_promtail_enabled: false

control-plane/roles/gardener-logging/meta/main.yaml

@@ -2,7 +2,7 @@
 galaxy_info:
   role_name: gardener-logging
   author: metal-stack
-  description: Deploys promtail into a Gardener seed.
+  description: Deploys Alloy into a Gardener seed.
   license: MIT
   min_ansible_version: "2.18"
   galaxy_tags: []

control-plane/roles/gardener-logging/tasks/gardener-shooted-seed.yaml

@@ -7,7 +7,25 @@
   ansible.builtin.set_fact:
     _shoot_kubeconfig: "{{ virtual_garden_kubeconfig | string | shoot_admin_kubeconfig('garden', gardener_logging_shooted_seed.name) | from_yaml }}"
 
-- name: Deploy Promtail
+- name: Deploy Alloy to seed
+  ansible.builtin.include_role:
+    name: metal-roles/control-plane/roles/logging-common
+    tasks_from: deploy-alloy.yaml
+  vars:
+    logging_common_alloy_namespace: "{{ gardener_logging_namespace }}"
+    logging_common_alloy_chart_repo: "{{ gardener_logging_alloy_chart_repo }}"
+    logging_common_alloy_chart_version: "{{ gardener_logging_alloy_chart_version }}"
+    logging_common_alloy_loki_write_endpoints: "{{ gardener_logging_alloy_loki_write_endpoints }}"
+    logging_common_alloy_cluster_label: "{{ gardener_logging_shooted_seed.name }}"
+    logging_common_alloy_service_monitor_enabled: false
+    logging_common_alloy_prometheus_write_endpoints: "{{ gardener_logging_alloy_prometheus_write_endpoints }}"
+    logging_common_alloy_prometheus_wal_truncate_frequency: "{{ gardener_logging_alloy_prometheus_wal_truncate_frequency }}"
+    logging_common_alloy_prometheus_wal_max_keepalive_time: "{{ gardener_logging_alloy_prometheus_wal_max_keepalive_time }}"
+    logging_common_alloy_config_raw: "{{ gardener_logging_alloy_config_raw | default('') }}"
+    logging_common_alloy_kubeconfig: "{{ _shoot_kubeconfig }}"
+  when: gardener_logging_alloy_enabled
+
+- name: Deploy Promtail (parallel migration window)
   kubernetes.core.helm:
     name: promtail
     chart_repo_url: "{{ gardener_logging_promtail_chart_repo }}"
@@ -17,3 +35,12 @@
     values: "{{ lookup('template', 'seed-promtail-values.yaml') | from_yaml }}"
     kubeconfig: "{{ _shoot_kubeconfig }}"
     create_namespace: true
+  when: gardener_logging_promtail_enabled
+
+- name: Remove Promtail from seed
+  kubernetes.core.helm:
+    name: promtail
+    namespace: "{{ gardener_logging_namespace }}"
+    kubeconfig: "{{ _shoot_kubeconfig }}"
+    state: absent
+  when: not gardener_logging_promtail_enabled | bool