chore: replace control-plane promtail with alloy

[ma-hartma]

Description

Replaces promtail with alloy in the control-plane.
The partition replacement is treated separately in #592 and should go first.

• deploy alloy
  • role for helm chart in k8s cluster
• convert existing configs
  • pods
  • events
  • labels
  • metrics
• logs and metrics for alloy itself (https://grafana.com/docs/alloy/latest/collect/metamonitoring/=) to loki and prometheus
• WAL persistence to allow restarts
• check promtail helm charts, as they had lots of defaults that alloy doesn't: https://github.com/grafana/helm-charts/tree/main/charts/promtail

Question Marks:

• evaluate using https://github.com/grafana/k8s-monitoring-helm instead of https://github.com/prometheus-operator/kube-prometheus

After successful replacement

• update monitoring docs for the website
• migration guide
  • convert does not work with helm chart
• deprecate promtail roles

Manually tested in the mini-lab and demo-rack

• metrics are pushed to thanos, accessible via Grafana
• labels work as documented and before
• cutover migration from promtail to alloy in the mini-lab including cleanup
• deprecation warnings show up

Still needs to be tested:

• enable Service Monitors

References:

• chore: replace partition promtail with alloy #592
• docs(monitoring): use alloy instead of promtail website#260 website docs update
• chore: add alloy helm chart releases#288
• Sflow integration #584 will have to replace promtail as well

Closes #552

Used AI-Tools ✨

• Claude Sonnet 4.6 for docs and migration guide generation and jinja templating

Release Notes

Breaking Change

The event `job` label was renamed from `monitoring/event-exporter` to `events`. Update any existing LogQL queries, dashboard filters, and alert rules that reference `{job="monitoring/event-exporter"}` to `{job="events"}`.

The monitoring role's basic auth configuration for `Thanos Receive Ingress` has changed. The raw `htpasswd` string `monitoring_thanos_receive_ingress_basic_auth` has been replaced by two plaintext variables `
monitoring_thanos_receive_ingress_basic_auth_user: thanos-receive` and `monitoring_thanos_receive_ingress_basic_auth_password: mysecret`.
See: https://github.com/metal-stack/metal-roles/blob/master/control-plane/roles/monitoring/README.md#thanos-receive-ingress-credentials

Required Actions

The control-plane `logging` role now deploys Grafana `Alloy` as the default log collector. Alloy needs to be configured before running the new version of the role, as the deprecated `Promtail` DaemonSet and the `event-exporter` resources are disabled and removed by default.

Follow the migration instructions:
- https://github.com/metal-stack/metal-roles/blob/master/control-plane/roles/logging-common/README.md#migration-from-promtail

Several migration scenarios are supported — from a hard cutover to a parallel run for verification before switching over. Note that running both simultaneously will produce duplicate log entries in Loki during the transition window.