fix(dashboard): restore pipeline health metrics

[kangkyunghyun]

Description

Restore the missing Cortex and Data Prepper metrics used by the Observability Pipeline Health dashboard.

The Docker Compose OpenTelemetry Collector configuration previously scraped only its own metrics and Envoy metrics. As a result, the Cortex and Data Prepper dashboard panels had no corresponding metric data and displayed No results found.

This change:

• Adds a Cortex metrics scrape from prometheus:9090/metrics.
• Adds a Data Prepper pipeline metrics scrape from data-prepper:4900/metrics/prometheus.
• Connects both receivers to the Collector metrics pipeline so the collected metrics are written to Cortex.
• Replaces the nonexistent otelcol_process_memory_rss_bytes query with the actual otelcol_process_memory_rss metric.
• Applies the corrected memory query to both the Docker Compose and Helm dashboard definitions.
• Adds a Helm unit test to prevent the incorrect metric name from being reintroduced.

Validation

• Ran docker compose config --quiet.
• Started the core stack and verified that the Cortex and Data Prepper scrape jobs started successfully.
• Queried Cortex and confirmed that the following queries return data:
  • avg(cortex_ingester_ingestion_rate_samples_per_second)
  • cortex_ingester_active_series
  • otelcol_process_memory_rss
  • up{job="data-prepper-pipelines"}
• Generated canary traffic and confirmed that Data Prepper processing and document-write rate queries return non-zero values.
• Ran opensearch-dashboards-init and verified that all 23 Pipeline Health dashboard panels were updated successfully.
• Ran helm dependency build charts/observability-stack.
• Ran helm lint charts/observability-stack.
• Ran helm template observability-stack charts/observability-stack.
• Ran all Helm unit tests: 74 passed.

Because the Data Prepper image change from #408 has not yet been merged, Docker Compose validation used opensearchproject/data-prepper:2.16.0 through an environment override. No .env changes from that override are included in this PR.

Issues Resolved

Closes #410

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.