Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
4f88319
feat: improve HA dashboard
fabio-silva Jul 21, 2026
c3e1a1e
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
fabio-silva Jul 21, 2026
e85b282
chore: improve layout
fabio-silva Jul 22, 2026
be3e335
chore: improve layout
fabio-silva Jul 23, 2026
ed7eae4
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
fabio-silva Jul 23, 2026
31c3645
chore: improve layout
fabio-silva Jul 24, 2026
c2e5333
Merge branch 'PMM-13860-finalize-ha-overview-dashboard' of https://gi…
fabio-silva Jul 24, 2026
89e21aa
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
fabio-silva Jul 24, 2026
db0e359
chore: remove unstyled leader from sidebar
fabio-silva Jul 27, 2026
604d122
Merge branch 'PMM-13860-finalize-ha-overview-dashboard' of https://gi…
fabio-silva Jul 27, 2026
78ffb41
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
fabio-silva Jul 27, 2026
e70a050
chore: format
fabio-silva Jul 27, 2026
368c6f5
chore: improve namespace filtering
fabio-silva Jul 28, 2026
d5a27b9
chore: change layout
fabio-silva Jul 28, 2026
09b2019
chore: improve colors
fabio-silva Jul 28, 2026
4d63de9
chore: improve colors
fabio-silva Jul 28, 2026
5b7a07f
chore: remove helm release selector
fabio-silva Jul 28, 2026
8e4916e
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Jul 28, 2026
84497da
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Jul 29, 2026
ba7c3aa
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Jul 29, 2026
05ccb18
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Jul 29, 2026
c0c1331
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Jul 29, 2026
5b6f1e6
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Jul 29, 2026
53c96e0
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Jul 29, 2026
fe02c57
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Jul 29, 2026
33af541
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Jul 30, 2026
2a84719
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Jul 30, 2026
b783f05
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Jul 30, 2026
bad6376
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Jul 31, 2026
73eb5a9
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Aug 3, 2026
fcf4b74
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Aug 3, 2026
c078aec
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Aug 3, 2026
753de8a
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Aug 3, 2026
f10b247
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Aug 4, 2026
35aac6e
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Aug 4, 2026
ea97f67
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Aug 4, 2026
733fe85
Merge branch 'main' into PMM-13860-finalize-ha-overview-dashboard
pmm-prbot[bot] Aug 4, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dashboards/dashboards/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ dashboards/dashboards/*.json
| `Valkey/` | Valkey/Redis clients, cluster, memory, replication, slowlog |
| `Insight/` | Home Dashboard, Advanced Data Exploration, VictoriaMetrics, Exporters |
| `Experimental/` | Databases Overview, DB Cluster Summary |
| `PMM Health/` | Environments Overview, PMM Health Overview, HA Health Overview |
| `PMM Health/` | Environments Overview, PMM Health Overview, PMM HA Overview |
| `Query Analytics/` | QAN panel wrapper (`pmm-qan.json`) |
| `Kubernetes (experimental)/` | Kubernetes operator monitoring |

Expand Down
37,268 changes: 35,836 additions & 1,432 deletions dashboards/dashboards/PMM Health/PMM_HA_Health_Overview.json

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like Alive/Expected is encoded as alive * 1000 + expected, then mapped back with a large value-mappings list (repeated across the health panels). That seems to be what makes the dashboard JSON quite large (~2MB), and it may also cap the display around 40/40.

Was that the intended approach for showing 3/3-style counts in Grafana?

If not, a simpler option could be:

  1. Two series in the same Stat (Alive, Expected) with textMode: value_and_name
  2. Or a small overview table: Component | Alive | Expected | Health

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion dashboards/pmm-app/src/plugin.json
Original file line number Diff line number Diff line change
Expand Up @@ -305,7 +305,7 @@
},
{
"type": "dashboard",
"name": "PMM HA Health Overview",
"name": "PMM HA Overview",
"path": "dashboards/PMM Health/PMM_HA_Health_Overview.json"
},
{
Expand Down
8 changes: 4 additions & 4 deletions documentation/docs/install-pmm/install-HA-clustered.md
Original file line number Diff line number Diff line change
Expand Up @@ -773,15 +773,15 @@ To create additional service tokens manually, see the [PMM documentation on serv

#### Monitor cluster health

Use the **PMM HA Health Overview** dashboard to monitor your entire HA deployment from a single view.
Use the **PMM HA Overview** dashboard to monitor your entire HA deployment from a single view.

This dashboard shows real-time health status for all critical components including PMM server replicas, PostgreSQL, ClickHouse, VictoriaMetrics, and HAProxy.
This dashboard shows alive versus expected instance counts, active or primary nodes, replication health, and real-time status for PMM, PostgreSQL, ClickHouse, VictoriaMetrics, and HAProxy.

Access the dashboard from **All Dashboards > Browse all dashboards > Experimental > PMM HA Health Overview**.
Access the dashboard from **PMM HA > Overview** in the main menu, or from **All Dashboards > Browse all dashboards > PMM Health > PMM HA Overview**.

The dashboard helps you quickly identify component failures, resource constraints, and stability issues across your high-availability infrastructure.

For detailed information about each panel and what to check, see the [PMM HA Health Overview dashboard reference](../reference/dashboards/dashboard-ha-health-overview.md).
For detailed information about each panel and what to check, see the [PMM HA Overview dashboard reference](../reference/dashboards/dashboard-ha-health-overview.md).

#### Identify the leader node

Expand Down
Original file line number Diff line number Diff line change
@@ -1,26 +1,26 @@
# PMM HA Health Overview
# PMM HA Overview

![PMM HA Health Overview Dashboard](../../images/PMM_HA_Health_Overview.png)
![PMM HA Overview Dashboard](../../images/PMM_HA_Health_Overview.png)

The PMM HA Health Overview dashboard provides at-a-glance monitoring of your PMM High Availability Cluster deployment health.
The PMM HA Overview dashboard provides at-a-glance monitoring of cluster-level and instance-level health for your PMM High Availability deployment.

Use this dashboard to quickly identify component failures, resource constraints, and stability issues across your high-availability infrastructure.
Use the **Alive**, **Expected**, and **Health** values to identify component failures. Green means all expected instances are alive, yellow means the service is degraded, and red means no instances are alive or no expected instances were discovered.

This dashboard monitors all critical components: PMM server replicas, PostgreSQL database cluster, ClickHouse query analytics storage, VictoriaMetrics time series storage, and HAProxy load balancers.
The dashboard monitors PMM server replicas, PostgreSQL, ClickHouse, VictoriaMetrics, and HAProxy. It also identifies the active PMM instance, PostgreSQL primary, ClickHouse Keeper leader, VictoriaMetrics component health, and HAProxy backend server status.

## Overview

### PMM

Shows the overall health status of PMM server pods with green **Healthy** when all pods are running normally or red **Not Healthy** when one or more pods are down.
Shows PMM instances alive versus expected and the resulting health percentage.

In HA Cluster mode, you have three PMM server replicas providing full redundancy. If one replica fails, the remaining two continue serving requests with no user-visible impact, but you should investigate quickly to restore full redundancy.

Check the **PMM Pods** table in the Pod Status Details section below to see which specific pods are affected.
The **Active PMM Instance** panel identifies the current Raft leader. Check the **PMM Pods** table to see every instance's status and Active or Follower role.
Comment on lines +15 to +19

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Use the configured PMM replica count throughout the dashboard reference.

The installation guide supports scaling PMM replicas beyond three. Make both sections use count-independent wording:

  • documentation/docs/reference/dashboards/dashboard-ha-health-overview.md#L15-L19: describe the configured replica count, with three as the default example.
  • documentation/docs/reference/dashboards/dashboard-ha-health-overview.md#L216-L222: replace all three and the fixed two-pod threshold with guidance based on the configured count.
📍 Affects 1 file
  • documentation/docs/reference/dashboards/dashboard-ha-health-overview.md#L15-L19 (this comment)
  • documentation/docs/reference/dashboards/dashboard-ha-health-overview.md#L216-L222
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@documentation/docs/reference/dashboards/dashboard-ha-health-overview.md`
around lines 15 - 19, Update the HA overview wording at
documentation/docs/reference/dashboards/dashboard-ha-health-overview.md:15-19 to
refer to the configured PMM replica count, mentioning three only as the default
example; revise
documentation/docs/reference/dashboards/dashboard-ha-health-overview.md:216-222
to replace “all three” and the fixed two-pod threshold with guidance derived
from the configured replica count. Keep the Active PMM Instance and PMM Pods
guidance unchanged.


### PostgreSQL

Shows the overall health status of PostgreSQL database pods. Green **Healthy** indicates the cluster is fully operational with primary and replicas running. Red **Not Healthy** signals the cluster is degraded, putting metadata and configuration storage at risk.
Shows PostgreSQL instances alive versus expected. The **PostgreSQL Primary** panel identifies the current primary, and the instance table reports replication streaming health.

PostgreSQL stores PMM's critical data including user accounts, dashboard configurations, alerting rules, and inventory information.

Expand All @@ -30,20 +30,21 @@ Check the [**PostgreSQL Pods**](#postgresql-pods) panel in the **PostgreSQL Pod'

### ClickHouse

Shows the overall health status of ClickHouse pods. Green **Healthy** indicates the cluster is operating normally with all keeper and database nodes running. Red **Not Healthy** signals Query Analytics data storage is degraded.
Shows ClickHouse database and Keeper nodes alive versus expected.

ClickHouse stores all Query Analytics (QAN) data. When pods fail, QAN dashboards will show incomplete query performance metrics, you won't be able to analyze slow queries during the outage, and you'll have permanent gaps in your historical query analysis data.

Check the **ClickHouse Pods** panel in the **ClickHouse Pod's Status** section below to see which pods are down.
Check the **ClickHouse Pods** panel to distinguish database (`pmmdb`) and Keeper nodes and to identify the Keeper leader. The dashboard reports Kubernetes node role and Keeper election state; it does not infer ClickHouse logical shard topology.

Loss of keeper nodes (which handle coordination) is more critical than loss of a single database node (which handles storage).

### VictoriaMetrics

Shows the overall health status of VictoriaMetrics components. Green **Healthy** indicates all components are operational and metrics collection and querying are working properly. Red **Not Healthy** signals the time series infrastructure is degraded.
Shows VictoriaMetrics instances alive versus expected. The **VictoriaMetrics Components** panel breaks health down by `vmauth`, `vminsert`, `vmselect`, `vmstorage`, and `vmagent`.

VictoriaMetrics uses multiple components working together:
- `vminsert` receives metrics from monitored services, - `vmselect` processes dashboard queries
VictoriaMetrics uses multiple components working together:
- `vminsert` receives metrics from monitored services
- `vmselect` processes dashboard queries
- `vmstorage` stores time series data
- `vmagent` scrapes metrics from targets
- `vmauth` handles authentication
Expand All @@ -54,7 +55,7 @@ Failures in `vminsert` or `vmstorage` are most critical as they prevent metrics

### HAProxy

Shows the overall health status of HAProxy load balancer pods. Green **Healthy** means load balancing is working and traffic is properly distributed across your PMM replicas. Red **Not Healthy** means the load balancer has problems and you may not be able to access PMM.
Shows HAProxy instances alive versus expected. The backend server table reports the UP or DOWN state of each server in each backend.

HAProxy is how you get into your PMM cluster. It routes web traffic to healthy PMM server replicas and handles automatic failover when the leader changes.

Expand Down Expand Up @@ -196,7 +197,7 @@ A healthy PMM HA deployment should be mostly green with only short gaps during p

### PostgreSQL Pods

Shows each of your PostgreSQL pods with their status (UP or DOWN) and role (Primary or Replica). In a healthy cluster, you should see one Primary and the rest as Replicas, all showing UP in green.
Shows each PostgreSQL instance with its status (UP or DOWN), role (Primary or Replica), and replication health. In a healthy cluster, you should see one Primary, the remaining instances as Replicas, and all rows showing UP and Healthy.

If the Primary shows DOWN, a failover is either in progress or just completed. Check that a Replica was promoted to become the new Primary.

Expand All @@ -212,18 +213,18 @@ After a failover, verify that the new Primary is handling writes correctly.

### PMM Pods

Shows each of your PMM server pods with their current status. Green UP means the pod is running normally. Red DOWN means the pod has failed or isn't running.
Shows each PMM server instance with its current status and Raft role. Green UP means the pod is running normally; **Active** identifies the Raft leader and **Follower** identifies the remaining instances.

If one pod shows DOWN, identify which replica is affected and investigate the cause. Two or more DOWN pods means your deployment is at serious risk—investigate immediately.

If all three show DOWN, your entire PMM system is unavailable.

Use this table to identify which specific PMM server pods need attention when the [**PMM**](#pmm) health indicator shows **Not Healthy**.
Use this table to identify which PMM server instances need attention when the [**PMM**](#pmm) health percentage is below 100%.


### ClickHouse Pods

Shows each of your ClickHouse pods with their status (UP or DOWN) and role (Leader or Follower).
Shows every ClickHouse database and Keeper node with its UP or DOWN status. Keeper nodes also show their Leader or Follower role.

ClickHouse uses two types of pods: keeper nodes that handle coordination and metadata, and database nodes that handle storage.

Expand All @@ -243,7 +244,7 @@ Check for failures in `vmstorage` pods first—each stores a subset of your metr

Failures in `vmselect` pods reduce query capacity but don't cause data loss. Down `vmagent` or `vmauth` pods affect metric scraping and authentication respectively.

Use this table to identify which specific components need attention when the [**VictoriaMetrics**](#victoriametrics) health indicator shows **Not Healthy**.
Use this table to identify which components need attention when the [**VictoriaMetrics**](#victoriametrics) health percentage is below 100%.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Replace the remaining Not Healthy condition.

This section now uses a health percentage below 100%, but the Common patterns entry at Line 308 still says Not Healthy. Use the percentage-based condition consistently.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@documentation/docs/reference/dashboards/dashboard-ha-health-overview.md` at
line 247, Update the Common patterns entry near the health-overview guidance to
replace the remaining “Not Healthy” condition with the percentage-based
condition, using health percentage below 100% consistently with the surrounding
VictoriaMetrics instructions.


## HAProxy Pod's Status

Expand All @@ -257,11 +258,11 @@ If one pod shows DOWN, your load balancing is still working but with reduced cap

Investigate DOWN pods quickly to restore full redundancy and prevent a single point of failure. Check pod logs to identify whether the issue is configuration, resource limits, or connectivity problems.

Use this table to identify which specific HAProxy pods need attention when the [**HAProxy**](#haproxy) health indicator shows **Not Healthy**.
Use this table to identify which HAProxy pods need attention when the [**HAProxy**](#haproxy) health percentage is below 100%.

### HAProxy Backends
### HAProxy Backend Servers

Shows the health status of each backend service that HAProxy routes traffic to.
Shows the health status of every server in each backend that HAProxy routes traffic to.

Green UP means HAProxy can reach the backend and will route traffic to it. Red DOWN means HAProxy detected the backend is unavailable and won't route traffic there.

Expand All @@ -278,7 +279,7 @@ Check pod status in the tables above to identify why backends remain unavailable
## Dashboard Usage Tips

### Refresh rate
This dashboard auto-refreshes every 30 seconds to provide near real-time monitoring.
Use Grafana's refresh control to update the dashboard while investigating failovers or degraded components.

### Filters
Use the namespace and Helm release variables at the top to focus on your specific PMM HA deployment if you have multiple installations.
Expand Down
11 changes: 6 additions & 5 deletions ui/apps/pmm/src/contexts/navigation/navigation.constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -790,13 +790,14 @@ export const NAV_HIGH_AVAILABILITY: NavItem = {
id: 'high-availability',
icon: 'cluster',
text: 'PMM HA',
url: `${PMM_NEW_NAV_GRAFANA_PATH}/high-availability`,
url: `${PMM_NEW_NAV_GRAFANA_PATH}/d/pmm-ha-health-overview/pmm-ha-overview`,
};

export const NAV_HIGH_AVAILABILITY_LEADER: NavItem = {
id: 'high-availability-leader',
text: 'Leader:',
type: 'menu-text',
export const NAV_HIGH_AVAILABILITY_OVERVIEW: NavItem = {
id: 'high-availability-overview',
icon: 'overview',
text: 'Overview',
url: `${PMM_NEW_NAV_GRAFANA_PATH}/d/pmm-ha-health-overview/pmm-ha-overview`,
};

export const NAV_HIGH_AVAILABILITY_NODES: NavItem = {
Expand Down
12 changes: 3 additions & 9 deletions ui/apps/pmm/src/contexts/navigation/navigation.utils.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ import {
NAV_HIGH_AVAILABILITY,
NAV_USERS_AND_ACCESS,
NAV_ACCESS_CONTROL,
NAV_HIGH_AVAILABILITY_LEADER,
NAV_HIGH_AVAILABILITY_NODES,
NAV_HIGH_AVAILABILITY_OVERVIEW,
NAV_HOME_PAGE,
} from './navigation.constants';
import { CombinedSettings } from 'contexts/settings';
Expand Down Expand Up @@ -249,20 +249,14 @@ export const addConfiguration = (
return NAV_CONFIGURATION;
};

export const addHighAvailability = ({ health, leader }: HAInfo): NavItem => {
export const addHighAvailability = ({ health }: HAInfo): NavItem => {
const item = { ...NAV_HIGH_AVAILABILITY };

item.badge = <HighAvailabilityBadge health={health} />;
item.icon = <HighAvailabilityIcon health={health} />;
item.badgeAlwaysVisible = true;

item.children = [
{
...NAV_HIGH_AVAILABILITY_LEADER,
secondaryText: leader?.nodeName || 'Unknown',
},
NAV_HIGH_AVAILABILITY_NODES,
];
item.children = [NAV_HIGH_AVAILABILITY_OVERVIEW, NAV_HIGH_AVAILABILITY_NODES];

return item;
};
Expand Down
Loading