PMM-14956 HA alert templates - #5730
Conversation
WalkthroughPMM HA now normalizes peer configuration, exposes the expected node count through HA metrics, and adds five alert templates for cluster health conditions. Documentation describes template setup, alert-rule creation, remediation, and detection limitations. ChangesPMM HA alerting and metrics
Possibly related PRs
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5730 +/- ##
==========================================
+ Coverage 43.59% 45.21% +1.61%
==========================================
Files 415 218 -197
Lines 43134 27867 -15267
==========================================
- Hits 18804 12599 -6205
+ Misses 22454 13915 -8539
+ Partials 1876 1353 -523
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@coderabbitai full review |
✅ Action performedFull review finished. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@managed/cmd/pmm-managed/main.go`:
- Around line 1245-1256: Update parseHAPeers to accept the existing
*logrus.Entry main logger, and replace the package-level logrus.Warnf duplicate
warning with a structured field for the peer while preserving the warning
message and duplicate-handling behavior. Update its caller to pass main.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 83f38b89-81fb-4540-b835-65783dc99e13
📒 Files selected for processing (13)
documentation/docs/alert/templates_list.mddocumentation/docs/install-pmm/install-HA-clustered.mdmanaged/cmd/pmm-managed/main.gomanaged/cmd/pmm-managed/main_test.gomanaged/data/alerting-templates/ha_leader_flapping.ymlmanaged/data/alerting-templates/ha_no_leader.ymlmanaged/data/alerting-templates/ha_node_unreachable.ymlmanaged/data/alerting-templates/ha_quorum_at_risk.ymlmanaged/data/alerting-templates/ha_split_brain.ymlmanaged/services/ha/ha.gomanaged/services/ha/ha_metrics.gomanaged/services/ha/ha_metrics_test.gomanaged/services/ha/haservice.go
| func parseHAPeers(peers string) []string { | ||
| var nodes []string | ||
| seen := make(map[string]struct{}) | ||
|
|
||
| for node := range strings.SplitSeq(peers, ",") { | ||
| node = strings.TrimSpace(node) | ||
| if node == "" { | ||
| continue | ||
| } | ||
| if _, ok := seen[node]; ok { | ||
| logrus.Warnf("Ignoring duplicate entry %q in PMM_HA_PEERS.", node) | ||
| continue |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Use a structured logrus.Entry for duplicate-peer warnings.
Line 1255 uses the package-level logrus.Warnf. Pass the existing main entry to parseHAPeers. Log the peer as a field so the message retains component context and supports structured filtering.
As per coding guidelines, use “structured logging, such as s.l.WithField("key", value).Error("message"), and pass *logrus.Entry rather than *logrus.Logger.”
Proposed fix
- nodes := parseHAPeers(*haPeers)
+ nodes := parseHAPeers(l, *haPeers)
-func parseHAPeers(peers string) []string {
+func parseHAPeers(l *logrus.Entry, peers string) []string {
...
- logrus.Warnf("Ignoring duplicate entry %q in PMM_HA_PEERS.", node)
+ l.WithField("peer", node).Warn("Ignoring duplicate entry in PMM_HA_PEERS.")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| func parseHAPeers(peers string) []string { | |
| var nodes []string | |
| seen := make(map[string]struct{}) | |
| for node := range strings.SplitSeq(peers, ",") { | |
| node = strings.TrimSpace(node) | |
| if node == "" { | |
| continue | |
| } | |
| if _, ok := seen[node]; ok { | |
| logrus.Warnf("Ignoring duplicate entry %q in PMM_HA_PEERS.", node) | |
| continue | |
| func parseHAPeers(l *logrus.Entry, peers string) []string { | |
| var nodes []string | |
| seen := make(map[string]struct{}) | |
| for node := range strings.SplitSeq(peers, ",") { | |
| node = strings.TrimSpace(node) | |
| if node == "" { | |
| continue | |
| } | |
| if _, ok := seen[node]; ok { | |
| l.WithField("peer", node).Warn("Ignoring duplicate entry in PMM_HA_PEERS.") | |
| continue |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@managed/cmd/pmm-managed/main.go` around lines 1245 - 1256, Update
parseHAPeers to accept the existing *logrus.Entry main logger, and replace the
package-level logrus.Warnf duplicate warning with a structured field for the
peer while preserving the warning message and duplicate-handling behavior.
Update its caller to pass main.
Source: Coding guidelines
4 participants
PMM-14956
Feature build
What this adds
Five built-in alert templates for a PMM Server High Availability cluster:
pmm_ha_no_leaderpmm_ha_split_brainpmm_ha_leader_flappingpmm_ha_node_unreachablePMM_HA_PEERSdeclarespmm_ha_quorum_at_riskDocumentation
documentation/docs/alert/templates_list.md- new "PMM High Availabilitytemplates" section with the table, the steps to create rules from the templates,
and the coverage limitations.
documentation/docs/install-pmm/install-HA-clustered.md- short pointer to thatsection from the HA install page.
Implementation details
last_over_time(...[2m])on every query, not a bare selector. The metricsdatabase's own staleness window is longer than the alert needs. Without the
bounded lookback a departed leader's final sample keeps counting, which delays
pmm_ha_no_leaderand can make an ordinary failover look like a split brain.The 2m window is deliberately shorter than every
for:.max by(node_id)before aggregating. Collapses duplicate series for one node,so extra scrape labels cannot make a single node count twice and raise a split
brain on its own.
< bool/> boolrather than a bare comparison. A bare comparison filtersthe series away and Grafana evaluates the empty result as a label-less NoData
instance, which breaks alert identity and grouping. The
boolform keeps oneinstance carrying its labels.
pmm_ha_quorum_at_risksuppresses itself below three nodes viaand on() (... > 2). On one- and two-node clusters the condition is permanentlytrue, so it would be noise.
pmm_ha_node_unreachablecovers those sizes.pmm_ha_node_unreachableusesunlessplus anabsent()-guarded fallback.A node that is down emits nothing, so it cannot be named from a live series. The
first term names nodes seen in the last 6h but not the last 2m; the fallback
catches a node that never reported or has been gone longer than 6h, and reports it
without a name. The guard stops both terms firing for the same node.
pmm_ha_uprather than reporting 0. Thatis why the counting alerts compare against
pmm_ha_expected_nodesinstead ofsumming a health gauge.
templates silent on standalone PMM.
Summary by CodeRabbit
New Features
pmm_ha_expected_nodesmetric to report configured cluster size.Bug Fixes
Documentation