PMM-15227 Remove stale HA replicas from Inventory - #5738
Open
4nte wants to merge 8 commits into
Open
Conversation
catalinaadam
temporarily deployed
to
PMM-15227-remove-stale-ha-nodes-from-inventory - pmm-doc-3.9.0 PR #5738
August 6, 2026 08:03 — with 
Render
Destroyed
Render
Destroyed
ademidoff
temporarily deployed
to
PMM-15227-remove-stale-ha-nodes-from-inventory - pmm-doc-3 PR #5738
August 6, 2026 08:03 — with
Render
Destroyed
Render
Destroyed
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5738 +/- ##
==========================================
+ Coverage 43.59% 45.07% +1.48%
==========================================
Files 415 218 -197
Lines 43134 27916 -15218
==========================================
- Hits 18804 12584 -6220
+ Misses 22454 13977 -8477
+ Partials 1876 1355 -521
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
4nte
marked this pull request as ready for review
August 6, 2026 09:18
4nte
requested review from
ademidoff and
maxkondr
and removed request for
a team
August 6, 2026 09:18
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🔗 Linked repositories identifiedCodeRabbit considers these linked repositories for cross-repo context during reviews:
🚧 Files skipped from review as they are similar to previous changes (2)
WalkthroughHA setup now removes eligible stale PMM Server nodes when trusted peer data identifies scaled-down replicas. It preserves active replicas and nodes with monitored services. Tests cover cleanup and invalid peer data. Documentation describes the resulting Inventory behavior. ChangesHA stale-node cleanup
Sequence Diagram(s)sequenceDiagram
participant HASetup as setupPMMServerHAAgents
participant Cleanup as RemoveStaleHANodes
participant Nodes as PMM node records
participant Agents as PMM agent records
HASetup->>Cleanup: Pass HA node ID and configured peers
Cleanup->>Nodes: Find obsolete PMM Server nodes
Cleanup->>Agents: Check monitored services
Cleanup->>Nodes: Remove eligible stale nodes
Cleanup-->>HASetup: Return cleanup result
Possibly related PRs
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@documentation/docs/install-pmm/install-HA-clustered.md`:
- Line 832: Revise the statement about removed replica Nodes in the HA cluster
cleanup documentation to say that only eligible stale Nodes disappear after
remaining pods restart. Mention that removal is skipped when peer data is
untrusted or the Node still monitors Services, and instruct operators to move
those Services to a running replica before removal.
In `@managed/models/node_helpers_test.go`:
- Around line 272-275: Replace the live PostgreSQL setup in the
RemoveStaleHANodes unit test with a go-sqlmock database, configuring only the
SQL expectations needed by RemoveStaleHANodes. Keep testdb.Open out of this unit
test; move the coverage to integration tests only if the behavior cannot be
validated with mocked database interactions.
In `@managed/models/node_helpers.go`:
- Around line 375-394: Exclude the legacy PMM Server node from stale-replica
removal by adding a check for node.NodeID == PMMServerNodeID in the loop before
removeNode is called. Preserve the existing handling for eligible HA replicas
and retain the legacy node without attempting removal.
- Around line 411-420: Update the peer parsing helper around the existing
strings.TrimSpace/strings.Cut logic to call net.ParseIP on the full trimmed peer
entry before splitting at “:”, rejecting unbracketed IPv6 addresses instead of
treating their first segment as a node label. Preserve the existing host and
label validation for non-IP entries, and add an unbracketed IPv6 case to the
untrusted-peer tests.
- Around line 359-399: Replace the package-level logrus.Warnf and logrus.Infof
calls in the stale HA cleanup flow with structured *logrus.Entry logging. Attach
relevant fields such as node_id, node_name, peer, and ha_node_id to each event
while preserving the existing messages and control flow.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: f0769d83-7fa0-483d-b09c-be3789de1b70
📒 Files selected for processing (4)
documentation/docs/install-pmm/install-HA-clustered.mdmanaged/models/database.gomanaged/models/node_helpers.gomanaged/models/node_helpers_test.go
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
percona/pmm-qa(manual)percona/pmm(manual)
| - HAProxy continues routing to available pods during rollout | ||
| - No data loss (distributed storage) | ||
| - Rolling update strategy minimizes downtime | ||
| - The Nodes of removed replicas disappear from **Inventory > Nodes** once the remaining pods restart |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
State the retention condition.
This sentence guarantees Node removal after restart. Cleanup retains a stale Node when it still monitors Services. Cleanup also skips removal when peer data is not trusted.
State that only eligible stale replica Nodes disappear. Explain that operators must move monitored Services to a running replica before removal.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@documentation/docs/install-pmm/install-HA-clustered.md` at line 832, Revise
the statement about removed replica Nodes in the HA cluster cleanup
documentation to say that only eligible stale Nodes disappear after remaining
pods restart. Mention that removal is skipped when peer data is untrusted or the
Node still monitors Services, and instruct operators to move those Services to a
running replica before removal.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PMM-15227
Percona-Lab/pmm-submodules#4513
Problem
Every PMM Server replica in an HA deployment registers its own inventory Node at first boot,
and nothing ever removed it. After scaling the deployment down, the departed replica kept
appearing under Inventory > Nodes with status Unknown.
Scope
HA deployments only, and inventory rows only. No API or schema change; metrics already
written to VictoriaMetrics are left alone.
Solution
At startup, remove PMM Server Nodes whose name is absent from the configured
PMM_HA_PEERS.The peer list is the signal because the chart regenerates it from the replica count and
recreates every pod when it changes, whereas a member missing from the memberlist cluster may
simply be restarting.
Cleanup is skipped when the peer list cannot be trusted (an entry that carries no node name,
or a list that omits this replica), and a departing replica that still monitors Services is
kept with a warning instead of having those exporters cascade-deleted.