PMM-14665 Skip nodes internal to a PMM deployment when adding a service - #5704
Open
ademidoff wants to merge 9 commits into
Open
PMM-14665 Skip nodes internal to a PMM deployment when adding a service#5704ademidoff wants to merge 9 commits into
ademidoff wants to merge 9 commits into
Conversation
In an HA deployment the PostgreSQL cluster backing PMM's persistence layer runs pmm-client sidecars, so its pods were offered in the Nodes dropdown as valid monitoring delegates. Those Nodes are dedicated and must not take on extra monitoring workloads. PMM Server now accepts PMM_INTERNAL_NODE_NAME_PREFIXES, reports the Nodes it matches as is_pmm_internal_node, and rejects requests which delegate monitoring of a remote address to an Agent running on one of them. Local addresses remain allowed, so those Nodes keep monitoring the services inside their own pod. AddAzureDatabaseRequest gains pmm_agent_id, which the UI has always sent but the server discarded, placing Azure exporters on the PMM Server unconditionally.
ademidoff
requested review from
4nte and
JiriCtvrtka
and removed request for
a team
July 27, 2026 22:25
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #5704 +/- ##
==========================================
+ Coverage 43.59% 45.43% +1.84%
==========================================
Files 415 418 +3
Lines 43134 43397 +263
==========================================
+ Hits 18804 19718 +914
+ Misses 22454 21732 -722
- Partials 1876 1947 +71
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
`pmm-agent setup` registered the Node on every run, and a forced registration makes PMM Server drop the Node together with every Service on it. That is harmless for the database operators, whose sidecars re-add their own services on each container start, but it would silently delete user-added services from a long-lived PMM Client pod, which is what the pmm-ha Helm chart is about to pre-provision as monitoring delegates. Registration is now skipped when the Agent already holds an ID and is being set up against the same PMM Server it is registered with. `--force` still registers the Node again, and pointing an Agent at a different PMM Server registers it there as before.
…gent Skipping registration for an Agent which already holds an ID strands it whenever PMM Server no longer knows that ID, e.g. after the server was reinstalled or restored from a backup taken before the Agent was registered. pmm-agent then loops on "No Agent with ID" while the Node monitors nothing. `pmm-agent setup` now asks the server whether it still knows the Agent and registers the Node again when it does not. Credentials the server rejects count as an answer too: registering reports that with an actionable message, whereas skipping it would fail silently later on. An unreachable server is not an answer: a failed check keeps the registration, so an Agent still starts while PMM Server has no leader elected yet.
ademidoff
marked this pull request as draft
July 28, 2026 07:56
Keeping this ticket to what it is about: reporting the Nodes of a PMM deployment's own infrastructure as internal and refusing to delegate monitoring to them. Making `pmm-agent setup` idempotent is what would let a pre-provisioned PMM Client pod register once and keep its identity across restarts, but the chart can do that with a one-time init container instead. The agent side is worth its own effort: it changes pmm-agent CLI behaviour, it needs a release note, and the surrounding questions - how an Agent recovers when PMM Server no longer knows it, and how an orphaned Node is reclaimed - deserve their own design and review.
ademidoff
marked this pull request as ready for review
July 30, 2026 07:40
WalkthroughThe change adds internal-node classification and eligibility checks to management services. It adds optional PMM agent selection for Azure Database resources and exposes both features through management API schemas and configuration. ChangesInternal node management
Estimated code review effort: 4 (Complex) | ~45 minutes Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@api/management/v1/azure.proto`:
- Around line 133-134: Add an optional protobuf validation rule to pmm_agent_id
requiring a minimum length of 1 while allowing it to remain empty, matching the
validation used by existing PMM agent request fields. Preserve the service-level
existence and eligibility checks, then run make gen to regenerate the affected
code.
In `@api/management/v1/json/v1.json`:
- Around line 807-809: Update the is_pmm_internal_node descriptions in ListNodes
at api/management/v1/json/v1.json:807-809 and GetNode at
api/management/v1/json/v1.json:1369-1371 to state that internal nodes must not
receive delegated monitoring for remote addresses, while preserving that
local-address monitoring is permitted.
In `@api/swagger/swagger.json`:
- Around line 27695-27698: Update the source .proto field corresponding to
pmm_agent_id so its description clearly states that it identifies the selected
PMM agent running Azure exporters, then regenerate api/swagger/swagger.json
using the project’s prescribed generation process; do not edit the generated
Swagger file directly.
In `@managed/services/management/azure_database.go`:
- Around line 199-207: Update the RequestStateUpdate call in the
AddAzureDatabase flow to pass the selected pmmAgentID rather than
models.PMMServerAgentID, ensuring custom agent assignments receive the refresh
notification. Preserve the existing state-update timing and error handling.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: a8b9758b-4ff3-494e-b793-5f47e80b23cf
⛔ Files ignored due to path filters (3)
api/descriptor.binis excluded by!**/*.binapi/management/v1/azure.pb.gois excluded by!**/*.pb.goapi/management/v1/node.pb.gois excluded by!**/*.pb.go
📒 Files selected for processing (24)
api/management/v1/azure.pb.validate.goapi/management/v1/azure.protoapi/management/v1/json/client/management_service/add_azure_database_responses.goapi/management/v1/json/client/management_service/get_node_responses.goapi/management/v1/json/client/management_service/list_nodes_responses.goapi/management/v1/json/v1.jsonapi/management/v1/node.pb.validate.goapi/management/v1/node.protoapi/swagger/swagger-dev.jsonapi/swagger/swagger.jsonmanaged/cmd/pmm-managed/main.gomanaged/cmd/pmm-managed/main_test.gomanaged/services/management/add_service_exporter_timeout_test.gomanaged/services/management/agent_test.gomanaged/services/management/annotation_test.gomanaged/services/management/azure_database.gomanaged/services/management/internal_node_test.gomanaged/services/management/node.gomanaged/services/management/node_test.gomanaged/services/management/rds_test.gomanaged/services/management/service.gomanaged/services/management/service_test.gomanaged/utils/envvars/parser.gomanaged/utils/envvars/parser_test.go
Comment on lines
+807
to
+809
| "is_pmm_internal_node": { | ||
| "description": "True if this node belongs to the internal infrastructure of a PMM deployment\n(e.g. the HA persistence layer) and must not host user monitoring workloads.", | ||
| "type": "boolean", |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Align both descriptions with the enforced eligibility rule.
The current text states that internal nodes must not host user monitoring workloads. The stated behavior permits local-address monitoring. Document the remote-address delegation restriction instead.
api/management/v1/json/v1.json#L807-L809: update theListNodesdescription.api/management/v1/json/v1.json#L1369-L1371: update theGetNodedescription.
📍 Affects 1 file
api/management/v1/json/v1.json#L807-L809(this comment)api/management/v1/json/v1.json#L1369-L1371
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@api/management/v1/json/v1.json` around lines 807 - 809, Update the
is_pmm_internal_node descriptions in ListNodes at
api/management/v1/json/v1.json:807-809 and GetNode at
api/management/v1/json/v1.json:1369-1371 to state that internal nodes must not
receive delegated monitoring for remote addresses, while preserving that
local-address monitoring is permitted.
Comment on lines
+27695
to
+27698
| "pmm_agent_id": { | ||
| "description": "The pmm-agent identifier which should run agents. Defaults to the PMM Server's own pmm-agent.", | ||
| "type": "string", | ||
| "x-order": 26 |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Describe the selected exporter in pmm_agent_id.
pmm_agent_id selects the PMM agent that runs Azure exporters. The current description says “should run agents,” which can mislead API clients. Update the source .proto description, then regenerate this file.
As per path instructions, generated Swagger files must not be edited manually; update the source .proto and regenerate them.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@api/swagger/swagger.json` around lines 27695 - 27698, Update the source
.proto field corresponding to pmm_agent_id so its description clearly states
that it identifies the selected PMM agent running Azure exporters, then
regenerate api/swagger/swagger.json using the project’s prescribed generation
process; do not edit the generated Swagger file directly.
Source: Path instructions
Comment on lines
+199
to
+207
| pmmAgentID := models.PMMServerAgentID | ||
| if req.GetPmmAgentId() != "" { | ||
| pmmAgentID = req.GetPmmAgentId() | ||
| } | ||
| err := s.checkNodeIsEligible(ctx, pmmAgentID, req.Address) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Fix stale-agent notification for custom Azure agent assignment.
pmmAgentID (lines 199-207) can now be any pmm-agent, and the exporters/QAN agent are correctly created under pmmAgentID (lines 273, 284, 315). Line 337 still calls s.state.RequestStateUpdate(ctx, models.PMMServerAgentID).
When req.PmmAgentId names an agent other than models.PMMServerAgentID, this notifies the wrong pmm-agent. The pmm-agent that owns the new exporters does not get the refresh signal, so it does not start monitoring the new resources promptly.
Change the state-update call to use the selected agent:
🐛 Proposed fix
- s.state.RequestStateUpdate(ctx, models.PMMServerAgentID)
+ s.state.RequestStateUpdate(ctx, pmmAgentID)
return &managementv1.AddAzureDatabaseResponse{}, nil
}No test in this PR sets a non-default PmmAgentId on a non-internal Node for AddAzureDatabase and asserts RequestStateUpdate receives that agent, so this bug currently goes undetected. Do you want me to add that regression test alongside the fix?
#!/bin/bash
# Description: Confirm whether other Add* handlers notify the request's actual PmmAgentId (not a hardcoded constant).
rg -n -B5 'RequestStateUpdate\(ctx,' managed/services/management/*.goAlso applies to: 337-337
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@managed/services/management/azure_database.go` around lines 199 - 207, Update
the RequestStateUpdate call in the AddAzureDatabase flow to pass the selected
pmmAgentID rather than models.PMMServerAgentID, ensuring custom agent
assignments receive the refresh notification. Preserve the existing state-update
timing and error handling.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ticket number: PMM-14665
Feature build: Percona-Lab/pmm-submodules#4498
In an HA deployment the PostgreSQL cluster backing PMM's persistence layer runs pmm-client sidecars, so its pods were offered in the "Nodes" dropdown of Add Service as valid monitoring delegates. Those nodes are dedicated and must not take on any extra monitoring workloads.
PMM Server now accepts
PMM_INTERNAL_NODE_NAME_PREFIXES(set by the pmm-ha Helm chart), reports the nodes it matches asis_pmm_internal_nodeonListNodes/GetNode, and rejects requests which delegate monitoring of a remote address to an agent running on one of them.Local addresses (empty,
localhost,127.0.0.1,::1) stay allowed on those nodes: that is how the PostgreSQL operator's sidecar registers its own PostgreSQL and Patroni services on every pod start, so a blanket rejection would break the persistence layer's own monitoring. An airtight alternative is to key the check on the registering service account instead, which is deliberately deferred - see the ticket.AddAzureDatabaseRequestgainspmm_agent_id, which the UI has always sent but the server discarded, placing Azure exporters on the PMM Server unconditionally.PMM Server nodes stay eligible on purpose: until the Azure API accepts an agent, Azure monitoring cannot run anywhere else.
If this PR adds, removes or alters one or more API endpoints, please review and update the relevant API documentation as well:
If this PR is related to other PRs, contributions, or ongoing work in this or other repositories, please reference them here:
Summary by CodeRabbit
New Features
Bug Fixes