Repository navigation
Support collect - #1632
Support collect#1632
Conversation
f3fbfa2 to
ce1674d
Compare
93e4da7 to
66348cb
Compare
c5093ce to
0fca608
Compare
0fca608 to
642954f
Compare
The support script was installed by the bin package, and its -p option relied on gpg being in the image only as a side effect of podman pulling in libgpgme. Give it a package of its own with an encrypt option that selects gnupg2, enabled in all non-minimal defconfigs, so the dependency is explicit. The WebUI calls the tool, so it selects the package. Signed-off-by: Mattias Walström <lazzer@gmail.com>
One wedged command stalled the whole collection, and the EXIT trap removed the log a failed run needed, which is why #1303 closed without a root cause. -o FILE writes the archive to a file so a dropped session does not lose the only copy, and the gpg passphrase reaches gpg on a private descriptor rather than its command line. Signed-off-by: Mattias Walström <lazzer@gmail.com>
f88e3c1 to
461f80a
Compare
troglobit
left a comment
troglobit
left a comment
There was a problem hiding this comment.
I must admit, I was a bit skeptical at first about the premise of doing this over an RPC -- and I'm still worried about the 60 sec. timeout coming to bite us -- but it turned out much better than I feared so it's a go from me, except for the documentation bits.
Nice work on all the side issues as well! 🌟
| A few things to know about this path: | ||
|
|
||
| - The RPC is denied by default (`nacm:default-deny-all`), so only groups | ||
| with an explicit NACM permit rule can call it. | ||
| - Collection runs in `/var/lib/support`, like the `support` command, | ||
| and the archive is removed once it has been returned, so a successful | ||
| call leaves nothing behind on the device. | ||
| - An archive above 16 MiB is not returned inline. The reply then holds | ||
| `size` and `filename` instead, and the file stays in `/var/lib/support` | ||
| for you to fetch. | ||
| - Over NETCONF the archive arrives as one base64 text node, and libxml2 | ||
| rejects text nodes over 10 MB unless the parser is opened with | ||
| `XML_PARSE_HUGE`. Clients built on it, lxml and ncclient among them, | ||
| need that option (`huge_tree=True`) to receive an archive above about | ||
| 7.5 MB. RESTCONF returns JSON and has no such limit. | ||
| - The system log is tailed for 5 seconds, rather than the 30 the command | ||
| line defaults to, so that the whole collection finishes inside the | ||
| client's RPC timeout (`CONFD_TIMEOUT` in `/etc/default/confd`, 60 | ||
| seconds by default). | ||
| - Pass `password` to get the archive GPG encrypted, for handing on to | ||
| someone else afterwards. The management session is already encrypted, | ||
| so this is not needed to protect the transfer itself. The password | ||
| must be a single line. It is handed to gpg on a private file | ||
| descriptor and never appears in the process list. Devices built | ||
| without the `BR2_PACKAGE_SUPPORT_ENCRYPT` option have no gpg and reject | ||
| the request. | ||
| - `confd` is busy for the duration of the collection, like it is during a | ||
| software upgrade, so a configuration change made at the same time has to | ||
| wait for the collection to finish. | ||
| - The RPC has 60 seconds to finish. If the collection takes longer, the | ||
| call fails with a timeout and nothing is kept, so there is no file to | ||
| look for afterwards. On a device where that happens, with many ports | ||
| or a lot of logging, log in over SSH and collect to a file instead: | ||
|
|
||
| ```bash | ||
| $ ssh admin@host 'sudo support collect -o /var/lib/support' | ||
| /var/lib/support/support-host-2026-09-11T13:05:42+02:00.tar.gz | ||
| $ scp admin@host:/var/lib/support/support-host-2026-09-11T13:05:42+02:00.tar.gz . | ||
| ``` | ||
| - A collection that fails keeps its log in `/var/lib/support`. The error | ||
| message says why and names the log. | ||
| - Each call removes what earlier calls left behind once it is a week | ||
| old, the same as `support clean` does. | ||
|
|
||
| From a shell on the device, use the `support` command rather than the | ||
| RPC. A base64 blob on your terminal is of no use to anyone. |
There was a problem hiding this comment.
This text is waaay too long, intricate, and talks too much about implementation details. Please simplify and focus on the core bits an end user would otherwise trip over.
Also, the SSH alternative is already documented so we should be able to just add an intra-link reference instead of repeating an example -- if the existing examples are incomplete, we should update them instead.
Me too, but it is mainly when doing it over NETCONF/RESTCONF, when you do it over the web, the connection is to localhost :) |
Lets clients that only speak the management protocol collect support data. Archives up to 16 MiB come back base64 encoded, larger ones stay in /var/lib/support, as does the log of a failed run, until a later call prunes them after a week. nacm:default-deny-all like install-bundle, the archive carries logs and the full configuration. The abort event sysrepo sends after a caller timeout is ignored, it used to run the collection twice. Signed-off-by: Mattias Walström <lazzer@gmail.com>
No transport could read an RPC reply, and ssh ran with LogLevel QUIET, which turned a dead transport into a bare exit code 255. That is why the support_collect flake in #1303 never got a root cause. Replies are parsed with huge_tree, libxml2 stops at 10 MB text nodes and the library's receive thread died silently on that. Signed-off-by: Mattias Walström <lazzer@gmail.com>
Streaming the archive over ssh made a dead transport look like a failed collection, and the collection.log fallback could never work since the tool removes the file. Signed-off-by: Mattias Walström <lazzer@gmail.com>
Exports a datastore without its secrets, the way NACM filters them for a user without read access. The models already mark what is secret, so new ones are covered as they come. The user password in ietf-system predates the convention and is matched by name. Signed-off-by: Mattias Walström <lazzer@gmail.com>
Private keys, password hashes and RADIUS secrets are of no use to support and a hazard in transit. Export the datastores with copy -r and drop the environment dump. --no-redact keeps them, the RPC always redacts. Signed-off-by: Mattias Walström <lazzer@gmail.com>
This is tracked by #1614 when it is fixed, this test should be removed, but for now, we hide the issue. No need to stop tests for this issue, that is tracked but unplanned.
The handler ran the tool itself, as root and past NACM, so any logged-in user could download the archive. The RPC runs as the user. Signed-off-by: Mattias Walström <lazzer@gmail.com>
The rootfs partitions had 27M of slack each while /var had only 84M usable, too little for a support archive once the container tests have left their images behind. Signed-off-by: Mattias Walström <lazzer@gmail.com>
A single lost ping during shutdown made wait_boot latch onto the old instance and then wait 20 min on a stale neighbor entry. Sleep the topology's shutdown_time before checking, and re-ping before each probe.
461f80a to
7f05f49
Compare
Really looking forward to your yangerd work here. Will speed up support tool a lot |
Yupp, any century now. |
2 participants
The main goal for this PR is to fix a security issue in the WebUI that support collect did not honor NACM, any user could get the support bunde (including all passwords). Now the support bundle honor NACM and also reduct all passwords, they are of no use for when debugging issues.
Description
Checklist
Tick relevant boxes, this PR is-a or has-a: