evetest: return errors from device file and publication reads

[europaul]

Description

The evetest read helpers -- EdgeDevice.ReadFile, ReadPublication and
ReadAllPublications -- ended the test through t.Fatalf on any failure.
That is the wrong contract for state EVE produces asynchronously: a caller
polling for a file or a publication that has not appeared yet dies on its
first attempt instead of failing one Eventually iteration and retrying.

Two call sites depend on polling today:

• tests/networking/dns_test.go reads /run/nim/dnsmasq.mgmt.servers inside
  an Eventually while nim is still writing it.
• tests/storage/zvol_provisioned_size_test.go reads volumemgr's
  AppDiskMetric the same way. ReadAllPublications was racy in a second
  way: it lists the topic directory and then reads each file, so an unpublish
  in between killed the test.

ReadPublication already supported polling, but reported absence as a bool
after a separate FileExists probe. That costs an extra SSH round trip per
read, races between the probe and the read, and collapses "not published yet"
and "device unreachable" into the same false. An error covers both and says
which -- and matches RunShellScript, the other way tests reach into a
device. Fatalf stays for harness faults, where no test can proceed.

ReadFile is broken on master today

While validating the above, ReadFile turned out to fail for every path,
not just as an edge case. scpFromEVE/scpToEVE shell-quote the remote path,
but scp has spoken the SFTP protocol by default since OpenSSH 9.0 and evetest
passes no -O to select the legacy one, so no remote shell ever sees the
path. The quotes become part of the file name:

scp: '/etc/resolv.conf': No such file or directory

The clearest demonstration is from TestZVolProvisionedSizeReported:
ReadAllPublications runs find on the device, which locates
/run/volumemgr/AppDiskMetric/dev-zvol-....json, and scp then reports that
exact path -- quoted -- as missing.

Spaces were the reason the quoting was added (types.DefaultVaultName is
"Application Data Store"), but they are already safe: exec runs scp directly
with no local shell, so the path reaches scp as a single argv element either
way. Verified against a throwaway sshd with the same OpenSSH 9.9 client the
evetest image ships, invoking scp with no shell so the quotes survive into
argv exactly as they do here:

remote-path argument	spaced path	plain path
shell-quoted (current master)	FAIL	FAIL
raw	OK	OK

This affects dns_test.go directly and vault_trim_test.go /
pciback_error_test.go through ReadPublication. It was previously masked:
scpFromEVE returned a bare exit status 1, and ReadFile turned that into
a Fatalf that named no cause.

Also in this PR

• scpFromEVE and scpToEVE now wrap scp's own stderr, which is what makes
  "missing file" distinguishable from "unreachable device" -- and what made
  the bug above legible in the first place.
• ReadAllPublications split find output with strings.Fields. find
  prints one path per line, and pubsub keys become file names, so a key
  containing a space was torn into several nonexistent paths. Splitting on
  whitespace happened to work only because every key read so far has been a
  UUID, "global", or a device path.
• evetest/VERSION 1.1 -> 1.2. tests/ is bind-mounted into the evetest
  container, but the framework it links against is compiled into the image,
  so a test written for the new signatures does not build until the image is
  rebuilt -- which only happens for a version the local docker daemon does
  not already have.

How to test and validate this PR

Both tests were run against a locally built live image from this branch
(make live, then EVETEST_EVE_LIVE_IMAGE=true). That matters: both assert
recent pillar fixes (24ef31510 zvol volsize, 2e739c88e mgmt dnsmasq) that
are absent from 17.0.0-lts, so against a published release they fail before
reaching any of this code.

make live
EVETEST_EVE_LIVE_IMAGE=true make evetest NAME=TestZVolProvisionedSizeReported  # PASSES
EVETEST_EVE_LIVE_IMAGE=true make evetest NAME=TestDNSFunctionality

• TestZVolProvisionedSizeReported passes, exercising
  ReadAllPublications end to end (error return and the newline split).
• TestDNSFunctionality gets through Phase 1 and Phase 2 -- Phase 2 is all
  three changed ReadFile calls, one of them inside an Eventually -- and
  then fails in Phase 3 on an unrelated assertion about EVE rejecting a
  broken-DNS DPC and falling back to the previous one. That phase reads
  device info from the controller and touches none of this code. I have not
  investigated it; flagging it rather than claiming this PR green.

Before the scp fix, both tests failed at the changed calls with the quoted
path error above, so the fix is what the runs actually demonstrate.

Changelog notes

No user-facing changes. Test-framework only.

PR Backports

- 17.0-stable: Eventually to be backported (the entire evetest dir).
- 16.0-stable: No, as evetest is not available there.
- 14.5-stable: No, as evetest is not available there.
- 13.4-stable: No, as evetest is not available there.

Checklist

• I've provided a proper description
• I've added the proper documentation
• I've tested my PR on amd64 device
• I've tested my PR on arm64 device
• I've written the test verification instructions
• I've set the proper labels to this PR

And the last but not least:

• I've checked the boxes above, or I've provided a good reason why I didn't
  check them.

Documentation: evetest/README.md is updated for the new signatures. Not
tested on arm64 -- the change is arch-independent and the local live image is
amd64.

[europaul]

@milan-zededa heads-up on a bug this turned up in current master, beyond the scope I opened the PR for.

scpFromEVE/scpToEVE shell-quote the remote path. scp has spoken SFTP by default since OpenSSH 9.0 and we pass no -O, so no remote shell ever sees the path — the quotes just become part of the file name, and every ReadFile fails, spaced path or not:

scp: '/etc/resolv.conf': No such file or directory

The clearest demonstration comes from TestZVolProvisionedSizeReported: ReadAllPublications runs find on the device, which locates /run/volumemgr/AppDiskMetric/dev-zvol-....json, and scp then reports that exact path — quoted — as missing.

It affects dns_test.go directly, and vault_trim_test.go / pciback_error_test.go through ReadPublication. It has been invisible because scpFromEVE returned a bare exit status 1 and ReadFile turned that into a Fatalf naming no cause; the stderr-capture commit here is what made it legible.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 24.49%. Comparing base (75e4099) to head (07fdfd3).
⚠️ Report is 12 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6279      +/-   ##
==========================================
+ Coverage   24.12%   24.49%   +0.36%     
==========================================
  Files         512      522      +10     
  Lines       93653    95369    +1716     
==========================================
+ Hits        22598    23359     +761     
- Misses      69267    70040     +773     
- Partials     1788     1970     +182     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andrewd-zededa]

Good, I just ran into this on #6257