Skip to content

shinytest2 test-Distribution.R snapshot is stale, and the diff changes the analytical conclusion (WideSample/MaxMin path) #46

Description

@ms609

Surfaced by #45 (the MaxMin CI fix). Not caused by it in substance, but revealed by it — and the
revealed difference looks behavioural rather than cosmetic, so it wants a human eye before the
snapshot is simply re-recorded.

What happens

EasyTrees shinytest2 (windows-latest), run
30919105631:
FAIL 1 | WARN 1 | SKIP 0 | PASS 191.

Failure ('test-Distribution.R:9:5'): Distribution and concordance plots render correctly
Snapshot of `file` has changed.
Diff in snapshot file `DistributionDistribution-001-ClusterCons.zip`
< trees <- WideSample(trees[1:125], 48)      # stored snapshot
> trees <- WideSample(trees[77:125], 48)     # this run
...
< # No significant clustering was found.
> # Best clustering was part. around medoids:

Why #45 exposed it rather than caused it

Before #45 this job could not reach its tests: Set up R dependencies failed on the
unresolvable MaxMin Suggests (see #43). With that fixed, setup passes and the suite runs for
the first time in a while.

WideSample() is the one MaxMin-dependent export, and it stop()s when MaxMin is absent
(R/WideSample.R:149-152) — there is no silent fallback. So the stored snapshot was recorded in
an environment whose WideSample() behaved differently from a current, MaxMin-present one.

Why this needs judgement, not just snapshot_accept()

The two halves of the diff are not the same kind of change:

  1. trees[1:125]trees[77:125] — the input range in the app's generated script changed.
  2. "No significant clustering was found.""Best clustering was part. around medoids" — the
    analytical conclusion changed.

(2) plausibly follows from (1), and (1) is the part worth understanding: a different subset being
fed to WideSample() is either a deliberate consequence of MaxMin now being available, or a
symptom. Re-recording the snapshot would make the test green either way, which is exactly why it
should not be the first move.

Note also that R/WideSample.R is in the unowned statistics/support-metrics cluster identified in
#42 — so no red-team area owns this code path today.

Not a merge blocker for #45

The job is continue-on-error: true, deliberately (see shiny-tests-not-in-ci history: the
suite rotted while CI-unwired and was repaired in 6c341bce, then wired as a non-blocking
Windows job). gh pr checks reports it as fail, but it does not gate the workflow conclusion.
Filed separately so #45 stays a CI-plumbing change.

install-pandoc: false on that job — which #45 sets — is not implicated: the only pandoc
mentions in the log are the input echo and package sysreqs metadata, and the download test that
fails produces a zipped R script, not a rendered document. 191 tests pass with it unset.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingin-progressBeing fixed; claiming comment names the branch

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions