feat(perception): re-identify tracklets a tracker lost

[Moshiii]

Contribution path

• Small, safe change that does not need a tracking issue

Problem

A tracker associates across adjacent frames, so it breaks whenever continuity does:
an occlusion, a robot turning away, a detector miss. Every break makes a new object
where there was one. Nothing puts the pieces back.

Solution

Geometry vetoes, appearance decides. Two chairs of one model are identical to a
similarity model, so a threshold high enough to merge one chair's fragments also
merges two different chairs — and a wrong merge is worse than fragmentation, because
fragments are honest while a merge invents an object. What separates identical things
is where and when they were, so the cheap constraints run first and the embedding
comparison only sees pairs that survive them. On a 7.8-minute office capture that
refused 673,000 pairs before a single embedding was compared.

Ground truth needs no labelling: cut a tracklet the tracker held continuously and the
pieces are one object; two tracklets overlapping in time are two objects.

Pairwise: 100.0% precision, zero wrong merges. The 99.0% recall that stood
beside it was measured before the reachability veto moved to the gap boundary
(below) and wants one rerun.

The budget is spent on the gap, so it is measured across the gap

The speed allowance models how far a thing could travel while unseen, but the veto
compared whole-run centroids — charging a continuation for a walk that happened inside
its own tracklet, and forgiving a jump when two runs happened to average to the same
place. It now compares where the earlier run ended against where the later one resumed.
Precision is untouched: this benchmark's negatives are pairs overlapping in time,
refused before any distance is computed. Recall can move either way and has not been
remeasured. (Raised by Greptile.)

What is measured and what is not

	status
find_merges — pairs	measured, shipped defaults, table in the doc
connected — groups	unmeasured, opt-in
OnlineAssociator — ids	unmeasured, not for deployment

The grouping figures this PR originally carried (38.5% closure, 23.9% online) are
withdrawn. They were taken against a version that applied the instant-overlap veto
only between the two tracklets of a pair, never to the group a link would create — and
this benchmark's negative class is "overlapping in time". So every wrong merge it
counted was a pair the module's own hardest veto would have refused, had the grouping
path asked it.

The veto now runs group-wide, and co_occurring binds every member of an id rather
than one track of it. The benchmark cannot referee that change: its negative class and
the new check are the same predicate, so rerunning it scores the check against itself.
connected stays opt-in — now because grouping is unmeasured, not because it was
measured and failed. A real number needs a negative class that does not reduce to
co-visibility: pairs of one model never in frame together, labelled by hand, few enough
to label in an afternoon. docs/capabilities/perception/tracklet_reid.md keeps the
withdrawn rows struck through and says the rest.

What the geometry veto is worth

Upstream (EmbeddingIDSystem) vetoes only co-occurrence. Adding position and speed
refused 48 pairs upstream would have merged, all 48 correctly — things that look
alike and were never seen together, like two chairs of one model at opposite ends of a
room. The benchmark cannot score this: its negatives are defined as overlapping in
time, which co-occurrence already catches, so the 48 fall outside the labelled set.

Boundary

Embeddings arrive already computed by whatever model the caller picked, so the module
does not own that half — but it does own the boundary. Un-normalised vectors do not
lower a cosine, they inflate it (orthogonal tracklets scored 3.29 and merged at the
shipped threshold); a NaN row scores NaN and nan < threshold is False, so it merged
and was counted as nothing. Both are now refused, loudly, naming the tracklet.

numpy and the standard library, nothing else. No torch, no store, no message type, no
module system. Verified by import.

Open for the reviewer

1. OnlineAssociator overlaps dimos/perception/detection/reid/embedding_id_system.py
   — same 0.63, same 500-cap, same min-10, same top-k mean, same co-occurrence
   negatives — and the doc measures the two as identical on pairs. The genuinely new
   part is the geometry veto, ~15 lines. Is the right shape to add that veto to
   EmbeddingIDSystem and drop the second associator? That is ~170 lines deleted.
2. connected ships with no number behind it. Keeping it as published evidence is
   one call; deleting it until there is a benchmark that can score it is another.

Placement was the third question here and is settled: the module now sits in
dimos/perception/detection/reid/ beside EmbeddingIDSystem, and
docs/capabilities/perception/index.md links the page and says which of the two is
wired into a module. The packages above it carry no __init__, so the numpy-only
claim survives the move.

Prior automated review

This supersedes #3769, which was closed to reset a noisy force-push timeline; the code
is identical. Greptile reviewed it there and raised two P1s, both real and both fixed
here, each with a regression test that fails without the fix:

• Reachability compared whole-run centroids, not the sightings either side of the
  gap. A continuation that moved inside its own tracklet was refused; a jump across
  the gap was accepted when the two runs happened to average alike. See the section
  above — this is the one that put the recall figure back in the queue for a rerun.
• Eviction left the dropped key inside surviving exclusion sets. Trackers reuse
  ids, so the next holder of that key inherited a co-occurrence exclusion it never
  earned, and was refused a match it should have made.

How to Test

uv run pytest dimos/perception/detection/reid/test_tracklet_reid.py

33 tests, numpy and pytest only, no fixtures. Nine are regressions for the vetoes
above: a group never holds two tracklets seen at one instant; connected refuses
tracklets it was not given; an id does not absorb a track declared a different object;
movement inside a run is not charged to the gap; a jump across the gap is refused
though the runs average alike; a reused track id is not barred by the exclusion it
inherited; the track dropped is the least recently seen; un-normalised embeddings are
refused not rescaled; a Tracklet can go in a set. ruff and mypy clean.

There is no blueprint to run: the module is numpy in, dataclasses out, and has no
callers yet by design.

AI assistance

Claude Code with Opus 5, used as an implementation tool.

The approach is mine. Geometry vetoes before appearance decides. A wrong merge is worse
than fragmentation, so this fragments rather than guesses. Ground truth for
re-identification can be built by cutting tracklets the tracker held continuously
instead of by labelling anything. A negative result about grouping is worth publishing
rather than tuning away. What the vetoes are, what find_merges returns, why
connected is opt-in, and the decision to withdraw the grouping numbers rather than
rerun a benchmark that cannot referee its own check — those are my calls, and they are
what this PR is actually proposing.

Claude implemented against that design, and in a later pass audited the result: it
found the group-level veto gap, the NaN and normalisation fail-opens, and the
least-recently-seen eviction bug, then wrote those fixes and six regression tests.
Every number in this description was re-run except the withdrawn ones, which is why
they are withdrawn rather than restated. I have reviewed the diff and understand the code.

Checklist

• I have read and approved the CLA.

[greptile-apps]

Greptile Summary

This change introduces tracklet re-identification grouping and online association behavior, with focused tests and capability documentation. A reproduced failure in merge grouping causes valid generator-based merge inputs to produce no connected identity groups when tracklet validation is enabled. This should be corrected before merging.

Confidence Score: 4/5

Not safe to merge until one-shot merge iterables are preserved for both validation and grouping.

A minimal executable reproduction directly demonstrated that valid merge links disappear when passed as a generator with matching tracklets.

Files Needing Attention: dimos/perception/detection/reid/tracklet_reid.py needs attention at the merge-key validation and subsequent grouping iteration.

T-Rex Logs

What T-Rex did

• T-Rex produced a proof for a posted P1 finding and directed reviewers to the associated review comment for details.
• T-Rex produced a separate proof for a second posted P1 finding.
• T-Rex performed general contract validation and observed that the generator produced an empty result (returned_groups and generator_remaining were empty), then traced the behavior to the tracklet_reid.py implementation that consumes merges and then sorts the exhausted iterable.

_{Ran code and verified through T-Rex}

Comments Outside Diff (1)

1. General comment

   connected exhausts generator merges before grouping when tracklets are provided

   • Bug
     • Passing a one-shot generator of valid merges together with complete tracklets produces [] instead of the expected connected group.
   • Cause
     • At dimos/perception/detection/reid/tracklet_reid.py:313, the missing-key comprehension iterates merges. The later grouping loop at line 339 iterates the same iterable again, but a generator has already been exhausted.
   • Fix
     • Materialize merges once before the missing-key validation and reuse that collection for both validation and sorted grouping, or otherwise iterate it only once.

   _{Ran code and verified through T-Rex}

_{Reviews (1): Last reviewed commit: "feat(perception): re-identify tracklets ..." | Re-trigger Greptile}

[greptile-apps]

Validation exhausts merge generators

When callers provide tracklets and a one-shot iterable of Merge values, the missing-key check iterates through merges before the grouping loop runs. The subsequent sorted(merges, ...) therefore receives an exhausted generator and returns no connected groups, even for valid links. Materialize the iterable once before validation and reuse that collection for grouping.

Artifacts

Minimal generator reproduction script

• Python source creates matching tracklets and a one-shot generator of two valid merges, then prints the groups returned by connected; it directly exercises the reported generator path and provides the reproduction input.

Generator reproduction command output

• Captured output from running the minimal reproduction in the documented narrow environment shows returned_groups=[], the expected three-key group, and an exhausted generator; the claimed defect reproduces.

_{Ran code and verified through T-Rex}