Cyclic flow+sediment deadlocks at MPIsize>1 (collective-count desync in sedChange)

[tristan-salles]

Summary

Cyclic (periodic) boundaries run correctly in serial but deadlock at MPIsize ≥ 2 when sediment transport is active (erosion on). This is the second of two cyclic-parallel bugs; the first (a partition-dependent flat-vs-curved geometry/velocity branch, which broke cyclic advection) is fixed in #476. With #476 in place, cyclic advection is parallel-correct, but cyclic flow+sediment still hangs.

The serial-only guard (cyclic bc + MPIsize>1 → error, added in #475) stays until this is fixed.

Symptom

tests/fixtures/cyclic_cyl.yml (cylinder mesh, bc:'ococ', spl.K>0) under mpirun -n 2 (and -n 3, -n 4) hangs indefinitely (watchdog-killed at 90s). Serial (-n 1) completes fine.

Diagnosis (per-rank log files, to avoid mpirun stdout buffering)

The two ranks desync by one or more collectives — a classic collective-count mismatch → deadlock:

• rank 0 blocks at the post-deposition localToGlobal/globalToLocal in sedplex._updateSinks (logged updateSinks: before applyDeposit, never reaches after applyDeposit).
• rank 1 has already passed that, finished _updateSinks, entered seaplex.seaChange, and blocks in _distanceCoasts → _globalCoastsTree's Allreduce/Allgatherv.

So from some point in the sedChange pipeline onward, one rank issues a different number of MPI collective calls than the other on the cyclic mesh. By the time they reach _updateSinks's post-deposit localToGlobal, they are permanently offset.

Ruled out

• With no active pits (active=0 on both ranks — confirmed) _updateSinks itself is symmetric, so the divergence is upstream: in _getSedFlux, fillElevation(sed=True) (a second pit-filling on the sediment-filled topology), or the _distributeSediment/_moveDownstream cascade.
• The .any()/early-return gates in _updateSinks/_diffuseLargePit/_addPitMicroTilt are collective-safe (they key on global per-pit data — pitParams/pitVol), so it's NOT those.
• It's not the geometry branch fixed in fix(mesh): partition-invariant flat-vs-curved branch (fixes cyclic advection in parallel) #476 (that fix is in; advection is now consistent; this hang remains).

Suggested next step

Systematic per-rank collective-count audit of the sedChange path: instrument every MPI.*reduce/Allgatherv/bcast, dm.localToGlobal/globalToLocal, and ksp.solve with a per-rank counter, run cyclic_cyl.yml at np=2, and diff the counts to find the first divergent call. The prime suspects are a while-loop whose iteration count differs per rank, or a collective gated by a rank-local condition, somewhere in _getSedFlux / fillElevation(sed=True) / _moveDownstream.

Once fixed, add parallel cyclic regression tests (np=2/3/4) — all current cyclic tests are serial-only (the conftest fixtures are single-rank), which is why this shipped undetected — and lift the serial-only guard in inputparser._readDomain.

References

• Guard: PR perf(tecto): cache + single-pass 2D advection; fix IIOE2; guard cyclic in parallel #475. Advection fix (root cause Initial Update #1): PR fix(mesh): partition-invariant flat-vs-curved branch (fixes cyclic advection in parallel) #476. Cyclic feature origin: PR feat(flow): flat-model boundaries (o/f/w/c, NESW) + cyclic advection; gflex teardown #473.
• Fixtures: tests/fixtures/cyclic_cyl.{npz,yml}.
• Code: gospl/sed/sedplex.py (sedChange/_distributeSediment/_moveDownstream/_updateSinks), gospl/sed/seaplex.py (seaChange/_distanceCoasts), gospl/flow/pitfilling.py (fillElevation).

[github-actions]

🤖 This is an automated response from the goSPL triage bot. A maintainer will follow up if further attention is needed.

Triage

Label: bug, parallel, cyclic-boundaries, sediment-transport, help-wanted
Priority: Medium — serial behaviour is correct and the serial-only guard in inputparser._readDomain (PR #475) prevents silent incorrect results or crashes in the meantime. No existing parallel workflow is broken; this is a known limitation blocking the lift of that guard.
Affects: dev branch only (cyclic BC is a dev-only feature, not in any released conda/PyPI package).

---

What is happening

The deadlock is a collective-count mismatch: once the two ranks diverge by one MPI collective call anywhere in the sedChange pipeline, every subsequent collective permanently blocks. The diagnosis in the issue is well-grounded.

The most likely culprits, in rough order of suspicion:

1. while-loop iteration count differs per rank

_moveDownstream and fillElevation(sed=True) both contain convergence loops whose termination condition is evaluated from rank-local data (e.g. residual sediment remaining in local nodes, local pit-fill progress). If any loop body contains a collective (a localToGlobal, dm.globalToLocal, ksp.solve, or Allreduce) and the two ranks exit the loop after different numbers of iterations, they deadlock on the first collective inside the mismatched iteration.

Check: grep for while loops in sedplex._moveDownstream, pitfilling.fillElevation, and sedplex._distributeSediment. For each one, verify that the loop-exit condition is derived from a globally-reduced quantity (e.g. MPI.COMM_WORLD.allreduce(local_residual, op=MPI.SUM)) rather than a per-rank local residual.

2. Collective gated on a rank-local condition

From AGENTS.md (the coastline-gated cache rebuild warning):

The "did the coastline move?" test MUST be reduced across ranks … without the reduce, one rank rebuilds while another reuses, the two take different collective paths, and the run deadlocks at np>1.

The same pattern can appear anywhere in the sediment path. Any if local_condition: dm.localToGlobal(...) or if len(local_pits) > 0: ksp.solve(...) that is NOT preceded by an Allreduce/bcast to ensure all ranks take the same branch is a potential site.

Check: in _getSedFlux, _distributeSediment, _updateSinks, look for collectives inside if blocks whose predicate reads self.seaID, self.pitIDs, self.lsink, or any other rank-local array.

3. fillElevation(sed=True) collective structure on the cyclic mesh

The second pit-fill (on the sediment-filled topology) calls into pitfilling.fillElevation. On the cyclic mesh, the seam nodes appear on both sides of the partition boundary. If _fillFromEdges or _unifyLabels emits a different number of Allgatherv/Allreduce calls depending on whether the partition sees seam-crossing spillways (which can differ per rank on a cylinder mesh), the count diverges.

Check: run with active=0 pits confirmed (as noted in the issue), then temporarily disable fillElevation(sed=True) (replace with a no-op) to see if the deadlock moves downstream or disappears — this isolates whether the fill step itself is the source.

---

Recommended debugging approach

The issue already outlines the right method. Here is a concrete recipe:

# Temporary instrumentation — add to gospl/sed/sedplex.py and gospl/sed/seaplex.py
from mpi4py import MPI
_coll_count = [0]

def _coll(label):
    _coll_count[0] += 1
    rank = MPI.COMM_WORLD.Get_rank()
    with open(f"rank{rank}_collectives.log", "a") as f:
        f.write(f"{_coll_count[0]:06d}

---
*You can ask follow-up questions by commenting `@gospl-bot <your question>`.*