Draft: expose required Hydra PMI_FD handling for MANA checkpoint/restart

[alan-lira]

Refs #485

Why I am opening this draft in MANA

My complete multi-node MPICH/Hydra checkpoint/restart environment requires one additional condition in the DMTCP socket plugin bundled by MANA:

(getenv("PMI_FD"))

Without this condition, the complete validated environment does not checkpoint/restart correctly. With the complete modified stack, repeated chained checkpoint/restart tests pass across multiple nodes and ranks.

I am reporting the requirement to MANA first because it is exposed by MANA's MPICH/Hydra workflow and because MANA selects the DMTCP revision used by its releases. At the same time, I do not want to propose a final dependency on my personal DMTCP fork or assume that MANA must permanently carry a downstream patch.

What this draft contains

This branch adds the exact DMTCP patch as a normal file under contrib/dmtcp-patches, together with an idempotent helper that applies it to the checked-out submodule for local validation.

The relevant source change is:

if ((getenv("SLURM_JOBID")) ||
    (getenv("SLURM_JOB_ID")) ||
    (getenv("HYDI_CONTROL_FD")) ||
    (getenv("PMI_FD"))) {
  return;
}

The PR deliberately does not:

• point .gitmodules at my fork;
• change the MANA gitlink to a commit unavailable from the official DMTCP remote;
• commit dirty submodule contents;
• prescribe the permanent ownership model.

What I am asking from the maintainers

I would appreciate guidance on which final path the project prefers:

1. carry this patch temporarily in MANA;
2. implement an equivalent MANA-side compatibility solution;
3. ask me to submit the focused change directly to DMTCP;
4. coordinate the DMTCP change from the MANA project;
5. use another design preferred by the maintainers.

I am prepared to follow whichever route is most maintainable for the project.

Reproduction and validation

The full validation environment covers:

• native MPICH/Hydra execution;
• two or more physical nodes;
• multiple MPI ranks;
• persistent coordinator;
• blocking checkpoint;
• termination and distributed restart;
• rollback and forward-progress verification;
• continued MPI_Allreduce communication;
• repeated checkpoint/restart chains;
• custom MPI workloads;
• selected NAS Parallel Benchmarks.

Results included with the submission:

MANA commit: 7c3218629f4cb5f733f9d97c74b3c1b902b66c52
DMTCP commit: ff8e28969def414dcc5e51ae611412ca389b759f
MPICH version/configuration: MPICH 4.3.2, CH3:Nemesis device, Hydra process manager
Node/rank layout: 2 physical nodes, 2 MPI ranks total, 1 rank per node
Without required change: complete distributed checkpoint/restart did not succeed with MPICH/Hydra because PMI_FD was not handled by the DMTCP socket-connection logic
With required change: the complete two-node checkpoint/restart workflow succeeded, including checkpoint-image validation, rollback, MPI_Allreduce correctness, and continued forward progress
Repeated chained suite: all requested MPI application profiles passed chained repeated checkpoint/restart validation

Status

This is intentionally a draft proof-of-concept and ownership discussion. It makes the required source delta concrete and testable while leaving the final upstream/downstream decision to the maintainers.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 00c712b7-1040-4952-ba61-ac4f81c638b8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}