New feature: Allow QuEST to use a user-provided MPI_Comm

[otbrown]

This is largely for my own nefarious purposes, but is possibly useful to anyone seeking to simulate a hybrid quantum-classical application which is non-trivially parallelised. This functionality allows the user to define a separate MPI communicator for QuEST and avoid resource contention between simultaneous QuEST and non-QuEST parts of the program.

This shouldn't be too difficult to implement, but the interface might require a bit of thought, and in particular an elegant way to distinguish when finalizeQuESTEnv() should and should not also finalise MPI could be tricky.

Requirements:

• User should be able to provide an MPI_Comm at initialisation which QuEST will use exclusively.
• finalizeQuESTEnv should not necessarily finalise MPI.
• Tests should be written for all new functionality.
• Should not break existing codes therefore original QuESTEnv interface should be preserved, and should default to MPI_COMM_WORLD.

[otbrown]

Probably sensible to extend initCustomQuESTEnv for this and let initQuESTEnv do what it's always done.

[otbrown]

I've tried to come up with a way of doing this that doesn't involve conditionally including mpi.h in environment.h/cpp, as I was rather fond of the fact that only the source code in comm ever knew or cared that MPI exists. Unfortunately, every solution I can come up with is deranged, so we may just have to take our #if COMPILE_MPI medicine. The sticking points are:

• MPI communicators are of type MPI_Comm.
• MPI_Comm is defined in mpi.h.
• The MPI metadata held by QuESTEnv, rank and numNodes, are both dependent on the specific MPI communicator being used.
• QuESTEnv is deliberately immutable after initialisation.
• For the user to set a custom MPI communicator there must be at least one user-facing interface which includes MPI_Comm, so the absolute separation of the QuEST interface from mpi.h will have to be broken somewhere.

[otbrown]

Okay, I think I now have the concept of a plan for how to implement this in a way that isn't horrible:

• Replace MPI_COMM_WORLD in all comm routines with mpiCommQuest which will be statically defined in comm_config.hpp.
  • This is best practice for libraries in any case.
  • By default mpiCommQuest will just be a duplicate of MPI_COMM_WORLD preserving original behaviour.
• The ability to set QuEST's communicator will be enabled iff a new CMake variable ENABLE_SUBCOMM is set.
  • The interface will be in a new public header file, and MPI will be a PUBLIC link library iff ENABLE_SUBCOMM is set.
  • That public header will be the only new place that #include <mpi.h> will be required.
• This approach will require QuESTEnv to be mutable, but we can hide this in a private interface that only exists if ENABLE_SUBCOMM is set, and it only needs to update the rank and numNodes members.
  • The update will just require calling comm_getRank() and comm_getNumNodes() again, as they will already be using mpiCommQuest (which will have been updated!).

[otbrown]

This is now mostly implemented in a fork at otbrown/QuEST/mpi-update. I still need to provide the user facing interface, but all the infrastructure's there I think.

Two positive side effects independent from the ability to set an MPI comm for QuEST from an MPI perspective:

• QuEST no longer explicitly uses MPI_COMM_WORLD.
• The user now has a way to positively assert that they will take responsibility for MPI, allowing it to be used independently before and after QuEST.

I won't open a PR until I have tests/docs etc., but for anyone looking for something like this in the interim, you know where to go!

Mini example of a user-owned MPI program:

#include <mpi.h>
#include "quest.h"

int main (void)
{
  const int USE_DISTRIB = 1;
  const int USER_MPI = 1;
  const int USE_OPENMP = 1;
  const int USE_GPU = 0;

  MPI_Init(NULL, NULL);
  initCustomMpiQuESTEnv(USE_DISTRIB, USER_MPI, USE_GPU, USE_OPENMP);
  reportQuESTEnv();
  finalizeQuESTEnv();
  MPI_Finalize();

  return 0;
}

[otbrown]

Okay, this is in prototype now over at the branch linked above, enabling programs of the form:

#include <cstdio>
#include <mpi.h>
#include <quest.h>

int main (void)
{
  int nprocs, quest_nprocs, world_rank, quest_rank;
  MPI_Comm comm_split, comm_quantum, comm_classical;

  MPI_Init(NULL, NULL);

  MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
  MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

  const int I_AM_QUANTUM = world_rank % 2;

  std::printf("[%d] Hello from rank %d of %d in MPI_COMM_WORLD.\n", world_rank, world_rank, nprocs);

  MPI_Comm_split(MPI_COMM_WORLD, I_AM_QUANTUM, world_rank, &comm_split);

  if (I_AM_QUANTUM) {
   MPI_Comm_dup(comm_split, &comm_quantum);
   MPI_Comm_size(comm_quantum, &quest_nprocs);
   MPI_Comm_rank(comm_quantum, &quest_rank);
   std::printf("[%d] Hello from rank %d of %d in comm_quantum.\n", world_rank, quest_rank, quest_nprocs);
  } else {
   MPI_Comm_dup(comm_split, &comm_classical);
   quest_rank = -1;
   quest_nprocs = -1;
  }

  // only procs in quantum comm initialise QuEST
  if (I_AM_QUANTUM) {
    std::printf("[%d] Initialising QuEST.\n", world_rank);
    initCustomMpiCommQuESTEnv(comm_quantum, modeflag::USE_AUTO, modeflag::USE_AUTO);

    reportQuESTEnv();

    std::printf("[%d] Finalising QuEST.\n", world_rank);
    finalizeQuESTEnv();
  }


  MPI_Comm_free(&comm_split);
  if (I_AM_QUANTUM) {
   MPI_Comm_free(&comm_quantum);
  } else {
   MPI_Comm_free(&comm_classical);
  }

  MPI_Finalize();

  return 0;
}

It should enable less stupid programs too ;)