Chap_API_Sets_Groups.tex

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Chapter: Process Sets and Groups
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Process Sets and Groups}
\label{chap:api_sets_groups}

\ac{PMIx} supports two slightly related, but functionally different concepts
known as \emph{process sets} and \emph{process groups}. This chapter defines
these two concepts and describes how they are utilized, along with their
corresponding \acp{API}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Process Sets}
\label{chap:api_sets_groups:sets}

A \ac{PMIx} \emph{Process Set} is a user-provided or host environment assigned
label associated with a given set of application processes. Processes can
belong to multiple process \emph{sets} at a time. Users may define a \ac{PMIx}
process set at time of application execution. For example, if using the command line parallel launcher "prun", one could specify process sets as follows:

\cspecificstart
\begin{codepar}
\$ prun -n 4 --pset ocean myoceanapp : -n 3 --pset ice myiceapp
\end{codepar}
\cspecificend

In this example, the processes in the first application will be labeled with a \refattr{PMIX_PSET_NAMES} attribute with a value of \emph{ocean} while those in the second application will be labeled with an \emph{ice} value. During the execution, application processes could lookup the process set attribute for any process using \refapi{PMIx_Get}. Alternatively, other executing applications could utilize the \refapi{PMIx_Query_info} \acp{API} to obtain the number of declared process sets in the system, a list of their names, and other information about them. In other words, the \emph{process set} identifier provides a label by which an application can derive information about a process and its application - it does \emph{not}, however, confer any operational function.

Host environments can create or delete process sets at any time through the
\refapi{PMIx_server_define_process_set} and
\refapi{PMIx_server_delete_process_set} \acp{API}. \ac{PMIx} servers shall
notify all local clients of process set operations via the
\refconst{PMIX_PROCESS_SET_DEFINE} or \refconst{PMIX_PROCESS_SET_DELETE}
events.

Process \emph{sets} differ from process \emph{groups} in several key ways:

\begin{itemize}
    \item Process \emph{sets} have no implied relationship between their members - i.e., a process in a process set has no concept of a ``pset rank'' as it would in a process \emph{group}.
    %
    \item Process \emph{set} identifiers are set by the host environment or by the user at time of application submission for execution -
    there are no \ac{PMIx} \acp{API} provided by which an application can define a process set or
    change a process \emph{set} membership. In contrast, \ac{PMIx} process
    \emph{groups} can only be defined dynamically by the application.
    %
    \item Process \emph{sets} are immutable - members cannot be added or removed once the set has been defined. In contrast, \ac{PMIx} process \emph{groups} can dynamically change their membership using the appropriate \acp{API}.
    %
    \item Process \emph{groups} can be used in calls to \ac{PMIx} operations. Members of process \emph{groups} that are involved in an operation are translated by their \ac{PMIx} server into their \emph{native} identifier prior to the operation being passed to the host environment. For example, an application can define a process group to consist of ranks 0 and 1 from the host-assigned namespace of \emph{210456}, identified by the group id of \emph{foo}. If the application subsequently calls the \refapi{PMIx_Fence} \ac{API} with a process identifier of \code{\{foo, PMIX_RANK_WILDCARD\}}, the \ac{PMIx} server will replace that identifier with an array consisting of \code{\{210456, 0\}} and \code{\{210456, 1\}} - the host-assigned identifiers of the participating processes - prior to processing the request.
    %
    \item Process \emph{groups} can request that the host environment assign a unique \code{size_t} \ac{PGCID} to the group at time of group construction. An \ac{MPI} library may, for example, use the \ac{PGCID} as the \ac{MPI} communicator identifier for the group.
    %
\end{itemize}

The two concepts do, however, overlap in that they both
involve collections of processes. Users desiring to create a process group
based on a process set could, for example, obtain the membership array of the
process set and use that as input to \refapi{PMIx_Group_construct}, perhaps
including the process set name as the group identifier for clarity. Note that
no linkage between the set and group of the same name is implied nor
maintained - e.g., changes in process group membership can not be
reflected in the process set using the same identifier.

\advicermstart
The host environment is responsible for ensuring:

\begin{itemize}
    \item consistent knowledge of process set membership across all involved
    \ac{PMIx} servers; and
    \item that process set names do not conflict with system-assigned namespaces within the scope of the set.
\end{itemize}

\advicermend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Process Set Constants}

\versionMarker{4.0}
The \ac{PMIx} server is required to send a notification to all local clients upon creation or deletion of process sets. Client processes wishing to receive such
notifications must register for the corresponding event:

\begin{constantdesc}
%
\declareconstitemvalue{PMIX_PROCESS_SET_DEFINE}{-55}
The host environment has defined a new process set - the event will include the process set name (\refattr{PMIX_PSET_NAME}) and the membership (\refattr{PMIX_PSET_MEMBERS}).
%
\declareconstitemvalue{PMIX_PROCESS_SET_DELETE}{-56}
The host environment has deleted a process set - the event will include the process set name (\refattr{PMIX_PSET_NAME}).
%
\end{constantdesc}


%%%%%%%%%%%
\subsection{Process Set Attributes}

\versionMarker{4.0}
Several attributes are provided for querying the system regarding process sets using the \refapi{PMIx_Query_info} \acp{API}.

%
\declareAttribute{PMIX_QUERY_NUM_PSETS}{"pmix.qry.psetnum"}{size_t}{
Return the number of process sets defined in the specified range (defaults
to \refconst{PMIX_RANGE_SESSION}).
}
%
\declareAttribute{PMIX_QUERY_PSET_NAMES}{"pmix.qry.psets"}{pmix_data_array_t*}{
Return a \refstruct{pmix_data_array_t} containing an array of strings of the
process set names defined in the specified range (defaults to \refconst{PMIX_RANGE_SESSION}).
}
%
\declareAttribute{PMIX_QUERY_PSET_MEMBERSHIP}{"pmix.qry.pmems"}{pmix_data_array_t*}{
Return an array of \refstruct{pmix_proc_t} containing
the members of the specified process set.
}
%

\vspace{\baselineskip}
The \refconst{PMIX_PROCESS_SET_DEFINE} event shall include the name of the newly defined process set and its members:
%
\declareAttribute{PMIX_PSET_NAME}{"pmix.pset.nm"}{char*}{
The name of the newly defined process set.
}
%
\declareAttribute{PMIX_PSET_MEMBERS}{"pmix.pset.mems"}{pmix_data_array_t*}{
An array of \refstruct{pmix_proc_t} containing
the members of the newly defined process set.
}

\vspace{\baselineskip}
In addition, a process can request (via \refapi{PMIx_Get}) the process sets to which a given process (including itself) belongs:

%
\declareAttribute{PMIX_PSET_NAMES}{"pmix.pset.nms"}{pmix_data_array_t*}{
Returns an array of \code{char*} string names of the process sets in which the given process is a member.
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Process Groups}
\label{chap:api_sets_groups:groups}

\ac{PMIx} \emph{Groups} are defined as a collection of processes desiring a common, unique identifier for operational purposes such as passing events or participating in \ac{PMIx} fence operations. As with processes that assemble via \refapi{PMIx_Connect}, each member of the group is provided with both the job-level information of any other namespace represented in the group, and the contact information for all group members.

However, members of \ac{PMIx} Groups are \emph{loosely coupled} as opposed to \emph{tightly connected} when constructed via \refapi{PMIx_Connect}. Thus, \emph{groups} differ from \refapi{PMIx_Connect} assemblages in several key areas, as detailed in the following sections.

\subsection{Relation to the host environment}

Calls to \ac{PMIx} Group \acp{API} are first processed within the local \ac{PMIx} server. When constructed, the server creates a tracker that associates the specified processes with the user-provided group identifier, and assigns a new \emph{group rank} based on their relative position in the array of processes provided in the call to \refapi{PMIx_Group_construct}. Members of the group can subsequently utilize the group identifier in \ac{PMIx} function calls to address the group’s members, using either \refconst{PMIX_RANK_WILDCARD} to refer to all of them or the group-level rank of specific members. The \ac{PMIx} server will translate the specified processes into their \ac{RM}-assigned identifiers prior to passing the request up to its host. Thus, the host environment has no visibility into the group’s existence or membership.

In contrast, calls to \refapi{PMIx_Connect} are relayed to the host environment. This means that the host \ac{RM} should treat the failure of any process in the specified assemblage as a reportable event and take appropriate action. However, the environment is not required to define a new identifier for the connected assemblage or any of its member processes, nor does it define a new rank for each process within that assemblage. In addition, the \ac{PMIx} server does not provide any tracking support for the assemblage. Thus, the caller is responsible for addressing members of the connected assemblage using their \ac{RM}-provided identifiers.

\adviceuserstart
User-provided group identifiers must be distinct from both other group identifiers within the system and namespaces provided by the \ac{RM} so as to avoid collisions between group identifiers and \ac{RM}-assigned namespaces. This can usually be accomplished through the use of an application-specific prefix – e.g., ``myapp-foo''
\adviceuserend


\subsection{Construction procedure}
\label{chap:api_sets_groups:construct}

\refapi{PMIx_Connect} calls require that every process call the \ac{API} before completing – i.e., it is modeled upon the bulk synchronous traditional \ac{MPI} connect/accept methodology. Thus, a given application thread can only be involved in one connect/accept operation at a time, and is blocked in that operation until all specified processes participate. In addition, there is no provision for replacing processes in the assemblage due to failure to participate, nor a mechanism by which a process might decline participation.

In contrast, \ac{PMIx} Groups are designed to be more flexible in their construction procedure by relaxing these constraints. For purposes of constructing \ac{PMIx} groups, \ac{PMIx} defines two classes of group members:

\begin{itemize}
\item \emph{leaders} have some global view of the group at time of
construction. This might consist of knowing the number of
leaders in the group, or knowing the process IDs of all
group members. All leaders must, of course, know the group
ID they are attempting to construct.

\item \emph{members} know only that they are to participate in a given
group ID, but have incomplete or no knowledge of the full group. For example,
a member may not know how many processes will be in the group
or any of their process IDs. The only requirement for membership
is that the process know the group ID to which they are to belong.
\end{itemize}

Within that context, \ac{PMIx} supports three methods for constructing \ac{PMIx} groups (we will use the
\refapi{PMIx_Group_construct} as an example, but the explanation applies equally to the non-blocking
form of that function):

\begin{itemize}
\item Collective method. The collective method is considered the more traditional form of the construct operation
but requires all group members to know the process ID
of all other members prior to calling the API. In essence, this method results in all participants serving as
group \textit{leaders}, with each participant providing the full group membership to \refapi{PMIx_Group_construct}.
The ordering of process IDs in the group construct call is not significant.

Since all participants are equivalent in
this method, \ac{PMIx} provides an attribute (\refattr{PMIX_GROUP_LEADER}) by which one or more processes can declare themselves
to be an overall \textit{leader} of the collective operation. This identifies the process(es) that should be notified
if any participant fails to participate within a specified timeout, or abnormally terminates prior to
participating in the construct operation. If a participant declares itself as the collective leader, then it is the
responsibility of that process (or collection of processes) to deal with the related events. In the absence of one or more collective leaders, the host
environment will notify all participants of the failure to complete the construct operation by returning an
error from the \refapi{PMIx_Group_construct} \ac{API}. Once all participants have participated, any further failures will
result in return of an error from the \ac{API} without event notification of leaders.

\item Bootstrap method. In contrast,
the bootstrap method is a somewhat more dynamic form of the construct operation
that assumes each leader only knows the number of group leaders,
but does not know their process IDs. This is commonly the case when
two or mote collections of processes wish to join together (e.g., in an
MPI connect/accept operation), but only one process in each collection
(commonly denoted in MPI as the \textit{root} process) know of each other.
In \ac{PMIx} terminology, these processes are group \textit{leaders}. As
the leaders know of each other, they therefore know how many leaders are
participating. In addition, each leader typically knows the
process ID of all processes in its collection. It is therefore able to
call the group construct operation specifying only their own process ID
in the \code{procs} array, while including the \refattr{PMIX_GROUP_BOOTSTRAP}
attribute specifying the number of leaders that will participate plus
the \refattr{PMIX_GROUP_ADD_MEMBERS} attribute identifying the process IDs
of the other processes in its collection.

Processes listed as "add members" participate in the construction by
calling \refapi{PMIx_Group_construct} with a \code{NULL} for the
\code{procs} parameter. This indicates that they are not participating
as a leader, but still wish to belong to the resulting group. The host
environment is responsible for ensuring that all members (leaders and
"add" members) have participated before declaring the construction
complete.

Note that specific processes can be included more than once as "add members"
by a given leader (e.g., specifying a namespace with a \refconst{PMIX_RANK_WILDCARD}
and also specifying the process ID itself) or across multiple leaders. It is the
responsibility of the host environment to resolve any overlap.

Similar to the collective method, leaders in the bootstrap method can provide the \refattr{PMIX_GROUP_LEADER} attribute
to request notification of participant failures, thereby allowing the application to deal with such events. In the
absence of one or more collective leaders, the host
environment will notify all participants of the failure to complete the construct operation by returning an
error from the called \ac{API}. Once all participants have participated, any further failures will
result in return of an error from the \ac{API} without event notification of leaders.

\item Invite method. \ac{PMIx} also supports dynamic definition of group membership based on an invite/join model. A process can asynchronously initiate construction of a group of any processes via the \refapi{PMIx_Group_invite} function call. Invitations are delivered via a \ac{PMIx} event (using the \refconst{PMIX_GROUP_INVITED} event) to the invited processes which can then either accept or decline the invitation using the \refapi{PMIx_Group_join} \ac{API}. The initiating process tracks responses by registering for the events generated by the call to \refapi{PMIx_Group_join}, timeouts, or process terminations, optionally replacing processes that decline the invitation, fail to respond in time, or terminate without responding.
\end{itemize}

Regardless of the chosen method, the final group membership is communicated to each member of the new group upon completion of the construct operation.

\subsection{Destruct procedure}

Analogous to the construct operation, \ac{PMIx} provides two methods for destructing the group:

\begin{itemize}
\item Collective method. This operation consists of each process calling \refapi{PMIx_Group_destruct} or its non-blocking equivalent. In this method, the
group is considered dissolved once all members have participated in the group destruct operation. This is akin to \refapi{PMIx_Disconnect} for the connect
operation.

\item Asynchronous method. Members of a \ac{PMIx} Group may depart the group at any time via the \refapi{PMIx_Group_leave} \ac{API}. Other members are notified of the departure via the \refconst{PMIX_GROUP_LEFT} event to distinguish such events from those reporting process termination. This leaves the remaining members free to continue group operations.
\end{itemize}

In contrast, processes that assemble via \refapi{PMIx_Connect} must all depart the assemblage together – i.e., no member can depart the assemblage while leaving the remaining members in it. Even the non-blocking form of \refapi{PMIx_Disconnect} retains this requirement in that members remain a part of the assemblage until all members have called \refapi{PMIx_Disconnect_nb}

Note that applications supporting dynamic group behaviors such as asynchronous departure take responsibility for ensuring global consistency in the group definition prior to executing group collective operations - i.e., it is the application's responsibility to either ensure that knowledge of the current group membership is globally consistent across the participants, or to register for appropriate events to deal with the lack of consistency during the operation.

\adviceuserstart
The reliance on \ac{PMIx} events in the \ac{PMIx} Group concept dictates that processes utilizing these \acp{API} must register for the corresponding events. Failure to do so will likely lead to operational failures. Users are recommended to utilize the \refattr{PMIX_TIMEOUT} directive (or retain an internal timer) on calls to \ac{PMIx} Group \acp{API} (especially the blocking form of those functions) as processes that have not registered for required events will never respond.
\adviceuserend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Process Group Events}

\versionMarker{4.0}
Asynchronous process group operations rely heavily on \ac{PMIx} events.  The following events have been defined for that purpose.

\begin{constantdesc}
%
\declareconstitemvalue{PMIX_GROUP_INVITED}{-159}
The process has been invited to join a \ac{PMIx} Group - the identifier of the group and the ID's of other invited (or already joined) members will be included in the notification.
%
\declareconstitemvalue{PMIX_GROUP_LEFT}{-160}
A process has asynchronously left a \ac{PMIx} Group - the process identifier of the departing process will in included in the notification.
%
\declareconstitemvalue{PMIX_GROUP_MEMBER_FAILED}{-170}
A member of a \ac{PMIx} Group has abnormally terminated (i.e., without formally leaving the group prior to termination) - the process identifier of the failed process will be included in the notification.
%
\declareconstitemvalue{PMIX_GROUP_INVITE_ACCEPTED}{-161}
A process has accepted an invitation to join a \ac{PMIx} Group - the identifier of the group being joined will be included in the notification.
%
\declareconstitemvalue{PMIX_GROUP_INVITE_DECLINED}{-162}
A process has declined an invitation to join a \ac{PMIx} Group - the identifier of the declined group will be included in the notification.
%
\declareconstitemvalue{PMIX_GROUP_INVITE_FAILED}{-163}
An invited process failed or terminated prior to responding to the invitation - the identifier of the failed process will be included in the notification.
%
\declareconstitemvalue{PMIX_GROUP_MEMBERSHIP_UPDATE}{-164}
The membership of a \ac{PMIx} group has changed - the identifiers of the revised membership will be included in the notification.
%
\declareconstitemvalue{PMIX_GROUP_CONSTRUCT_ABORT}{-165}
Any participant in a \ac{PMIx} group construct operation that returns \refconst{PMIX_GROUP_CONSTRUCT_ABORT} from the \emph{leader failed} event handler will cause all participants to receive an event notifying them of that status. Similarly, the leader may elect to abort the procedure by either returning this error code from the handler assigned to the \refconst{PMIX_GROUP_INVITE_ACCEPTED} or \refconst{PMIX_GROUP_INVITE_DECLINED} codes, or by generating an event for the abort code. Abort events will be sent to all invited or existing members of the group.
%
\declareconstitemvalue{PMIX_GROUP_CONSTRUCT_COMPLETE}{-166}
The group construct operation has completed - the final membership will be included in the notification.
%
\declareconstitemvalue{PMIX_GROUP_LEADER_FAILED}{-168}
The current \emph{leader} of a group including this process has abnormally terminated - the group identifier will be included in the notification.
%
\declareconstitemvalue{PMIX_GROUP_LEADER_SELECTED}{-167}
One or more new \emph{leaders} of a group including this process have been selected - the identifier(s) of the new leader(s) will be included in the notification.
%
\declareconstitemvalue{PMIX_GROUP_CONTEXT_ID_ASSIGNED}{-169}
A new \ac{PGCID} has been assigned by the host environment to a group that includes this process - the group identifier will be included in the notification.
%
\end{constantdesc}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Process Group Attributes}

\versionMarker{4.0}
Attributes for querying the system regarding process groups include:

%
\declareAttribute{PMIX_QUERY_NUM_GROUPS}{"pmix.qry.pgrpnum"}{size_t}{
Return the number of process groups defined in the specified range (defaults
to session). OPTIONAL QUALIFERS: \refattr{PMIX_RANGE}.
}
%
\declareAttribute{PMIX_QUERY_GROUP_NAMES}{"pmix.qry.pgrp"}{pmix_data_array_t*}{
Return a \refstruct{pmix_data_array_t} containing an array of string names of
the process groups defined in the specified range (defaults to session). OPTIONAL QUALIFERS: \refattr{PMIX_RANGE}.
}
%
\declareAttribute{PMIX_QUERY_GROUP_MEMBERSHIP}{"pmix.qry.pgrpmems"}{pmix_data_array_t*}{
Return a \refstruct{pmix_data_array_t} of \refstruct{pmix_proc_t} containing
the members of the specified process group. REQUIRED QUALIFIERS: \refattr{PMIX_GROUP_ID}.
}
%

\vspace{\baselineskip}
The following attributes are used as directives in \ac{PMIx} Group operations:

\declareAttribute{PMIX_GROUP_ID}{"pmix.grp.id"}{char*}{
User-provided group identifier - as the group identifier may be used in
\ac{PMIx} operations, the user is required to ensure that the provided ID is unique within the scope of the host environment (e.g., by including some user-specific or application-specific prefix or suffix to the string).
}
%
\declareAttribute{PMIX_GROUP_LEADER}{"pmix.grp.ldr"}{bool}{
This process is a leader of the group.
}
%
\declareAttribute{PMIX_GROUP_OPTIONAL}{"pmix.grp.opt"}{bool}{
Participation is optional - do not return an error if any of the specified processes terminate without having joined. The default is \code{false}.
}
%
\declareAttribute{PMIX_GROUP_NOTIFY_TERMINATION}{"pmix.grp.notterm"}{bool}{
Notify remaining members when another member terminates without first leaving the group.
}
%
\declareAttribute{PMIX_GROUP_FT_COLLECTIVE}{"pmix.grp.ftcoll"}{bool}{
Adjust internal tracking on-the-fly for terminated processes during a \ac{PMIx} group collective operation.
}
%
\declareAttribute{PMIX_GROUP_MEMBERSHIP}{"pmix.grp.mbrs"}{pmix_data_array_t*}{
Array \refstruct{pmix_proc_t} identifiers identifying the members of the specified group.
}
%
\declareAttribute{PMIX_GROUP_ASSIGN_CONTEXT_ID}{"pmix.grp.actxid"}{bool}{
Requests that the \ac{RM} assign a new context identifier to the newly created group. The identifier is an unsigned, \code{size_t} value that the \ac{RM} guarantees to be unique across the range specified in the request. Thus, the value serves as a means of identifying the group within that range. If no range is specified, then the request defaults to \refconst{PMIX_RANGE_SESSION}.
}
%
\declareAttribute{PMIX_GROUP_LOCAL_ONLY}{"pmix.grp.lcl"}{bool}{
Group operation only involves local processes. \ac{PMIx} implementations are \textit{required} to automatically scan an array of group members for local vs remote processes - if only local processes are detected, the implementation need not execute a global collective for the operation unless a context ID has been requested from the host environment. This can result in significant time savings. This attribute can be used to optimize the operation by indicating whether or not only local processes are represented, thus allowing the implementation to bypass the scan.
}
%
\versionMarkerProvisional{6.0}
\declareAttributeProvisional{PMIX_GROUP_LOCAL_CID}{"pmix.grp.lclid"}{size_t}{
Local context ID for the participating process member of a group. Provided
in the \refstruct{pmix_info_t} directives when the process calls construct.
These values are to be returned to all participants with the collected data
at completion of construct. Values can be retrieved via the \refapi{PMIx_Get}
\ac{API} using the process ID of the process that posted the value.
}
%
\versionMarkerProvisional{6.0}
\declareAttributeProvisional{PMIX_GROUP_BOOTSTRAP}{"pmix.grp.btstrp"}{size_t}{
Group construct call will not include all members as not all individual
participants know the entire membership list. Leaders provide only their
own process ID in the \code{procs} parameter to the construct
\ac{API}, and are \textit{required} to include the \refattr{PMIX_GROUP_BOOTSTRAP}
attribute in their array of \refstruct{pmix_info_t} directives, with
the value in that attribute set to equal the number
of leaders in the group construct operation. They may also provide the
\refattr{PMIX_GROUP_ADD_MEMBERS} attribute with an array of process IDs that are to
belong to the final group - each of those processes will also call the group
construct, but with a \code{NULL} process ID to indicate they are joining
as "add members" and not leaders. Construction will complete once all
leaders and "add members" have participated.
}
%
\versionMarkerProvisional{6.0}
\declareAttributeProvisional{PMIX_GROUP_ADD_MEMBERS}{"pmix.grp.add"}{pmix_data_array_t*}{
Array of \refstruct{pmix_proc_t} identifying procs that are not
included in the membership specified in the \code{procs} array passed to
the group construct call, but are to be included in the
final group. See \refattr{PMIX_GROUP_BOOTSTRAP} or the \hyperref[chap:api_sets_groups:construct]{construction section}
for a full description of the associated operation. The order of the process IDs in the array
is not significant.
}
%

\vspace{\baselineskip}
The following attributes are used to return information at the conclusion of a \ac{PMIx} Group operation and/or in event notifications:

%
\declareAttribute{PMIX_GROUP_CONTEXT_ID}{"pmix.grp.ctxid"}{size_t}{
Context identifier assigned to the group by the host \ac{RM}.
}
%
\declareAttribute{PMIX_GROUP_ENDPT_DATA}{"pmix.grp.endpt"}{pmix_byte_object_t}{
Data collected during group construction to ensure communication between group members is supported upon completion of the operation.
}
%
\versionMarkerProvisional{6.0}
\declareAttributeProvisional{PMIX_GROUP_JOB_INFO}{"pmix.grp.jinfo"}{pmix_byte_object_t}{
Job-level info for all group participants, returned at end of group construction
}

\vspace{\baselineskip}
In addition, a process can request (via \refapi{PMIx_Get}) the process groups to which a given process (including itself) belongs:

%
\declareAttribute{PMIX_GROUP_NAMES}{"pmix.pgrp.nm"}{pmix_data_array_t*}{
Returns an array of \code{char*} string names of the process groups in which the given process is a member.
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_Group_construct}}
\declareapi{PMIx_Group_construct}

%%%%
\summary

Construct a \ac{PMIx} process group.

%%%%
\format

\copySignature{PMIx_Group_construct}{4.0}{
pmix_status_t \\
PMIx_Group_construct(const char grp[], \\
\hspace*{21\sigspace}const pmix_proc_t procs[], size_t nprocs, \\
\hspace*{21\sigspace}const pmix_info_t directives[], \\
\hspace*{21\sigspace}size_t ndirs, \\
\hspace*{21\sigspace}pmix_info_t **results, \\
\hspace*{21\sigspace}size_t *nresults);
}

\begin{arglist}
\argin{grp}{\code{NULL}-terminated character array of maximum size \refconst{PMIX_MAX_NSLEN} containing the group identifier (string)}
\argin{procs}{Array of \refstruct{pmix_proc_t} structures containing the \ac{PMIx} identifiers of the member processes (array of handles)}
\argin{nprocs}{Number of elements in the \refarg{procs} array (\code{size_t})}
\argin{directives}{Array of \refstruct{pmix_info_t} structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{directives} array (\code{size_t})}
\arginout{results}{Pointer to a location where the array of \refstruct{pmix_info_t} describing the results of the operation is to be returned (pointer to handle)}
\arginout{nresults}{Pointer to a \code{size_t} location where the number of elements in \refarg{results} is to be returned (memory reference)}
\end{arglist}

\returnsimple

\reqattrstart
The following attributes are \textit{required} to be supported by all \ac{PMIx} libraries that support this operation:

\pasteAttributeItem{PMIX_GROUP_LEADER}
\pasteAttributeItem{PMIX_GROUP_OPTIONAL}
\pasteAttributeItem{PMIX_GROUP_LOCAL_ONLY}

Host environments that support this operation are \textit{required} to support the following attributes:

\pasteAttributeItem{PMIX_GROUP_ASSIGN_CONTEXT_ID}
\pasteAttributeItem{PMIX_GROUP_NOTIFY_TERMINATION}

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_TIMEOUT}
\pasteAttributeItem{PMIX_GROUP_LOCAL_CID}
\pasteAttributeItem{PMIX_GROUP_BOOTSTRAP}
\pasteAttributeItem{PMIX_GROUP_ADD_MEMBERS}
\pasteAttributeItem{PMIX_GROUP_FT_COLLECTIVE}

\optattrend

%%%%
\descr

Construct a new group composed of the specified processes and identified with the provided group identifier. The group identifier is a user-defined, \code{NULL}-terminated character array of length less than or equal to \refconst{PMIX_MAX_NSLEN}. Only characters accepted by standard string comparison functions (e.g., \emph{strncmp}) are supported. Processes may engage in multiple simultaneous group construct operations so long as each is provided with a unique group ID. The \refarg{directives} array can be used to pass user-level directives regarding timeout constraints and other options available from the \ac{PMIx} server.

Each provided \refstruct{pmix_proc_t} struct can pass \refconst{PMIX_RANK_WILDCARD} to indicate that all processes in the given namespace are participating.
The ordering of the entries in the \refarg{procs} has no significance, neither is the method used for identifying processes.
Some callers may describe the target set of processes using PMIX_RANK_WILDCARD while other
callers may list the individual processes of a namespace explicitly.
This is unlike \refapi{PMIx_Connect}, because in this group operation, the group name \refarg{grp} is used to determine the uniqueness of the collective.

If the \refattr{PMIX_GROUP_NOTIFY_TERMINATION} attribute is provided and has a value of \code{true}, then either the construct leader(s) (if \refattr{PMIX_GROUP_LEADER} was provided) or all participants who register for the \refconst{PMIX_GROUP_MEMBER_FAILED} event will receive events whenever a process fails or terminates prior to calling \refapi{PMIx_Group_construct} – i.e. if a \emph{group leader} is declared, \textit{only} that process will receive the event. In the absence of a declared leader, \textit{all} specified group members will receive the event.

The event will contain the identifier of the process that failed to join plus any other information that the host \ac{RM} provided. This provides an opportunity for the leader or the collective members to react to the event – e.g., to decide to proceed with a smaller group or to abort the operation. The decision is communicated to the \ac{PMIx} library in the results array at the end of the event handler. This allows \ac{PMIx} to properly adjust accounting for procedure completion. When construct is complete, the participating \ac{PMIx} servers will be alerted to any change in participants and each group member will receive an updated group membership (marked with the \refattr{PMIX_GROUP_MEMBERSHIP} attribute) as part of the \refarg{results} array returned by this \ac{API}.

Failure of a declared leader at any time will cause a \refconst{PMIX_GROUP_LEADER_FAILED} event to be delivered to all participants so they can optionally declare a new leader. A new leader is identified by providing the \refattr{PMIX_GROUP_LEADER} attribute in the results array in the return of the event handler. Any process is allowed to return that attribute for each group leader failed event, thereby declaring itself as a new leader. Results of the leader selection will be communicated to all participants via a \refconst{PMIX_GROUP_LEADER_SELECTED} event identifying the new leader(s) once all participants have completed their event handler. If no leader was selected, then the \refstruct{pmix_info_t} provided to the "leader selected" event handler will include that information so the participants can take appropriate action.

Any participant that returns \refconst{PMIX_GROUP_CONSTRUCT_ABORT} from either the \refconst{PMIX_GROUP_MEMBER_FAILED} or the \refconst{PMIX_GROUP_LEADER_FAILED} event handler will cause the construct process to abort, returning from the call with a \refconst{PMIX_GROUP_CONSTRUCT_ABORT} status.

If the \refattr{PMIX_GROUP_NOTIFY_TERMINATION} attribute is not provided or has a value of \code{false}, then the \refapi{PMIx_Group_construct} operation will simply return an error whenever a proposed group member fails or terminates prior to calling \refapi{PMIx_Group_construct}.

Providing the \refattr{PMIX_GROUP_OPTIONAL} attribute with a value of \code{true} directs the \ac{PMIx} library to consider participation by any specified group member as non-required - thus, the operation will return \refconst{PMIX_SUCCESS} if all members participate, or \refconst{PMIX_ERR_PARTIAL_SUCCESS} if some members fail to participate. The \refarg{results} array will contain the final group membership in the latter case. Note that this use-case can cause the operation to hang if the \refattr{PMIX_TIMEOUT} attribute is not specified and one or more group members fail to call \refapi{PMIx_Group_construct} while continuing to execute. Also, note that no leader or member failed events will be generated during the operation.

Processes in a group under construction are not allowed to leave the group until group construction is complete. Upon completion of the construct procedure, each group member will have access to the job-level information of all namespaces represented in the group plus any information posted via \refapi{PMIx_Put} (subject to the usual scoping directives) for every group member.

\adviceimplstart
At the conclusion of the construct operation, the \ac{PMIx} library is \emph{required} to ensure that job-related information from each participating namespace (via the \refattr{PMIX_GROUP_JOB_INFO} attribute) plus any information posted by group members via \refapi{PMIx_Put} (subject to scoping directives) is available to each member via calls to \refapi{PMIx_Get}.
\adviceimplend

\advicermstart
The collective nature of this \ac{API} generally results in use of a fence-like operation by the backend host environment. Host environments that utilize the array of process participants as a \emph{signature} for such operations may experience potential conflicts should both a \refapi{PMIx_Group_construct} and a \refapi{PMIx_Fence} operation involving the same participants be simultaneously executed. As \ac{PMIx} allows for such use-cases, it is therefore the responsibility of the host environment to resolve any potential conflicts.  Using the group name as part of the \emph{signature} may provide a better unique point of coordination.  Also be aware that the process participants can be specified in different ways by each caller (e.g. some callers may include all members of a namespace using \refconst{PMIX_RANK_WILDCARD} while others may list each process individually).
\advicermend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_Group_construct_nb}}
\declareapi{PMIx_Group_construct_nb}

%%%%
\summary

Non-blocking form of \refapi{PMIx_Group_construct}.

%%%%
\format

\copySignature{PMIx_Group_construct_nb}{4.0}{
pmix_status_t \\
PMIx_Group_construct_nb(const char grp[], \\
\hspace*{24\sigspace}const pmix_proc_t procs[], size_t nprocs, \\
\hspace*{24\sigspace}const pmix_info_t directives[], \\
\hspace*{24\sigspace}size_t ndirs, \\
\hspace*{24\sigspace}pmix_info_cbfunc_t cbfunc, void *cbdata);
}

\begin{arglist}
\argin{grp}{\code{NULL}-terminated character array of maximum size \refconst{PMIX_MAX_NSLEN} containing the group identifier (string)}
\argin{procs}{Array of \refstruct{pmix_proc_t} structures containing the \ac{PMIx} identifiers of the member processes (array of handles)}
\argin{nprocs}{Number of elements in the \refarg{procs} array (\code{size_t})}
\argin{directives}{Array of \refstruct{pmix_info_t} structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{directives} array (\code{size_t})}
\argin{cbfunc}{Callback function \refapi{pmix_info_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

\returnsimplenb

\returnstart
\begin{constantdesc}
\item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed successfully - the \refarg{cbfunc} will \textit{not} be called.
\end{constantdesc}
\returnend

If executed, the status returned in the provided callback function will be one of the following constants:

\begin{itemize}
\item \refconst{PMIX_SUCCESS} The operation succeeded and all specified members participated.
\item \refconst{PMIX_ERR_PARTIAL_SUCCESS} The operation succeeded but not all specified members participated - the final group membership is included in the callback function.
\item \refconst{PMIX_ERR_NOT_SUPPORTED} While the \ac{PMIx} server supports this operation, the host \ac{RM} does not.
\item a non-zero \ac{PMIx} error constant indicating a reason for the request's failure.
\end{itemize}

\reqattrstart
\ac{PMIx} libraries that choose not to support this operation \textit{must} return \refconst{PMIX_ERR_NOT_SUPPORTED} when the function is called.

The following attributes are \textit{required} to be supported by all \ac{PMIx} libraries that support this operation:

\pasteAttributeItem{PMIX_GROUP_LEADER}
\pasteAttributeItem{PMIX_GROUP_OPTIONAL}
\pasteAttributeItem{PMIX_GROUP_LOCAL_ONLY}

Host environments that support this operation are \textit{required} to provide the following attributes:

\pasteAttributeItem{PMIX_GROUP_ASSIGN_CONTEXT_ID}
\pasteAttributeItem{PMIX_GROUP_NOTIFY_TERMINATION}

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_TIMEOUT}
\pasteAttributeItem{PMIX_GROUP_LOCAL_CID}
\pasteAttributeItem{PMIX_GROUP_BOOTSTRAP}
\pasteAttributeItem{PMIX_GROUP_ADD_MEMBERS}
\pasteAttributeItem{PMIX_GROUP_FT_COLLECTIVE}


\optattrend

%%%%
\descr

Non-blocking version of the \refapi{PMIx_Group_construct} operation. The callback function will be called once all group members have called either \refapi{PMIx_Group_construct} or \refapi{PMIx_Group_construct_nb}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_Group_destruct}}
\declareapi{PMIx_Group_destruct}

%%%%
\summary

Destruct a \ac{PMIx} process group.

%%%%
\format

\copySignature{PMIx_Group_destruct}{4.0}{
pmix_status_t \\
PMIx_Group_destruct(const char grp[], \\
\hspace*{20\sigspace}const pmix_info_t directives[], \\
\hspace*{20\sigspace}size_t ndirs);
}

\begin{arglist}
\argin{grp}{\code{NULL}-terminated character array of maximum size \refconst{PMIX_MAX_NSLEN} containing the identifier of the group to be destructed (string)}
\argin{directives}{Array of \refstruct{pmix_info_t} structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{directives} array (\code{size_t})}
\end{arglist}

\returnsimple

\reqattrstart
For implementations and host environments that support the operation, there are no identified required
attributes for this \ac{API}.
\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_TIMEOUT}

\optattrend

%%%%
\descr

Destruct a group identified by the provided group identifier. Processes may engage in multiple simultaneous group destruct operations so long as each involves a unique group ID. The \refarg{directives} array can be used to pass user-level directives regarding timeout constraints and other options available from the \ac{PMIx} server.

The destruct \ac{API} will return an error if any group process fails or terminates prior to calling \refapi{PMIx_Group_destruct} or its non-blocking version unless the \refattr{PMIX_GROUP_NOTIFY_TERMINATION} attribute was provided (with a value of \code{false}) at time of group construction. If notification was requested, then the \refconst{PMIX_GROUP_MEMBER_FAILED} event will be delivered for each process that fails to call destruct and the destruct tracker updated to account for the lack of participation. The \refapi{PMIx_Group_destruct} operation will subsequently return \refconst{PMIX_SUCCESS} when the remaining processes have all called destruct – i.e., the event will serve in place of return of an error.

\advicermstart
The collective nature of this \ac{API} generally results in use of a fence-like operation by the backend host environment. Host environments that utilize the array of process participants as a \emph{signature} for such operations may experience potential conflicts should both a \refapi{PMIx_Group_destruct} and a \refapi{PMIx_Fence} operation involving the same participants be simultaneously executed. As \ac{PMIx} allows for such use-cases, it is therefore the responsibility of the host environment to resolve any potential conflicts.
\advicermend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_Group_destruct_nb}}
\declareapi{PMIx_Group_destruct_nb}

%%%%
\summary

Non-blocking form of \refapi{PMIx_Group_destruct}.

%%%%
\format

\copySignature{PMIx_Group_destruct_nb}{4.0}{
pmix_status_t \\
PMIx_Group_destruct_nb(const char grp[], \\
\hspace*{23\sigspace}const pmix_info_t directives[], \\
\hspace*{23\sigspace}size_t ndirs, \\
\hspace*{23\sigspace}pmix_op_cbfunc_t cbfunc, void *cbdata);
}

\begin{arglist}
\argin{grp}{\code{NULL}-terminated character array of maximum size \refconst{PMIX_MAX_NSLEN} containing the identifier of the group to be destructed (string)}
\argin{directives}{Array of \refstruct{pmix_info_t} structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{directives} array (\code{size_t})}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

\returnsimplenb

\returnstart
\begin{constantdesc}
\item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed successfully - the \refarg{cbfunc} will \textit{not} be called.
\end{constantdesc}
\returnend

If executed, the status returned in the provided callback function will be one of the following constants:

\begin{itemize}
\item \refconst{PMIX_SUCCESS} The operation was successfully completed.
\item \refconst{PMIX_ERR_NOT_SUPPORTED} While the \ac{PMIx} server supports this operation, the host \ac{RM} does not.
\item a non-zero \ac{PMIx} error constant indicating a reason for the request's failure.
\end{itemize}

\reqattrstart
\ac{PMIx} libraries that choose not to support this operation \textit{must} return \refconst{PMIX_ERR_NOT_SUPPORTED} when the function is called. For implementations and host environments that support the operation, there are no identified required
attributes for this \ac{API}.
\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_TIMEOUT}

\optattrend

%%%%
\descr

Non-blocking version of the \refapi{PMIx_Group_destruct} operation. The callback function will be called once all members of the group have executed either \refapi{PMIx_Group_destruct} or \refapi{PMIx_Group_destruct_nb}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_Group_invite}}
\declareapi{PMIx_Group_invite}

%%%%
\summary

Asynchronously construct a \ac{PMIx} process group.

%%%%
\format

\copySignature{PMIx_Group_invite}{4.0}{
pmix_status_t \\
PMIx_Group_invite(const char grp[], \\
\hspace*{18\sigspace}const pmix_proc_t procs[], size_t nprocs, \\
\hspace*{18\sigspace}const pmix_info_t directives[], size_t ndirs, \\
\hspace*{18\sigspace}pmix_info_t **results, size_t *nresult);
}

\begin{arglist}
\argin{grp}{\code{NULL}-terminated character array of maximum size \refconst{PMIX_MAX_NSLEN} containing the group identifier (string)}
\argin{procs}{Array of \refstruct{pmix_proc_t} structures containing the \ac{PMIx} identifiers of the processes to be invited (array of handles)}
\argin{nprocs}{Number of elements in the \refarg{procs} array (\code{size_t})}
\argin{directives}{Array of \refstruct{pmix_info_t} structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{directives} array (\code{size_t})}
\arginout{results}{Pointer to a location where the array of \refstruct{pmix_info_t} describing the results of the operation is to be returned (pointer to handle)}
\arginout{nresults}{Pointer to a \code{size_t} location where the number of elements in \refarg{results} is to be returned (memory reference)}
\end{arglist}

\returnsimple

\reqattrstart
The following attributes are \textit{required} to be supported by all \ac{PMIx} libraries that support this operation:

\pasteAttributeItem{PMIX_GROUP_OPTIONAL}
\pasteAttributeItem{PMIX_GROUP_FT_COLLECTIVE}

Host environments that support this operation are \textit{required} to provide the following attributes:

\pasteAttributeItem{PMIX_GROUP_ASSIGN_CONTEXT_ID}
\pasteAttributeItem{PMIX_GROUP_NOTIFY_TERMINATION}
\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_TIMEOUT}

\optattrend

%%%%
\descr

Explicitly invite the specified processes to join a group. The process making the \refapi{PMIx_Group_invite} call is automatically declared to be the \emph{group leader}. Each invited process will be notified of the invitation via the \refconst{PMIX_GROUP_INVITED} event - the processes being invited must therefore register for the \refconst{PMIX_GROUP_INVITED} event in order to be notified of the invitation. Note that the \ac{PMIx} event notification system caches events - thus, no ordering of invite versus event registration is required.

The invitation event will include the identity of the inviting process plus the name of the group. When ready to respond, each invited process provides a response using either the blocking or non-blocking form of \refapi{PMIx_Group_join}. This will notify the inviting process that the invitation was either accepted (via the \refconst{PMIX_GROUP_INVITE_ACCEPTED} event) or declined (via the \refconst{PMIX_GROUP_INVITE_DECLINED} event). The \refconst{PMIX_GROUP_INVITE_ACCEPTED} event is captured by the \ac{PMIx} client library of the inviting process – i.e., the application itself does not need to register for this event. The library will track the number of accepting processes and alert the inviting process (by returning from the blocking form of \refapi{PMIx_Group_invite} or calling the callback function of the non-blocking form) when group construction completes.

The inviting process should, however, register for the \refconst{PMIX_GROUP_INVITE_DECLINED} if the application allows invited processes to decline the invitation. This provides an opportunity for the application to either invite a replacement, declare ``abort'', or choose to remove the declining process from the final group. The inviting process should also register to receive \refconst{PMIX_GROUP_INVITE_FAILED} events whenever a process fails or terminates prior to responding to the invitation. Actions taken by the inviting process in response to these events must be communicated at the end of the event handler by returning the corresponding result so that the \ac{PMIx} library can adjust accordingly.

Upon completion of the operation, all members of the new group will receive access to the job-level information of each other’s namespaces plus any information posted via \refapi{PMIx_Put} by the other members.

The inviting process is automatically considered the leader of the asynchronous group construction procedure and will receive all failure or termination events for invited members prior to completion. The inviting process is required to provide a \refconst{PMIX_GROUP_CONSTRUCT_COMPLETE} event once the group has been fully assembled – this event is used by the \ac{PMIx} library as a trigger to release participants from their call to \refapi{PMIx_Group_join} and provides information (e.g., the final group membership) to be returned in the \refarg{results} array.

Failure of the inviting process at any time will cause a \refconst{PMIX_GROUP_LEADER_FAILED} event to be delivered to all participants so they can optionally declare a new leader. A new leader is identified by providing the \refattr{PMIX_GROUP_LEADER} attribute in the results array in the return of the event handler. Only one process is allowed to return that attribute, declaring itself as the new leader. Results of the leader selection will be communicated to all participants via a \refconst{PMIX_GROUP_LEADER_SELECTED} event identifying the new leader. If no leader was selected, then the status code provided in the event handler will provide an error value so the participants can take appropriate action.

\adviceuserstart
Applications are not allowed to use the group in any operations until group construction is complete. This is required in order to ensure consistent knowledge of group membership across all participants.
\adviceuserend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_Group_invite_nb}}
\declareapi{PMIx_Group_invite_nb}

%%%%
\summary

Non-blocking form of \refapi{PMIx_Group_invite}.

%%%%
\format

\copySignature{PMIx_Group_invite_nb}{4.0}{
pmix_status_t \\
PMIx_Group_invite_nb(const char grp[], \\
\hspace*{21\sigspace}const pmix_proc_t procs[], size_t nprocs, \\
\hspace*{21\sigspace}const pmix_info_t directives[], size_t ndirs, \\
\hspace*{21\sigspace}pmix_info_cbfunc_t cbfunc, void *cbdata);
}

\begin{arglist}
\argin{grp}{\code{NULL}-terminated character array of maximum size \refconst{PMIX_MAX_NSLEN} containing the group identifier (string)}
\argin{procs}{Array of \refstruct{pmix_proc_t} structures containing the \ac{PMIx} identifiers of the processes to be invited (array of handles)}
\argin{nprocs}{Number of elements in the \refarg{procs} array (\code{size_t})}
\argin{directives}{Array of \refstruct{pmix_info_t} structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{directives} array (\code{size_t})}
\argin{cbfunc}{Callback function \refapi{pmix_info_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

\returnsimplenb

\returnstart
\begin{constantdesc}
\item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed successfully - the \refarg{cbfunc} will \textit{not} be called.
\end{constantdesc}
\returnend

If executed, the status returned in the provided callback function will be one of the following constants:

\begin{itemize}
\item \refconst{PMIX_SUCCESS} The operation succeeded and all specified members participated.
\item \refconst{PMIX_ERR_PARTIAL_SUCCESS} The operation succeeded but not all specified members participated - the final group membership is included in the callback function.
\item \refconst{PMIX_ERR_NOT_SUPPORTED} While the \ac{PMIx} server supports this operation, the host \ac{RM} does not.
\item a non-zero \ac{PMIx} error constant indicating a reason for the request's failure.
\end{itemize}

\reqattrstart
The following attributes are \textit{required} to be supported by all \ac{PMIx} libraries that support this operation:

\pasteAttributeItem{PMIX_GROUP_OPTIONAL}
\pasteAttributeItem{PMIX_GROUP_FT_COLLECTIVE}

Host environments that support this operation are \textit{required} to provide the following attributes:

\pasteAttributeItem{PMIX_GROUP_ASSIGN_CONTEXT_ID}
\pasteAttributeItem{PMIX_GROUP_NOTIFY_TERMINATION}

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_TIMEOUT}

\optattrend

%%%%
\descr

Non-blocking version of the \refapi{PMIx_Group_invite} operation. The callback function will be called once all invited members of the group (or their substitutes) have executed either \refapi{PMIx_Group_join} or \refapi{PMIx_Group_join_nb}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_Group_join}}
\declareapi{PMIx_Group_join}

%%%%
\summary

Accept an invitation to join a \ac{PMIx} process group.

%%%%
\format

\copySignature{PMIx_Group_join}{4.0}{
pmix_status_t \\
PMIx_Group_join(const char grp[], \\
\hspace*{16\sigspace}const pmix_proc_t *leader, \\
\hspace*{16\sigspace}pmix_group_opt_t opt, \\
\hspace*{16\sigspace}const pmix_info_t directives[], size_t ndirs, \\
\hspace*{16\sigspace}pmix_info_t **results, size_t *nresult);
}

\begin{arglist}
\argin{grp}{\code{NULL}-terminated character array of maximum size \refconst{PMIX_MAX_NSLEN} containing the group identifier (string)}
\argin{leader}{Process that generated the invitation (handle)}
\argin{opt}{Accept or decline flag (\refstruct{pmix_group_opt_t})}
\argin{directives}{Array of \refstruct{pmix_info_t} structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{directives} array (\code{size_t})}
\arginout{results}{Pointer to a location where the array of \refstruct{pmix_info_t} describing the results of the operation is to be returned (pointer to handle)}
\arginout{nresults}{Pointer to a \code{size_t} location where the number of elements in \refarg{results} is to be returned (memory reference)}
\end{arglist}

\returnsimple

\reqattrstart
There are no identified required attributes for implementers.

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_TIMEOUT}

\optattrend

%%%%
\descr

Respond to an invitation to join a group that is being asynchronously constructed. The process must have registered for the \refconst{PMIX_GROUP_INVITED} event in order to be notified of the invitation. When called, the event information will include the \refstruct{pmix_proc_t} identifier of the process that generated the invitation along with the identifier of the group being constructed. When ready to respond, the process provides a response using either form of \refapi{PMIx_Group_join}.

\adviceuserstart
Since the process is alerted to the invitation in a \ac{PMIx} event handler, the process \emph{must not} use the blocking form of this call unless it first ``thread shifts'' out of the handler and into its own thread context. Likewise, while it is safe to call the non-blocking form of the \ac{API} from the event handler, the process \emph{must not} block in the handler while waiting for the callback function to be called.
\adviceuserend

Calling this function causes the inviting process (aka the \emph{group leader}) to be notified that the process has either accepted or declined the request. The blocking form of the \ac{API} will return once the group has been completely constructed or the group’s construction has failed (as described below) – likewise, the callback function of the non-blocking form will be executed upon the same conditions.

Failure of the leader during the call to \refapi{PMIx_Group_join} will cause a \refconst{PMIX_GROUP_LEADER_FAILED} event to be delivered to all invited participants so they can optionally declare a new leader. A new leader is identified by providing the \refattr{PMIX_GROUP_LEADER} attribute in the results array in the return of the event handler. Only one process is allowed to return that attribute, declaring itself as the new leader. Results of the leader selection will be communicated to all participants via a \refconst{PMIX_GROUP_LEADER_SELECTED} event identifying the new leader. If no leader was selected, then the status code provided in the event handler will provide an error value so the participants can take appropriate action.

Any participant that returns \refconst{PMIX_GROUP_CONSTRUCT_ABORT} from the leader failed event handler will cause all participants to receive an event notifying them of that status. Similarly, the leader may elect to abort the procedure by either returning \refconst{PMIX_GROUP_CONSTRUCT_ABORT} from the handler assigned to the \refconst{PMIX_GROUP_INVITE_ACCEPTED} or \refconst{PMIX_GROUP_INVITE_DECLINED} codes, or by generating an event for the abort code. Abort events will be sent to all invited participants.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_Group_join_nb}}
\declareapi{PMIx_Group_join_nb}

%%%%
\summary

Non-blocking form of \refapi{PMIx_Group_join}

%%%%
\format

\copySignature{PMIx_Group_join_nb}{4.0}{
pmix_status_t \\
PMIx_Group_join_nb(const char grp[], \\
\hspace*{19\sigspace}const pmix_proc_t *leader, \\
\hspace*{19\sigspace}pmix_group_opt_t opt, \\
\hspace*{19\sigspace}const pmix_info_t directives[], size_t ndirs, \\
\hspace*{19\sigspace}pmix_info_cbfunc_t cbfunc, void *cbdata);
}

\begin{arglist}
\argin{grp}{\code{NULL}-terminated character array of maximum size \refconst{PMIX_MAX_NSLEN} containing the group identifier (string)}
\argin{leader}{Process that generated the invitation (handle)}
\argin{opt}{Accept or decline flag (\refstruct{pmix_group_opt_t})}
\argin{directives}{Array of \refstruct{pmix_info_t} structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{directives} array (\code{size_t})}
\argin{cbfunc}{Callback function \refapi{pmix_info_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

\returnsimplenb

\returnstart
\begin{constantdesc}
\item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed successfully - the \refarg{cbfunc} will \textit{not} be called.
\end{constantdesc}
\returnend

If executed, the status returned in the provided callback function will be one of the following constants:

\begin{itemize}
\item \refconst{PMIX_SUCCESS} The operation succeeded and group membership is in the callback function parameters.
\item \refconst{PMIX_ERR_NOT_SUPPORTED} While the \ac{PMIx} server supports this operation, the host \ac{RM} does not.
\item a non-zero \ac{PMIx} error constant indicating a reason for the request's failure.
\end{itemize}


\reqattrstart
There are no identified required attributes for implementers.

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_TIMEOUT}

\optattrend

%%%%
\descr

Non-blocking version of the \refapi{PMIx_Group_join} operation. The callback function will be called once all invited members of the group (or their substitutes) have executed either \refapi{PMIx_Group_join} or \refapi{PMIx_Group_join_nb}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Group accept/decline directives}
\declarestruct{pmix_group_opt_t}

\versionMarker{4.0}
The \refstruct{pmix_group_opt_t} type is a \code{uint8_t} value used with the \refapi{PMIx_Group_join} \ac{API} to indicate \emph{accept} or \emph{decline} of the invitation - these are provided for readability of user code:

\begin{constantdesc}
%
\declareconstitemvalue{PMIX_GROUP_DECLINE}{0}
Decline the invitation.
%
\declareconstitemvalue{PMIX_GROUP_ACCEPT}{1}
Accept the invitation.
%
\end{constantdesc}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_Group_leave}}
\declareapi{PMIx_Group_leave}

%%%%
\summary

Leave a \ac{PMIx} process group.

%%%%
\format

\copySignature{PMIx_Group_leave}{4.0}{
pmix_status_t \\
PMIx_Group_leave(const char grp[], \\
\hspace*{17\sigspace}const pmix_info_t directives[], \\
\hspace*{17\sigspace}size_t ndirs);
}

\begin{arglist}
\argin{grp}{\code{NULL}-terminated character array of maximum size \refconst{PMIX_MAX_NSLEN} containing the group identifier (string)}
\argin{directives}{Array of \refstruct{pmix_info_t} structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{directives} array (\code{size_t})}
\end{arglist}

\returnsimple

\reqattrstart
There are no identified required attributes for implementers.
\reqattrend


%%%%
\descr

Calls to \refapi{PMIx_Group_leave} (or its non-blocking form) will cause a \refconst{PMIX_GROUP_LEFT} event to be generated notifying all members of the group of the caller’s departure. The function will return (or the non-blocking function will execute the specified callback function) once the event has been locally generated and is not indicative of remote receipt.

\adviceuserstart
The \refapi{PMIx_Group_leave} API is intended solely for asynchronous departures of individual processes from a group as it is not a scalable operation – i.e., when a process determines it should no longer be a part of a defined group, but the remainder of the group retains a valid reason to continue in existence. Developers are advised to use \refapi{PMIx_Group_destruct} (or its non-blocking form) for all other scenarios as it represents a more scalable operation.
\adviceuserend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_Group_leave_nb}}
\declareapi{PMIx_Group_leave_nb}

%%%%
\summary

Non-blocking form of \refapi{PMIx_Group_leave}.

%%%%
\format

\copySignature{PMIx_Group_leave_nb}{4.0}{
pmix_status_t \\
PMIx_Group_leave_nb(const char grp[], \\
\hspace*{20\sigspace}const pmix_info_t directives[], \\
\hspace*{20\sigspace}size_t ndirs, \\
\hspace*{20\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{20\sigspace}void *cbdata);
}

\begin{arglist}
\argin{grp}{\code{NULL}-terminated character array of maximum size \refconst{PMIX_MAX_NSLEN} containing the group identifier (string)}
\argin{directives}{Array of \refstruct{pmix_info_t} structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{directives} array (\code{size_t})}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

\returnsimplenb

\returnstart
\begin{constantdesc}
\item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed successfully - the \refarg{cbfunc} will \textit{not} be called.
\end{constantdesc}
\returnend

If executed, the status returned in the provided callback function will be one of the following constants:

\begin{itemize}
\item \refconst{PMIX_SUCCESS} The operation succeeded - i.e., the \refconst{PMIX_GROUP_LEFT} event was generated.
\item \refconst{PMIX_ERR_NOT_SUPPORTED} While the \ac{PMIx} library supports this operation, the host \ac{RM} does not.
\item a non-zero \ac{PMIx} error constant indicating a reason for the request's failure.
\end{itemize}


\reqattrstart
There are no identified required attributes for implementers.

\reqattrend

%%%%
\descr

Non-blocking version of the \refapi{PMIx_Group_leave} operation. The callback function will be called once the event has been locally generated and is not indicative of remote receipt.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%