diff --git a/Chap_API_Fabric.tex b/Chap_API_Fabric.tex index fdbe1049..8a4be206 100644 --- a/Chap_API_Fabric.tex +++ b/Chap_API_Fabric.tex @@ -32,7 +32,7 @@ \chapter{Fabric Support Definitions} \end{enumerate} -Information on fabric coordinates, endpoints, and device distances are provided as \emph{reserved keys} as detailed in Chapter \ref{chap:api_rsvd_keys} - i.e., they are to be available at client start of execution and are subject to the retrieval rules of Section \ref{chap:rkeys:retrules}. Examples for retrieving fabric-related information include retrieval of: +Information on fabric coordinates, endpoints, and device distances are provided as \emph{reserved keys} as detailed in Chapter \ref{chap:api_rsvd_keys} - i.e., they are to be available at client start of execution and are subject to the retrieval rules of Section \ref{chap:api_rsvd_keys:retrules}. Examples for retrieving fabric-related information include retrieval of: \begin{itemize} \item An array of information on fabric devices for a node by passing \refattr{PMIX_FABRIC_DEVICES} as the key to \refapi{PMIx_Get} along with the \refattr{PMIX_HOSTNAME} of the node as a directive @@ -216,7 +216,7 @@ \subsection{Fabric Coordinate Structure} Note that the \refstruct{pmix_coord_t} structure does not imply nor mandate any requirement on how the coordinate data is to be stored within the \ac{PMIx} library. Implementers are free to store the coordinate in whatever format they choose. \adviceimplend -A fabric coordinate is associated with a given fabric device and must be unique within a given view. Fabric devices are associated with the operating system which hosts them - thus, fabric coordinates are logically grouped within the \emph{node} realm (as described in Section \ref{api:struct:attributes:retrieval}) and can be retrieved per the rules detailed in Section \ref{chap:res:nrealm}. +A fabric coordinate is associated with a given fabric device and must be unique within a given view. Fabric devices are associated with the operating system which hosts them - thus, fabric coordinates are logically grouped within the \emph{node} realm (as described in Section \ref{api:struct:attributes:retrieval}) and can be retrieved per the rules detailed in Section \ref{chap:api_rsvd_keys:nrealm}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Fabric coordinate support macros} @@ -319,7 +319,7 @@ \subsection{Fabric Geometry Structure} Note that the \refstruct{pmix_coord_t} structure does not imply nor mandate any requirement on how the coordinate data is to be stored within the \ac{PMIx} library. Implementers are free to store the coordinate in whatever format they choose. \adviceimplend -A fabric coordinate is associated with a given fabric device and must be unique within a given view. Fabric devices are associated with the operating system which hosts them - thus, fabric coordinates are logically grouped within the \emph{node} realm (as described in Section \ref{api:struct:attributes:retrieval}) and can be retrieved per the rules detailed in Section \ref{chap:res:nrealm}. +A fabric coordinate is associated with a given fabric device and must be unique within a given view. Fabric devices are associated with the operating system which hosts them - thus, fabric coordinates are logically grouped within the \emph{node} realm (as described in Section \ref{api:struct:attributes:retrieval}) and can be retrieved per the rules detailed in Section \ref{chap:api_rsvd_keys:nrealm}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -626,7 +626,7 @@ \section{Fabric Support Attributes} % % \vspace{\baselineskip} -The following attributes are related to the \emph{node realm} (as described in Section \ref{chap:res:nrealm}) and are retrieved according to those rules. +The following attributes are related to the \emph{node realm} (as described in Section \ref{chap:api_rsvd_keys:nrealm}) and are retrieved according to those rules. % \declareAttributeNEW{PMIX_FABRIC_DEVICES}{"pmix.fab.devs"}{pmix_data_array_t}{ @@ -698,7 +698,7 @@ \section{Fabric Support Attributes} } % \vspace{\baselineskip} -The following attributes are related to the \emph{process realm} (as described in Section \ref{chap:res:prealm}) and are retrieved according to those rules. +The following attributes are related to the \emph{process realm} (as described in Section \ref{chap:api_rsvd_keys:prealm}) and are retrieved according to those rules. % \declareAttributeNEW{PMIX_FABRIC_ENDPT}{"pmix.fab.endpt"}{pmix_data_array_t}{ @@ -706,7 +706,7 @@ \section{Fabric Support Attributes} } % \vspace{\baselineskip} -The following attributes are related to the \emph{job realm} (as described in Section \ref{chap:res:jrealm}) and are retrieved according to those rules. Note that distances to fabric devices are retrieved using the \refattr{PMIX_DEVICE_DISTANCES} key with the appropriate \refstruct{pmix_device_type_t} qualifier. +The following attributes are related to the \emph{job realm} (as described in Section \ref{chap:api_rsvd_keys:jrealm}) and are retrieved according to those rules. Note that distances to fabric devices are retrieved using the \refattr{PMIX_DEVICE_DISTANCES} key with the appropriate \refstruct{pmix_device_type_t} qualifier. % \declareAttributeNEW{PMIX_SWITCH_PEERS}{"pmix.speers"}{pmix_data_array_t}{ diff --git a/Chap_API_NonReserved_Keys.tex b/Chap_API_NonReserved_Keys.tex deleted file mode 100644 index 2bffdb44..00000000 --- a/Chap_API_NonReserved_Keys.tex +++ /dev/null @@ -1,289 +0,0 @@ -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -% Chapter: Process-Related Non-Reserved Keys -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\chapter{Process-Related Non-Reserved Keys} -\label{chap:nrkeys} - -\emph{Non-reserved keys} are keys whose string representation begin with a -prefix other than \code{"pmix"}. Such keys are typically defined by an -application when information needs to be exchanged between processes (e.g., -where connection information is required and the host environment does not -support the \refterm{instant on} option) or where the host environment does not -provide a required piece of data. Beyond the restriction on name prefix, -non-reserved keys are required to be unique across conflicting \emph{scopes} as defined in Section \ref{api:nres:scope} - e,g., a non-reserved key cannot be posted by the same process in both the \refconst{PMIX_LOCAL} and \refconst{PMIX_REMOTE} scopes (note that posting the key in the \refconst{PMIX_GLOBAL} scope would have met the desired objective). - -\ac{PMIx} provides support for two methods of exchanging non-reserved keys: - -\begin{itemize} - \item Global, collective exchange of the information prior to retrieval. This is accomplished by executing a barrier operation that includes collection and exchange of the data provided by each process such that each process has access to the full set of data from all participants once the operation has completed. \ac{PMIx} provides the \refapi{PMIx_Fence} function (or its non-blocking equivalent) for this purpose, accompanied by the \refattr{PMIX_COLLECT_DATA} qualifier. - \item Direct, on-demand retrieval of the information. No barrier or global exchange is conducted in this case. Instead, information is retrieved from the host where that process is executing upon request - i.e., a call to \refapi{PMIx_Get} results in a data exchange with the \ac{PMIx} server on the remote host. Various caching strategies may be employed by the host environment and/or \ac{PMIx} implementation to reduce the number of retrievals. Note that this method requires that the host environment both know the location of the posting process and support direct information retrieval. -\end{itemize} - -Both of the above methods are based on retrieval from a specific process - -i.e., the \refarg{proc} argument to \refapi{PMIx_Get} must include both the -namespace and the rank of the process that posted the information. However, in -some cases, non-reserved keys are provided on a globally unique basis and the -retrieving process has no knowledge of the identity of the process posting the -key. This is typically found in legacy applications (where the originating -process identifier is often embedded in the key itself) and in unstructured -applications that lack rank-related behavior. In these cases, the key remains -associated with the namespace of the process that posted it, but is retrieved -by use of the \refconst{PMIX_RANK_UNDEF} rank. In addition, the keys must be -globally exchanged prior to retrieval as there is no way for the host to -otherwise locate the source for the information. - -Note that the retrieval rules for non-reserved keys (detailed in Section \ref{chap:nrkeys:retrules}) differ significantly from those used for reserved keys. - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\section{Posting Key/Value Pairs} -\label{chap:api_kv_mgmt:set} - -\ac{PMIx} clients can post non-reserved key-value pairs associated with themselves by using \refapi{PMIx_Put}. Alternatively, \ac{PMIx} clients can cache arbitrary key-value pairs accessible only by the caller via the \refapi{PMIx_Store_internal} \ac{API}. - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\subsection{\code{PMIx_Put}} -\declareapi{PMIx_Put} - -%%%% -\summary - -Post a key/value pair for distribution. - -%%%% -\format - -\copySignature{PMIx_Put}{1.0}{ -pmix_status_t \\ -PMIx_Put(pmix_scope_t scope, \\ -\hspace*{9\sigspace}const pmix_key_t key, \\ -\hspace*{9\sigspace}pmix_value_t *val); -} - -\begin{arglist} -\argin{scope}{Distribution scope of the provided value (handle)} -\argin{key}{key (\refstruct{pmix_key_t})} -\argin{value}{Reference to a \refstruct{pmix_value_t} structure (handle)} -\end{arglist} - -\returnstart -\begin{itemize} -\item \refconst{PMIX_ERR_BAD_PARAM} indicating a reserved key is provided in the \refarg{key} argument. -\end{itemize} -\returnend - -%%%% -\descr - -Post a key-value pair for distribution. Depending upon the \ac{PMIx} implementation, the posted value may be locally cached in the client's \ac{PMIx} library until \refapi{PMIx_Commit} is called. - -The provided \refarg{scope} determines the ability of other processes to access the posted data, as defined in \specrefstruct{pmix_scope_t}. -Specific implementations may support different scope values, but all implementations must support at least \refconst{PMIX_GLOBAL}. - -The \refstruct{pmix_value_t} structure supports both string and binary values. -\ac{PMIx} implementations are required to support heterogeneous environments by properly converting binary values between host architectures, and will copy the provided \refarg{value} into internal memory prior to returning from \refapi{PMIx_Put}. - -\adviceuserstart -Note that keys starting with a string of ``\code{pmix}'' must not be used in calls to \refapi{PMIx_Put}. Thus, applications should never use a defined ``PMIX'' attribute as the key in a call to \refapi{PMIx_Put}. -\adviceuserend - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\subsubsection{Scope of Put Data} -\declarestruct{pmix_scope_t} -\label{api:nres:scope} - -\versionMarker{1.0} -The \refstruct{pmix_scope_t} structure is a \code{uint8_t} type that defines the availability of data passed to \refapi{PMIx_Put}. -The following constants can be used to set a variable of the type \refstruct{pmix_scope_t}. All definitions were introduced in version 1 of the standard unless otherwise marked. - -Specific implementations may support different scope values, but all implementations must support at least \refconst{PMIX_GLOBAL}. -If a specified scope value is not supported, then the \refapi{PMIx_Put} call must return \refconst{PMIX_ERR_NOT_SUPPORTED}. - -\begin{constantdesc} -% -\declareconstitemvalue{PMIX_SCOPE_UNDEF}{0} -Undefined scope. -% -\declareconstitemvalue{PMIX_LOCAL}{1} -The data is intended only for other application processes on the same node. -Data marked in this way will not be included in data packages sent to remote requesters - i.e., it is only available to processes on the local node. -% -\declareconstitemvalue{PMIX_REMOTE}{2} -The data is intended solely for applications processes on remote nodes. -Data marked in this way will not be shared with other processes on the same node - i.e., it is only available to processes on remote nodes. -% -\declareconstitemvalue{PMIX_GLOBAL}{3} -The data is to be shared with all other requesting processes, regardless of location. -% -\versionMarker{2.0} -\declareconstitemvalue{PMIX_INTERNAL}{4} -The data is intended solely for this process and is not shared with other processes. -% -\end{constantdesc} - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\subsection{\code{PMIx_Store_internal}} -\declareapi{PMIx_Store_internal} - -%%%% -\summary - -Store some data locally for retrieval by other areas of the process. - -%%%% -\format - -\copySignature{PMIx_Store_internal}{1.0}{ -pmix_status_t \\ -PMIx_Store_internal(const pmix_proc_t *proc, \\ -\hspace*{20\sigspace}const pmix_key_t key, \\ -\hspace*{20\sigspace}pmix_value_t *val); -} - -\begin{arglist} -\argin{proc}{process reference (handle)} -\argin{key}{key to retrieve (string)} -\argin{val}{Value to store (handle)} -\end{arglist} - -\returnstart -\begin{itemize} -\item \refconst{PMIX_ERR_BAD_PARAM} indicating a reserved key is provided in the \refarg{key} argument. -\end{itemize} -\returnend - -%%%% -\descr - -Store some data locally for retrieval by other areas of the process. -This is data that has only internal scope - it will never be posted externally. Typically used to cache data obtained by means outside of \ac{PMIx} so that it can be accessed by various areas of the process. - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\subsection{\code{PMIx_Commit}} -\declareapi{PMIx_Commit} - -%%%% -\summary - -Post all previously \refapi{PMIx_Put} values for distribution. - -%%%% -\format - -\copySignature{PMIx_Commit}{1.0}{ -pmix_status_t PMIx_Commit(void); -} - -\returnsimple - -%%%% -\descr - -\ac{PMIx} implementations may choose to locally cache non-reserved keys prior to submitting them for distribution. Accordingly, \ac{PMIx} provides a second \ac{API} specifically to stage all previously posted data for distribution - e.g., by transmitting the entire collection of data posted by the process to a server in one operation. This is an asynchronous operation that will immediately return to the caller while the data is staged in the background. - -\adviceuserstart -Users are advised to always include the call to \refapi{PMIx_Commit} in case the local implementation requires it. Note that posted data will not be circulated during \refapi{PMIx_Commit}. Availability of the data by other processes upon completion of \refapi{PMIx_Commit} therefore still relies upon the exchange mechanisms described at the beginning of this chapter. -\adviceuserend - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\section{Retrieval rules for non-reserved keys} -\label{chap:nrkeys:retrules} - -Since non-reserved keys cannot, by definition, have been provided by the host -environment, their retrieval follows significantly different rules than those -defined for reserved keys (as detailed in Section \ref{chap:rkeys:retrules}). -\refapi{PMIx_Get} for a non-reserved key will obey the -following precedence search: - -\begin{enumerate} - \item If the \refattr{PMIX_GET_REFRESH_CACHE} attribute is given, then the - request is first forwarded to the local \ac{PMIx} server which will then - update the client's cache. Note that this may not, depending upon - implementation details, result in any action. - - \item Check the local \ac{PMIx} client cache for the requested key - if not found and either the \refattr{PMIX_OPTIONAL} or \refattr{PMIX_GET_REFRESH_CACHE} attribute was given, the search will stop at this point and return the \refconst{PMIX_ERR_NOT_FOUND} status. - - \item Request the information from the local \ac{PMIx} server. The server - will check its cache for the specified key within the appropriate scope as - defined by the process that originally posted the key. If the value exists - in a scope that contains the requesting process, then the value shall be - returned. If the value exists, but in a scope that excludes the requesting - process, then the server shall immediately return the - \refconst{PMIX_ERR_EXISTS_OUTSIDE_SCOPE}. - - If the value still isn't found and the \refattr{PMIX_IMMEDIATE} attribute - was given, then the library shall return the \refconst{PMIX_ERR_NOT_FOUND} - error constant to the requester. Otherwise, the \ac{PMIx} server library - will take one of the following actions: - \begin{compactitemize} - \item If the target process has a rank of \refconst{PMIX_RANK_UNDEF}, - then this indicates that the key being requested is globally unique - and \emph{not} associated with a specific process. In this case, the - server shall hold the request until either the data appears at the - server or, if given, the \refattr{PMIX_TIMEOUT} is reached. In the - latter case, the server will return the \refconst{PMIX_ERR_TIMEOUT} - status. Note that the server may, depending on \ac{PMIx} - implementation, never respond if the caller failed to specify a - \refattr{PMIX_TIMEOUT} and the requested key fails to arrive at the - server. - - \item If the target process is \emph{local} (i.e., attached to the - same \ac{PMIx} server), then the server will hold the request until - either the target process provides the data or, if given, the - \refattr{PMIX_TIMEOUT} is reached. In the latter case, the server will - return the \refconst{PMIX_ERR_TIMEOUT} status. Note that data which is - posted via \refapi{PMIx_Put} but not staged with \refapi{PMIx_Commit} - may, depending upon implementation, never appear at the server. - - \item If the target process is \emph{remote} (i.e., not attached to - the same \ac{PMIx} server), the server will either: - \begin{compactitemize} - \item If the host has provided the - \refapi{pmix_server_dmodex_req_fn_t} module function - interface, then the server - shall pass the request to its host for servicing. The host is - responsible for determining the location of the target process and - passing the request to the \ac{PMIx} server at that location. - - When the remote data request is received, the target \ac{PMIx} - server will check its cache for the specified key. If the key is - not present, the request shall be held until either the target - process provides the data or, if given, the \refattr{PMIX_TIMEOUT} - is reached. In the latter case, the server will return the - \refconst{PMIX_ERR_TIMEOUT} status. The host shall convey the - result back to the originating \ac{PMIx} server, which will reply - to the requesting client with the result of the request when the - host provides it. - - Note that the target server may, depending on \ac{PMIx} - implementation, never respond if the caller failed to specify a - \refattr{PMIX_TIMEOUT} and the target process fails to post the - requested key. - - \item if the host does not support the - \refapi{pmix_server_dmodex_req_fn_t} interface, then - the server will immediately respond to the client with the - \refconst{PMIX_ERR_NOT_FOUND} status - \end{compactitemize} - \end{compactitemize} -\end{enumerate} - -\adviceimplstart -While there is no requirement that all \ac{PMIx} implementations follow the -client-server paradigm used in the above description, implementers are -required to provide behaviors consistent with the described search pattern. -\adviceimplend - -\adviceuserstart -Users are advised to always specify the \refattr{PMIX_TIMEOUT} value when -retrieving non-reserved keys to avoid potential deadlocks should the specified -key not become available. -\adviceuserend - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% diff --git a/Chap_API_Publish.tex b/Chap_API_Publish.tex index d81271da..894c57b7 100644 --- a/Chap_API_Publish.tex +++ b/Chap_API_Publish.tex @@ -4,7 +4,8 @@ \chapter{Publish/Lookup Operations} \label{chap:pub} -Chapter~\ref{chap:api_rsvd_keys} and Chapter~\ref{chap:nrkeys} discussed how reserved and non-reserved keys dealt with +Chapter~\ref{chap:api_rsvd_keys} and Chapter~\ref{chap:data_sharing:non_rsvd_keys} +discussed how reserved and non-reserved keys dealt with information that either was associated with a specific process (i.e., the retrieving process knew the identifier of the process that posted it) or required a synchronization operation prior to retrieval (e.g., the case of diff --git a/Chap_API_Reserved_Keys.tex b/Chap_API_Reserved_Keys.tex index 4305e292..80ff4ee7 100644 --- a/Chap_API_Reserved_Keys.tex +++ b/Chap_API_Reserved_Keys.tex @@ -14,7 +14,7 @@ \chapter{Reserved Keys} \ac{PMIx} implementations may choose to define their own custom-prefixed keys which may adhere to either the \emph{reserved} or the \emph{non-reserved} retrieval rules at the discretion of the implementation. Implementations may choose to provide such custom keys at client start of execution, but this is not required. -Host environments may also opt to define their own custom keys. However, \ac{PMIx} implementations are unlikely to recognize such host-defined keys and will therefore treat them according to the \emph{non-reserved} rules described in Chapter \ref{chap:nrkeys}. Users +Host environments may also opt to define their own custom keys. However, \ac{PMIx} implementations are unlikely to recognize such host-defined keys and will therefore treat them according to the \emph{non-reserved} rules described in Chapter \ref{chap:data_sharing}. Users are advised to check both the local \ac{PMIx} implementation and host environment documentation for a list of any custom prefixes they must avoid, and to learn of any non-standard keys that may require special handling. @@ -193,7 +193,7 @@ \subsection{Session realm attributes} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Job realm attributes} -\label{chap:res:jrealm} +\label{chap:api_rsvd_keys:jrealm} Job-related information is retrieved by including the namespace of the target job and a rank of \refconst{PMIX_RANK_WILDCARD} in the \refarg{proc} argument @@ -274,7 +274,7 @@ \subsection{Job realm attributes} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Application realm attributes} -\label{chap:res:aprealm} +\label{chap:api_rsvd_keys:aprealm} Application-related information can only be retrieved by including the \refattr{PMIX_APP_INFO} attribute in the \refarg{info} array passed to @@ -338,7 +338,7 @@ \subsection{Application realm attributes} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Process realm attributes} -\label{chap:res:prealm} +\label{chap:api_rsvd_keys:prealm} Process-related information is retrieved by referencing the namespace and rank of the target process in the call to \refapi{PMIx_Get}. If information @@ -437,7 +437,7 @@ \subsection{Process realm attributes} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Node realm keys} -\label{chap:res:nrealm} +\label{chap:api_rsvd_keys:nrealm} Information regarding the local node can be retrieved by directly requesting the node realm key in the call to \refapi{PMIx_Get} - the keys for node-related information are not shared across other realms. The target process identifier will be ignored for keys that are not dependent upon it. Information about a node other than the local node can be retrieved by specifying the \refattr{PMIX_NODE_INFO} attribute in the \refarg{info} array along with @@ -503,7 +503,7 @@ \subsection{Node realm keys} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Retrieval rules for reserved keys} -\label{chap:rkeys:retrules} +\label{chap:api_rsvd_keys:retrules} The retrieval rules for reserved keys are relatively simple as the keys are required, by definition, to be available when the client begins execution. @@ -545,7 +545,7 @@ \section{Retrieval rules for reserved keys} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Accessing information: examples} -\label{chap:api_kv:getex} +\label{chap:api_rsvd_keys:getex} This section provides examples illustrating methods for accessing information from the various realms. The intent of the examples is not to provide comprehensive coding guidance, but rather to further illustrate the use of \refapi{PMIx_Get} for obtaining information on a \refterm{session}, \refterm{job}, \refterm{application}, \refterm{process}, and \refterm{node}. @@ -602,7 +602,7 @@ \subsubsection{Session-level information} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsubsection{Job-level information} -Information regarding a job can be obtained by the methods detailed in Section \ref{chap:res:jrealm}. Example requests are shown below: +Information regarding a job can be obtained by the methods detailed in Section \ref{chap:api_rsvd_keys:jrealm}. Example requests are shown below: \cspecificstart \begin{codepar} @@ -628,7 +628,7 @@ \subsubsection{Job-level information} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsubsection{Application-level information} -Information regarding an application can be obtained by the methods described in Section \ref{chap:res:aprealm}. Example requests are shown below: +Information regarding an application can be obtained by the methods described in Section \ref{chap:api_rsvd_keys:aprealm}. Example requests are shown below: \cspecificstart \begin{codepar} @@ -674,13 +674,13 @@ \subsubsection{Application-level information} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsubsection{Process-level information} -Process-level information is accessed by providing the namespace and rank of the target process. In the absence of any directive as to the level of information being requested, the \ac{PMIx} library will always return the process-level value. See Section \ref{chap:res:prealm} for details. +Process-level information is accessed by providing the namespace and rank of the target process. In the absence of any directive as to the level of information being requested, the \ac{PMIx} library will always return the process-level value. See Section \ref{chap:api_rsvd_keys:prealm} for details. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsubsection{Node-level information} -Information regarding a node within the system can be obtained by the methods described in Section \ref{chap:res:nrealm}. Example requests are shown below: +Information regarding a node within the system can be obtained by the methods described in Section \ref{chap:api_rsvd_keys:nrealm}. Example requests are shown below: \cspecificstart \begin{codepar} diff --git a/Chap_API_Server.tex b/Chap_API_Server.tex index 6dad7f0c..c087932e 100644 --- a/Chap_API_Server.tex +++ b/Chap_API_Server.tex @@ -361,7 +361,7 @@ \subsection{\code{PMIx_server_register_nspace}} \divider -Host environments are required to provide a wide range of session-, job-, application-, node-, and process-realm information, and may choose to provide a similarly wide range of optional information. The information is broadly separated into categories based on the \emph{data realm} definitions explained in Section \ref{api:struct:attributes:retrieval}, and retrieved according to the rules detailed in Section \ref{chap:rkeys:retrules}. +Host environments are required to provide a wide range of session-, job-, application-, node-, and process-realm information, and may choose to provide a similarly wide range of optional information. The information is broadly separated into categories based on the \emph{data realm} definitions explained in Section \ref{api:struct:attributes:retrieval}, and retrieved according to the rules detailed in Section \ref{chap:api_rsvd_keys:retrules}. Session-realm information may be passed as individual \refstruct{pmix_info_t} entries, or as part of a \refstruct{pmix_data_array_t} using the \refattr{PMIX_SESSION_INFO_ARRAY} attribute. The list of data referenced in this way shall include: diff --git a/Chap_API_Sync_Access.tex b/Chap_API_Sync_Access.tex index 25373807..484f218b 100644 --- a/Chap_API_Sync_Access.tex +++ b/Chap_API_Sync_Access.tex @@ -1,179 +1,418 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -% Chapter: Synchronization and Data Access Operations +% Chapter: Data Access Operations %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\chapter{Synchronization and Data Access Operations} -\label{chap:api_sync_acc} +\chapter{Data Access and Sharing} +\label{chap:data_sharing} + +In \ac{PMIx} key-value pairs are the primary way that information is shared +between processes in the \ac{PMIx} universe. +A key-value pair consists of a string key and a value data structure +which includes the type of the value's data. +Each key is unique within the context or domain in which it exists. +The context or domain of a key in PMIx is called a data realm. +%% The pmix_info_t type (See \ref{chap:introduction:portability:attributes}), +%% is also built on key-values pairs. +%% However, pmix_info_t is used for influencing API calls and is not used for sharing +%% information between processes. +This chapter describes how key-value pairs are made available and accessed by clients. + +A key-value pair has a scope defined by its data realm. +PMIx defines data realms for sessions, jobs, applications, nodes, processes and a special +realm for publish/lookup information. +These are explained more fully in chapters \ref{chap:api_rsvd_keys} and \ref{chap:pub}. + +The source or provider of key-value pairs can be the host environment or a \ac{PMIx} client. +The host environment can provide key-values pairs in any realm except for the publish/lookup realm. +%%% Can we have a separate effort to change this restriction on publish/lookup for consistency? +%%% Would this get into too many other issues? +PMIx clients can only provide key-values pairs within their own process realm and the publish/lookup realm. +The value of a key is not necessarily static and can be modified by the original provider of the key. + +\ac{PMIx} classifies keys as either reserved or non-reserved. +Only keys provided by a host environment or \ac{PMIx} implementation can begin with \code{"pmix"} and are called reserved keys. +Host environments and \ac{PMIx} implementations must only provide keys that begin with \code{"pmix"} to avoid conflicts with keys that are provided by clients. +Reserved keys are documented in Chapter \ref{chap:api_rsvd_keys}. +Each standardized reserved key has an associated macro that is documented along with its description. +Likewise clients can only provide keys that do not begin with \code{"pmix"} to avoid conflicting with keys provided by the host environment or \ac{PMIx} implementation. +The string representation and the macro are equivalent methods to refer to a key. +Throughout this document, the macro is used rather than its string value to refer to reserved keys. +Clients cannot create reserved keys, nor can they modify the value of reserved keys. +The way reserved keys are accessed by clients +is different than for non-reserved keys as explained in Chapter \ref{chap:api_rsvd_keys}. + +Host environments are not required to provide all keys that are documented in the standard. +However, when providing keys that are documented, they must adhere to the behavior defined by the standard for reserved keys. +An implementation cannot provide a reserved key that behaves differently than what is in the standard. +Implementations are encouraged to standardize any non-standardized, reserved keys that they provide to avoid conflicting with other implementations or efforts to standardize the same key. + +Host environments are permitted to provide non-reserved keys and non-standardized reserved keys. +However there are important caveats to providing either of these categories of keys +that are explained in Section \ref{chap:data_sharing:non_rsvd_keys} and Chapter \ref{chap:api_rsvd_keys}. +% Would like to make this statement stronger, but there is text in chapter 6 currently that +% says it is allowed to use "custom-prefixed keys which may adhere to either the reserved or non-reserved +% retrieval rules. +Host environments should avoid providing reserved keys which are not standardized. +If the reserved key is later standardized to behave in a way that is different from the behavior +provided by the host environment, the host environment will not be compliant with the PMIx standard. + +Reserved keys are set by the host environment and are made available at client initialization. +Each key is associated with a data realm and the retrieval API call will indicate the realm being queried. +Reserved keys are used to access information about a client's execution context. +After initialization, the client can access reserved keys using the \refapi{PMIx_Get} API. +Common information that a client may access includes, but is not limited to, +the name of the host it is executing on, +the number of peer processes in its namespace, +the number of peer processes in its application, +and its job rank. + +%%% (Do we want to make any other statements about how and when values can change during +%%% the course of execution. +%%% PMIX_EXIT_CODE for example, UNIVERSE_SIZE could change if more resources are added?) + +Non-reserved keys are provided by clients. +A call to \refapi{PMIx_Put} will result in a key-value being associated with the calling client within +its process data realm. +There is no mechanism to retract a key-value that has been made available by a client through \refapi{PMIx_Put}. +The key can be assigned a new value, but cannot be removed entirely. +Although key values can originate from different sources, they are always retrieved using the +\refapi{PMIx_Get} and \refapi{PMIx_Get_nb} \ac{API}. + +The publish/lookup data realm is accessed through a separate set of \acp{API}. +\refapi{PMIx_Put} cannot add or modify key-values within the publish/lookup realm and +\refapi{PMIx_Lookup} cannot access key-values outside the publish/lookup realm. +This data realm is described in detail is chapter \ref{chap:pub}. +Although \refapi{PMIx_Publish} and \refapi{PMIx_Lookup} are analogous to \refapi{PMIx_Put} and \refapi{PMIx_Get} in that both pairs +of \acp{API} provide functionality for exposing and retreiving key-values, the semantics vary significantly. +For example, \refapi{PMIx_Lookup} includes the ability to wait until a key is defined before returning. +\refapi{PMIx_Publish} can restrict the scope of who can access data to its own namespace, the host environment, the session it is executing in, all processes or even custom scopes, while +\refapi{PMIx_Put} can restrict the scope of who can access data to the node it is executing on, remote nodes, +all nodes or only itself. +The publish/lookup data realm is useful for advertising information that is not necessarily specific to one process to other processes in the \ac{PMIx} universe. +The process accessing this information does \emph{not} need to know the identity of the process that provided the data. + +\ac{PMIx} does not provide a mechanism to asynchronously notify a process about the availability of key-value information once it is made available by another process. +However, the nonblocking accessor interfaces (e.g., \refapi{PMIx_Get_nb}, \refapi{PMIx_Lookup_nb}) may provide a degree of asynchronous notification on information availability. + +Process related key-value exchanges allow a \ac{PMIx} process to share information specific to +itself, and access information specific to one or more processes in the \ac{PMIx} universe. +These interactions occur within the process data realm. +The 'put/commit/get' exchange pattern is often used to exchange process related information. +Optionally, a 'put/commit/fence/get' exchange pattern adds the 'fence' synchronization +(and possible collective exchange) for applications that desire it. +Commonly, these exchange patterns are used in a +\declareterm{business card exchange} (a.k.a. \declareterm{modex exchange}) +where one \ac{PMIx} client shares its connectivity information, then other \ac{PMIx} clients access that information to establish a connection with that client. +In some environments that support ``instant-on'' all connectivity information for \ac{PMIx} clients is stored in the job-level information at process creation time and is accessible to the clients without the need to perform any additional key-value exchange. + +Keys can exist in multiple data realms, possibly with different values. +\ac{PMIx} clients can access available information associated with any of the data realms. +For example, a client can access the number of nodes (\refattr{PMIX_NUM_NODES}) used by a +session, job, or application. +Rather than having three different attributes, a single attribute is used but with the +data realm context of the query specified as additional attributes. +Examples of these access patterns are presented in Section \ref{chap:api_rsvd_keys:getex}. + +\ac{PMIx} clients can share key-value pairs associated with themselves by using the \refapi{PMIx_Put} function. +The \refapi{PMIx_Put} function automatically associates the key-value pair with the calling process, thus making it specific to that process. +A client may call \refapi{PMIx_Put} as many times as necessary and the data is not available to other processes until explicitly committed. +A client must call \refapi{PMIx_Commit} to make accessible all key-value pairs previously put by this process to all other processes in the \ac{PMIx} universe. +This put and commit pattern provides implementors the opportunity to make individual \refapi{PMIx_Put} calls efficient local operations, and then make the whole set of key-value pairs accessible in a single step. + +\ac{PMIx} clients can access the key-value pairs associated with any process data realm +in the \ac{PMIx} universe (including the calling process) by passing the specific process +name of the target process to the \refapi{PMIx_Get} and \refapi{PMIx_Get_nb} functions. +The \ac{PMIx} server local to the calling process will retrieve that key-value pair from +the \ac{PMIx} server associated with the target process. +Clients can also access session, job, application, node, and namespace level information +by using the \refapi{PMIx_Get} and \refapi{PMIx_Get_nb} functions as shown in Section +\ref{chap:api_rsvd_keys:getex}. +% and summarized in Table~\ref{tab:key-value-realms}. (Add in next release?) +The completion semantics for \refapi{PMIx_Get} and \refapi{PMIx_Get_nb} differ +depending on the type of key and its availability. See Sections +\ref{chap:data_sharing:retrules} and \ref{chap:api_rsvd_keys:retrules}. +For example, if a non-reserved key is not available, the \refapi{PMIx_Get} or \refapi{PMIx_Get_nb} call +will not complete, by default, until that key-value pair becomes available. -Applications may need to synchronize their operations at various points in -their execution. Depending on a variety of factors (e.g., the programming -model and where the synchronization point lies), the application may choose to -execute the operation using \ac{PMIx}. This is particularly useful in -situations where communication by other means is not yet available since -\ac{PMIx} relies on the host environment's infrastructure for such operations. +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Non-reserved keys} +\label{chap:data_sharing:non_rsvd_keys} -Synchronization operations also offer an opportunity for processes to exchange -data at a known point in their execution. Where required, this can include -information on communication endpoints for subsequent wireup of various -messaging protocols. +\emph{Non-reserved keys} are keys whose string representation begin with a +prefix other than \code{"pmix"}. Such keys are defined by a +client when information needs to be exchanged between processes, for example, +where connection information is required and the host environment does not +support the \refterm{instant on} option or where the host environment does not +provide a required piece of data. Other than the prefix, there are no +restrictions on the use or content of non-reserved keys. + +\ac{PMIx} provides support for two methods of exchanging non-reserved keys: + +\begin{itemize} + \item Global, collective exchange of the information prior to retrieval. This is accomplished by executing a barrier operation that includes collection and exchange of the data provided by each process such that each process has access to the full set of data from all participants once the operation has completed. \ac{PMIx} provides the \refapi{PMIx_Fence} function (or its non-blocking equivalent) for this purpose, accompanied by the \refattr{PMIX_COLLECT_DATA} qualifier. + \item Direct, on-demand retrieval of the information. No barrier or global exchange is conducted in this case. Instead, information is retrieved from the host where that process is executing upon request - i.e., a call to \refapi{PMIx_Get} results in a data exchange with the \ac{PMIx} server on the remote host. Various caching strategies may be employed by the host environment and/or \ac{PMIx} implementation to reduce the number of retrievals. Note that this method requires that the host environment both know the location of the posting process and support direct information retrieval. +\end{itemize} + +Both of the above methods are based on retrieval from a specific process - +i.e., the \refarg{proc} argument to \refapi{PMIx_Get} must include both the +namespace and the rank of the process that posted the information. However, in +some cases, non-reserved keys are provided on a globally unique basis and the +retrieving process has no knowledge of the identity of the process posting the +key. This is typically found in legacy applications (where the originating +process identifier is often embedded in the key itself) and in unstructured +applications that lack rank-related behavior. In these cases, the key remains +associated with the namespace of the process that posted it, but is retrieved +by use of the \refconst{PMIX_RANK_UNDEF} rank. In addition, the keys must be +globally exchanged prior to retrieval as there is no way for the host to +otherwise locate the source for the information. + +Note that the retrieval rules for non-reserved keys (detailed in Section +\ref{chap:data_sharing:retrules}) differ significantly from those used for reserved keys. -This chapter covers both the synchronization and data retrieval functions -provided under the \ac{PMIx} Standard. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\section{\code{PMIx_Fence}} -\declareapi{PMIx_Fence} +\section{Posting Key/Value Pairs} +\label{chap:data_sharing:set} + +\ac{PMIx} clients can post non-reserved key-value pairs associated with themselves by using \refapi{PMIx_Put}. Alternatively, \ac{PMIx} clients can cache both reserved and non-reserved key-value pairs accessible only by the caller via the \refapi{PMIx_Store_internal} \ac{API}. + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\subsection{\code{PMIx_Put}} +\declareapi{PMIx_Put} %%%% \summary -Execute a blocking barrier across the processes identified in the specified array, collecting information posted via \refapi{PMIx_Put} as directed. +Stage a key/value pair in prepartion for being made accessible to processes. %%%% \format -\copySignature{PMIx_Fence}{1.0}{ +\copySignature{PMIx_Put}{1.0}{ pmix_status_t \\ -PMIx_Fence(const pmix_proc_t procs[], size_t nprocs, \\ -\hspace*{11\sigspace}const pmix_info_t info[], size_t ninfo); +PMIx_Put(pmix_scope_t scope, \\ +\hspace*{9\sigspace}const pmix_key_t key, \\ +\hspace*{9\sigspace}pmix_value_t *val); } \begin{arglist} -\argin{procs}{Array of \refstruct{pmix_proc_t} structures (array of handles)} -\argin{nprocs}{Number of elements in the \refarg{procs} array (integer)} -\argin{info}{Array of info structures (array of handles)} -\argin{ninfo}{Number of elements in the \refarg{info} array (integer)} +\argin{scope}{Distribution scope of the provided value (handle)} +\argin{key}{key (\refstruct{pmix_key_t})} +\argin{value}{Reference to a \refstruct{pmix_value_t} structure (handle)} \end{arglist} -\returnsimple - -\reqattrstart -The following attributes are required to be supported by all \ac{PMIx} libraries: - -\pasteAttributeItem{PMIX_COLLECT_DATA} -\pasteAttributeItem{PMIX_COLLECT_GENERATED_JOB_INFO} - -\reqattrend - -\optattrstart -The following attributes are optional for \ac{PMIx} implementations: +\returnstart +\begin{itemize} +\item \refconst{PMIX_ERR_BAD_PARAM} indicating a reserved key is provided in the \refarg{key} argument. +\end{itemize} +\returnend -\pasteAttributeItem{PMIX_ALL_CLONES_PARTICIPATE} +%%%% +\descr +Post a key-value pair for distribution. Depending upon the \ac{PMIx} implementation, the posted value may be locally cached in the client's \ac{PMIx} library until \refapi{PMIx_Commit} is called. -The following attributes are optional for host environments: +The provided \refarg{scope} determines the ability of other processes to access the posted data, as defined in \specrefstruct{pmix_scope_t}. +Specific implementations may support different scope values, but all implementations must support at least \refconst{PMIX_GLOBAL}. -\pasteAttributeItem{PMIX_TIMEOUT} +The \refstruct{pmix_value_t} structure supports both string and binary values. +\ac{PMIx} implementations are required to support heterogeneous environments by properly converting binary values between host architectures, and will copy the provided \refarg{value} into internal memory prior to returning from \refapi{PMIx_Put}. -\optattrend +\adviceuserstart +Note that keys starting with a string of ``\code{pmix}'' must not be used in calls to \refapi{PMIx_Put}. Thus, applications should never use a defined ``PMIX'' attribute as the key in a call to \refapi{PMIx_Put}. +\adviceuserend -%%%% -\descr -Passing a \code{NULL} pointer as the \refarg{procs} parameter indicates that the fence is to span all processes in the client's namespace. -Each provided \refstruct{pmix_proc_t} struct can pass \refconst{PMIX_RANK_WILDCARD} to indicate that all processes in the given namespace are participating. +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\subsubsection{Scope of Put Data} +\declarestruct{pmix_scope_t} +\label{api:nres:scope} -The \refarg{info} array is used to pass user directives regarding the behavior of the fence operation. Note that for scalability reasons, the default behavior for \refapi{PMIx_Fence} is to not collect data posted by the operation's participants. +\versionMarker{1.0} +The \refstruct{pmix_scope_t} structure is a \code{uint8_t} type that defines the availability of data passed to \refapi{PMIx_Put}. +The following constants can be used to set a variable of the type \refstruct{pmix_scope_t}. All definitions were introduced in version 1 of the standard unless otherwise marked. -\adviceimplstart -\refapi{PMIx_Fence} and its non-blocking form are both \emph{collective} operations. Accordingly, the \ac{PMIx} server library is required to aggregate participation by local clients, passing the request to the host environment once all local participants have executed the \ac{API}. -\adviceimplend +Specific implementations may support different scope values, but all implementations must support at least \refconst{PMIX_GLOBAL}. +If a specified scope value is not supported, then the \refapi{PMIx_Put} call must return \refconst{PMIX_ERR_NOT_SUPPORTED}. -\advicermstart -The host will receive a single call for each collective operation. It is the responsibility of the host to identify the nodes containing participating processes, execute the collective across all participating nodes, and notify the local \ac{PMIx} server library upon completion of the global collective. -\advicermend +\begin{constantdesc} +% +\declareconstitemvalue{PMIX_SCOPE_UNDEF}{0} +Undefined scope. +% +\declareconstitemvalue{PMIX_LOCAL}{1} +The data is intended only for other application processes on the same node. +Data marked in this way will not be included in data packages sent to remote requesters - i.e., it is only available to processes on the local node. +% +\declareconstitemvalue{PMIX_REMOTE}{2} +The data is intended solely for applications processes on remote nodes. +Data marked in this way will not be shared with other processes on the same node - i.e., it is only available to processes on remote nodes. +% +\declareconstitemvalue{PMIX_GLOBAL}{3} +The data is to be shared with all other requesting processes, regardless of location. +% +\versionMarker{2.0} +\declareconstitemvalue{PMIX_INTERNAL}{4} +The data is intended solely for this process and is not shared with other processes. +% +\end{constantdesc} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\section{\code{PMIx_Fence_nb}} -\declareapi{PMIx_Fence_nb} +\subsection{\code{PMIx_Store_internal}} +\declareapi{PMIx_Store_internal} %%%% \summary -Execute a nonblocking \refapi{PMIx_Fence} across the processes identified in the specified array of processes, collecting information posted via \refapi{PMIx_Put} as directed. +Store some data locally for retrieval by other areas of the process. %%%% \format -\copySignature{PMIx_Fence_nb}{1.0}{ +\copySignature{PMIx_Store_internal}{1.0}{ pmix_status_t \\ -PMIx_Fence_nb(const pmix_proc_t procs[], size_t nprocs, \\ -\hspace*{14\sigspace}const pmix_info_t info[], size_t ninfo, \\ -\hspace*{14\sigspace}pmix_op_cbfunc_t cbfunc, void *cbdata); +PMIx_Store_internal(const pmix_proc_t *proc, \\ +\hspace*{20\sigspace}const pmix_key_t key, \\ +\hspace*{20\sigspace}pmix_value_t *val); } \begin{arglist} -\argin{procs}{Array of \refstruct{pmix_proc_t} structures (array of handles)} -\argin{nprocs}{Number of elements in the \refarg{procs} array (integer)} -\argin{info}{Array of info structures (array of handles)} -\argin{ninfo}{Number of elements in the \refarg{info} array (integer)} -\argin{cbfunc}{Callback function (function reference)} -\argin{cbdata}{Data to be passed to the callback function (memory reference)} +\argin{proc}{process reference (handle)} +\argin{key}{key to retrieve (string)} +\argin{val}{Value to store (handle)} \end{arglist} -\returnsimplenb - \returnstart \begin{itemize} - \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will \textit{not} be called. This can occur if the collective involved only processes on the local node. +\item \refconst{PMIX_ERR_BAD_PARAM} indicating a reserved key is provided in the \refarg{key} argument. \end{itemize} \returnend -\reqattrstart -The following attributes are required to be supported by all \ac{PMIx} libraries: +%%%% +\descr -\pasteAttributeItem{PMIX_COLLECT_DATA} -\pasteAttributeItem{PMIX_COLLECT_GENERATED_JOB_INFO} +Store some data locally for retrieval by other areas of the process. +This is data that has only internal scope - it will never be posted externally. Typically used to cache data obtained by means outside of \ac{PMIx} so that it can be accessed by various areas of the process. -\reqattrend -\optattrstart -The following attributes are optional for \ac{PMIx} implementations: +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\subsection{\code{PMIx_Commit}} +\declareapi{PMIx_Commit} -\pasteAttributeItem{PMIX_ALL_CLONES_PARTICIPATE} +%%%% +\summary +Make available to other processes all key-value pairs previously staged via \refapi{PMIx_Put}. -The following attributes are optional for host environments that support this operation: +%%%% +\format -\pasteAttributeItem{PMIX_TIMEOUT} +\copySignature{PMIx_Commit}{1.0}{ +pmix_status_t PMIx_Commit(void); +} -\optattrend +\returnsimple %%%% \descr -Nonblocking version of the \refapi{PMIx_Fence} routine. See the \refapi{PMIx_Fence} description for further details. +\ac{PMIx} implementations may choose to locally cache non-reserved keys prior to submitting them for distribution. \refapi{PMIx_Commit} initiates the operation of making available previously staged key-value pairs to other processes. Depending on the implementation, this may involve transmitting the entire collection of data posted by the process to a server. \refapi{PMIx_Commit} is an asynchronous operation that will immediately return to the caller while the data is staged in the background. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\subsection{Fence-related attributes} +\adviceuserstart +Users are advised to always include the call to \refapi{PMIx_Commit} in case the local implementation requires it. Note that posted data will not necessarily be circulated during \refapi{PMIx_Commit}. Availability of the data by other processes upon completion of \refapi{PMIx_Commit} therefore still relies upon the exchange mechanisms described at the beginning of this chapter. +\adviceuserend -The following attributes are defined specifically to support the fence operation: -% -\declareAttribute{PMIX_COLLECT_DATA}{"pmix.collect"}{bool}{ -Collect all data posted by the participants using \refapi{PMIx_Put} that -has been committed via \refapi{PMIx_Commit}, making the collection locally -available to each participant at the end of the operation. By default, this will include all job-level information that was locally generated by \ac{PMIx} servers unless excluded using the \refattr{PMIX_COLLECT_GENERATED_JOB_INFO} attribute. -} -% -\declareAttributeProvisional{PMIX_LOCAL_COLLECTIVE_STATUS}{"pmix.loc.col.st"}{pmix_status_t}{ -Status code for local collective operation being reported to the host by the server library. PMIx servers may aggregate the participation by local client processes in a collective operation - e.g., instead of passing individual client calls to \refapi{PMIx_Fence} up to the host environment, the server may pass only a single call to the host when all local participants have executed their \refapi{PMIx_Fence} call, thereby reducing the burden placed on the host. However, in cases where the operation locally fails (e.g., if a participating client abnormally terminates prior to calling the operation), the server upcall functions to the host do not include a \refstruct{pmix_status_t} by which the PMIx server can alert the host to that failure. This attribute resolves that problem by allowing the server to pass the status information regarding the local collective operation. -} -\advicermstart -The PMIx server is allowed to pass \refconst{PMIX_SUCCESS} using this attribute, but is not required to do so. PMIx implementations may choose to only report errors in this manner. The lack of an included status shall therefore be taken to indicate that the collective operation locally succeeded. -\advicermend +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Retrieval rules for non-reserved keys} +\label{chap:data_sharing:retrules} + +Since non-reserved keys cannot, by definition, have been provided by the host +environment, their retrieval follows significantly different rules than those +defined for reserved keys (as detailed in Section \ref{chap:api_rsvd_keys}). +\refapi{PMIx_Get} for a non-reserved key will obey the +following precedence search: + +\begin{enumerate} + \item If the \refattr{PMIX_GET_REFRESH_CACHE} attribute is given, then the + request is first forwarded to the local \ac{PMIx} server which will then + update the client's cache. Note that this may not, depending upon + implementation details, result in any action. + + \item Check the local \ac{PMIx} client cache for the requested key - if not found and either the \refattr{PMIX_OPTIONAL} or \refattr{PMIX_GET_REFRESH_CACHE} attribute was given, the search will stop at this point and return the \refconst{PMIX_ERR_NOT_FOUND} status. + + \item Request the information from the local \ac{PMIx} server. The server + will check its cache for the specified key. If the + value still isn't found and the \refattr{PMIX_IMMEDIATE} attribute was + given, then the library shall return the \refconst{PMIX_ERR_NOT_FOUND} + error constant to the requester. Otherwise, the \ac{PMIx} server library + will take one of the following actions: + \begin{compactitemize} + \item If the target process has a rank of \refconst{PMIX_RANK_UNDEF}, + then this indicates that the key being requested is globally unique + and \emph{not} associated with a specific process. In this case, the + server shall hold the request until either the data appears at the + server or, if given, the \refattr{PMIX_TIMEOUT} is reached. In the + latter case, the server will return the \refconst{PMIX_ERR_TIMEOUT} + status. Note that the server may, depending on \ac{PMIx} + implementation, never respond if the caller failed to specify a + \refattr{PMIX_TIMEOUT} and the requested key fails to arrive at the + server. + + \item If the target process is \emph{local} (i.e., attached to the + same \ac{PMIx} server), then the server will hold the request until + either the target process provides the data or, if given, the + \refattr{PMIX_TIMEOUT} is reached. In the latter case, the server will + return the \refconst{PMIX_ERR_TIMEOUT} status. Note that data which is + posted via \refapi{PMIx_Put} but not staged with \refapi{PMIx_Commit} + may, depending upon implementation, never appear at the server. + + \item If the target process is \emph{remote} (i.e., not attached to + the same \ac{PMIx} server), the server will either: + \begin{compactitemize} + \item If the host has provided the + \refapi{pmix_server_dmodex_req_fn_t} module function + interface, then the server + shall pass the request to its host for servicing. The host is + responsible for determining the location of the target process and + passing the request to the \ac{PMIx} server at that location. + + When the remote data request is received, the target \ac{PMIx} + server will check its cache for the specified key. If the key is + not present, the request shall be held until either the target + process provides the data or, if given, the \refattr{PMIX_TIMEOUT} + is reached. In the latter case, the server will return the + \refconst{PMIX_ERR_TIMEOUT} status. The host shall convey the + result back to the originating \ac{PMIx} server, which will reply + to the requesting client with the result of the request when the + host provides it. + + Note that the target server may, depending on \ac{PMIx} + implementation, never respond if the caller failed to specify a + \refattr{PMIX_TIMEOUT} and the target process fails to post the + requested key. + + \item if the host does not support the + \refapi{pmix_server_dmodex_req_fn_t} interface, then + the server will immediately respond to the client with the + \refconst{PMIX_ERR_NOT_FOUND} status + \end{compactitemize} + \end{compactitemize} +\end{enumerate} -% -\declareAttributeNEW{PMIX_COLLECT_GENERATED_JOB_INFO}{"pmix.collect.gen"}{bool}{ -Collect all job-level information (i.e., reserved keys) that was locally generated by \ac{PMIx} servers. Some job-level information (e.g., distance between processes and fabric devices) is best determined on a distributed basis as it primarily pertains to local processes. Should remote processes need to access the information, it can either be obtained collectively using the \refapi{PMIx_Fence} operation with this directive, or can be retrieved one peer at a time using \refapi{PMIx_Get} without first having performed the job-wide collection. -} -% -\declareAttributeNEW{PMIX_ALL_CLONES_PARTICIPATE}{"pmix.clone.part"}{bool}{ -All \refterm{clones} of the calling process must participate in the collective operation. -} +\adviceimplstart +While there is no requirement that all \ac{PMIx} implementations follow the +client-server paradigm used in the above description, implementers are +required to provide behaviors consistent with the described search pattern. +\adviceimplend +\adviceuserstart +Users are advised to always specify the \refattr{PMIX_TIMEOUT} value when +retrieving non-reserved keys to avoid potential deadlocks should the specified +key not become available. +\adviceuserend %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -239,15 +478,23 @@ \section{\code{PMIx_Get}} %%%% \descr -Retrieve information for the specified \refarg{key} associated with the process identified in the given \refstruct{pmix_proc_t}. See Chapters \ref{chap:api_rsvd_keys} and \ref{chap:nrkeys} for details on rules governing retrieval of information. Information will be returned according to provided directives: +Retrieve information for the specified \refarg{key} associated with the process identified in the given \refstruct{pmix_proc_t}. See Chapters \ref{chap:api_rsvd_keys} and \ref{chap:data_sharing:non_rsvd_keys} for details on rules governing retrieval of information. Information will be returned according to provided directives: \begin{itemize} \item In the absence of any directive, the returned \refstruct{pmix_value_t} shall be an allocated memory object. The caller is responsible for releasing the object when done. \item If \refattr{PMIX_GET_POINTER_VALUES} is given, then the function shall return a pointer to a \refstruct{pmix_value_t} in the \ac{PMIx} library's memory that contains the requested information. - \item If \refattr{PMIX_GET_STATIC_VALUES} is given, then the function shall return the information in the provided \refstruct{pmix_value_t} pointer. In this case, the caller must provide storage for the structure and pass the pointer to that storage in the \refarg{val} parameter. If the implementation cannot return a static value, then the call to \refapi{PMIx_Get} must return the \refconst{PMIX_ERR_NOT_SUPPORTED} status. + \item If \refattr{PMIX_GET_STATIC_VALUES} is given, then the function shall return the information in the provided \refstruct{pmix_value_t} pointer. In this case, the caller must provide storage for the structure and pass the pointer to that storage in the \refarg{val} parameter. If the implementation cannot return a static value, then the call to \refapi{PMIx_Get} must return the \refconst{PMIX_ERR_NOT_SUPPORTED} status. \end{itemize} -This is a blocking operation - the caller will block until the retrieval rules of Chapters \ref{chap:api_rsvd_keys} or \ref{chap:nrkeys} are met. +Retrieve information for the specified \refarg{key} associated with the process identified in the given \refstruct{pmix_proc_t}. See Chapters \ref{chap:api_rsvd_keys} and \ref{chap:data_sharing:non_rsvd_keys} for details on rules governing retrieval of information. Information will be returned according to provided directives: + +\begin{itemize} + \item In the absence of any directive, the returned \refstruct{pmix_value_t} shall be an allocated memory object. The caller is responsible for releasing the object when done. + \item If \refattr{PMIX_GET_POINTER_VALUES} is given, then the function shall return a pointer to a \refstruct{pmix_value_t} in the \ac{PMIx} library's memory that contains the requested information. + \item If \refattr{PMIX_GET_STATIC_VALUES} is given, then the function shall return the information in the provided \refstruct{pmix_value_t} pointer. In this case, the caller must provide storage for the structure and pass the pointer to that storage in the \refarg{val} parameter. +\end{itemize} + +This is a blocking operation - the caller will block until the retrieval rules of Section \ref{chap:api_rsvd_keys:retrules} or \ref{chap:data_sharing:retrules} are met. The \refarg{info} array is used to pass user directives regarding the get operation. @@ -263,12 +510,15 @@ \subsection{\code{PMIx_Get_nb}} %%%% \format -\copySignature{PMIx_Get_nb}{1.0}{ -pmix_status_t \\ -PMIx_Get_nb(const pmix_proc_t *proc, const char key[], \\ -\hspace*{12\sigspace}const pmix_info_t info[], size_t ninfo, \\ -\hspace*{12\sigspace}pmix_value_cbfunc_t cbfunc, void *cbdata); -} +\versionMarker{1.0} +\cspecificstart +\begin{codepar} +pmix_status_t +PMIx_Get_nb(const pmix_proc_t *proc, const char key[], + const pmix_info_t info[], size_t ninfo, + pmix_value_cbfunc_t cbfunc, void *cbdata); +\end{codepar} +\cspecificend \begin{arglist} \argin{proc}{Process identifier - a \code{NULL} value may be used in place of the caller's ID (handle)} @@ -321,7 +571,7 @@ \subsection{\code{PMIx_Get_nb}} %%%% \descr -The callback function will be executed once the retrieval rules of Chapters \ref{chap:api_rsvd_keys} or \ref{chap:nrkeys} are met. +The callback function will be executed once the retrieval rules of Chapters \ref{chap:api_rsvd_keys} or \ref{chap:data_sharing:retrules} are met. See \refapi{PMIx_Get} for a full description. Note that the non-blocking form of this function cannot support the \refattr{PMIX_GET_STATIC_VALUES} attribute as the user cannot pass in the required pointer to storage for the result. @@ -365,9 +615,186 @@ \subsection{Retrieval attributes} Caller requests that the \ac{PMIx} server wait until at least the specified number of values are found (a value of zero indicates \emph{all} and is the default). } +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% Chapter: Synchronization and Data Access Operations +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\chapter{Synchronization and Data Access Operations} +\label{chap:api_sync_acc} + +Applications may need to synchronize their operations at various points in +their execution. Depending on a variety of factors (e.g., the programming +model and where the synchronization point lies), the application may choose to +execute the operation using \ac{PMIx}. This is particularly useful in +situations where communication by other means is not yet available since +\ac{PMIx} relies on the host environment's infrastructure for such operations. + +Synchronization operations also offer an opportunity for processes to exchange +data at a known point in their execution. Where required, this can include +information on communication endpoints for subsequent wireup of various +messaging protocols. + +This chapter covers both the synchronization and data retrieval functions +provided under the \ac{PMIx} Standard. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{\code{PMIx_Fence}} +\declareapi{PMIx_Fence} + +%%%% +\summary + +Execute a blocking barrier across the processes identified in the specified array, collecting information posted via \refapi{PMIx_Put} as directed. + +%%%% +\format + +\copySignature{PMIx_Fence}{1.0}{ +pmix_status_t \\ +PMIx_Fence(const pmix_proc_t procs[], size_t nprocs, \\ +\hspace*{11\sigspace}const pmix_info_t info[], size_t ninfo); +} + +\begin{arglist} +\argin{procs}{Array of \refstruct{pmix_proc_t} structures (array of handles)} +\argin{nprocs}{Number of elements in the \refarg{procs} array (integer)} +\argin{info}{Array of info structures (array of handles)} +\argin{ninfo}{Number of elements in the \refarg{info} array (integer)} +\end{arglist} + +\returnsimple + +\reqattrstart +The following attributes are required to be supported by all \ac{PMIx} libraries: + +\pasteAttributeItem{PMIX_COLLECT_DATA} +\pasteAttributeItem{PMIX_COLLECT_GENERATED_JOB_INFO} + +\reqattrend + +\optattrstart +The following attributes are optional for \ac{PMIx} implementations: + +\pasteAttributeItem{PMIX_ALL_CLONES_PARTICIPATE} + + +The following attributes are optional for host environments: + +\pasteAttributeItem{PMIX_TIMEOUT} + +\optattrend + +%%%% +\descr + +Passing a \code{NULL} pointer as the \refarg{procs} parameter indicates that the fence is to span all processes in the client's namespace. +Each provided \refstruct{pmix_proc_t} struct can pass \refconst{PMIX_RANK_WILDCARD} to indicate that all processes in the given namespace are participating. + +The \refarg{info} array is used to pass user directives regarding the behavior of the fence operation. Note that for scalability reasons, the default behavior for \refapi{PMIx_Fence} is to not collect data posted by the operation's participants. + +\adviceimplstart +\refapi{PMIx_Fence} and its non-blocking form are both \emph{collective} operations. Accordingly, the \ac{PMIx} server library is required to aggregate participation by local clients, passing the request to the host environment once all local participants have executed the \ac{API}. +\adviceimplend + +\advicermstart +The host will receive a single call for each collective operation. It is the responsibility of the host to identify the nodes containing participating processes, execute the collective across all participating nodes, and notify the local \ac{PMIx} server library upon completion of the global collective. +\advicermend + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{\code{PMIx_Fence_nb}} +\declareapi{PMIx_Fence_nb} + +%%%% +\summary + +Execute a nonblocking \refapi{PMIx_Fence} across the processes identified in the specified array of processes, collecting information posted via \refapi{PMIx_Put} as directed. + +%%%% +\format + +\copySignature{PMIx_Fence_nb}{1.0}{ +pmix_status_t \\ +PMIx_Fence_nb(const pmix_proc_t procs[], size_t nprocs, \\ +\hspace*{14\sigspace}const pmix_info_t info[], size_t ninfo, \\ +\hspace*{14\sigspace}pmix_op_cbfunc_t cbfunc, void *cbdata); +} + +\begin{arglist} +\argin{procs}{Array of \refstruct{pmix_proc_t} structures (array of handles)} +\argin{nprocs}{Number of elements in the \refarg{procs} array (integer)} +\argin{info}{Array of info structures (array of handles)} +\argin{ninfo}{Number of elements in the \refarg{info} array (integer)} +\argin{cbfunc}{Callback function (function reference)} +\argin{cbdata}{Data to be passed to the callback function (memory reference)} +\end{arglist} + +\returnsimplenb + +\returnstart +\begin{itemize} + \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will \textit{not} be called. This can occur if the collective involved only processes on the local node. +\end{itemize} +\returnend + +\reqattrstart +The following attributes are required to be supported by all \ac{PMIx} libraries: + +\pasteAttributeItem{PMIX_COLLECT_DATA} +\pasteAttributeItem{PMIX_COLLECT_GENERATED_JOB_INFO} + +\reqattrend + +\optattrstart +The following attributes are optional for \ac{PMIx} implementations: + +\pasteAttributeItem{PMIX_ALL_CLONES_PARTICIPATE} + + +The following attributes are optional for host environments that support this operation: + +\pasteAttributeItem{PMIX_TIMEOUT} + +\optattrend + +%%%% +\descr + +Nonblocking version of the \refapi{PMIx_Fence} routine. See the \refapi{PMIx_Fence} description for further details. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\subsection{Fence-related attributes} + +The following attributes are defined specifically to support the fence operation: + +% +\declareAttribute{PMIX_COLLECT_DATA}{"pmix.collect"}{bool}{ +Collect all data posted by the participants using \refapi{PMIx_Put} that +has been committed via \refapi{PMIx_Commit}, making the collection locally +available to each participant at the end of the operation. By default, this will include all job-level information that was locally generated by \ac{PMIx} servers unless excluded using the \refattr{PMIX_COLLECT_GENERATED_JOB_INFO} attribute. +} + +\declareAttributeProvisional{PMIX_LOCAL_COLLECTIVE_STATUS}{"pmix.loc.col.st"}{pmix_status_t}{ +Status code for local collective operation being reported to the host by the server library. PMIx servers may aggregate the participation by local client processes in a collective operation - e.g., instead of passing individual client calls to \refapi{PMIx_Fence} up to the host environment, the server may pass only a single call to the host when all local participants have executed their \refapi{PMIx_Fence} call, thereby reducing the burden placed on the host. However, in cases where the operation locally fails (e.g., if a participating client abnormally terminates prior to calling the operation), the server upcall functions to the host do not include a \refstruct{pmix_status_t} by which the PMIx server can alert the host to that failure. This attribute resolves that problem by allowing the server to pass the status information regarding the local collective operation. +} +\advicermstart +The PMIx server is allowed to pass \refconst{PMIX_SUCCESS} using this attribute, but is not required to do so. PMIx implementations may choose to only report errors in this manner. The lack of an included status shall therefore be taken to indicate that the collective operation locally succeeded. +\advicermend +% +\declareAttributeNEW{PMIX_COLLECT_GENERATED_JOB_INFO}{"pmix.collect.gen"}{bool}{ +Collect all job-level information (i.e., reserved keys) that was locally generated by \ac{PMIx} servers. Some job-level information (e.g., distance between processes and fabric devices) is best determined on a distributed basis as it primarily pertains to local processes. Should remote processes need to access the information, it can either be obtained collectively using the \refapi{PMIx_Fence} operation with this directive, or can be retrieved one peer at a time using \refapi{PMIx_Get} without first having performed the job-wide collection. +} +% +\declareAttributeNEW{PMIX_ALL_CLONES_PARTICIPATE}{"pmix.clone.part"}{bool}{ +All \refterm{clones} of the calling process must participate in the collective operation. +} + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + \section{Query} \label{chap:api_proc_mgmt:query} @@ -669,7 +1096,7 @@ \subsection{Query-specific constants} \begin{constantdesc} % -\declareconstitemvalue{PMIX_QUERY_PARTIAL_SUCCESS}{-104} +\declareconstitem{PMIX_QUERY_PARTIAL_SUCCESS} Some, but not all, of the requested information was returned. % \end{constantdesc} @@ -870,7 +1297,6 @@ \subsubsection{Query structure support macros} \end{codepar} \cspecificend - \littleheader{Initialize the query structure} \declaremacro{PMIX_QUERY_CONSTRUCT} diff --git a/Makefile b/Makefile index 06cea6d1..4a73642c 100644 --- a/Makefile +++ b/Makefile @@ -16,7 +16,6 @@ CHAPTERS= \ Chap_API_Init.tex \ Chap_API_Sync_Access.tex \ Chap_API_Reserved_Keys.tex \ - Chap_API_NonReserved_Keys.tex \ Chap_API_Publish.tex \ Chap_API_Proc_Mgmt.tex \ Chap_API_Job_Mgmt.tex \ diff --git a/pmix-standard.tex b/pmix-standard.tex index 7d4cd6fd..449f6e34 100644 --- a/pmix-standard.tex +++ b/pmix-standard.tex @@ -191,7 +191,6 @@ % - put, get, commit, fence, (un)publish, lookup \input{Chap_API_Sync_Access.tex} \input{Chap_API_Reserved_Keys.tex} - \input{Chap_API_NonReserved_Keys.tex} \input{Chap_API_Publish.tex} % Event Handling