Chap_API_NonReserved_Keys.tex

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Chapter: Process-Related Non-Reserved Keys
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Process-Related Non-Reserved Keys}
\label{chap:nrkeys}

\emph{Non-reserved keys} are keys whose string representation begin with a
prefix other than \code{"pmix"}. Such keys are typically defined by an
application when information needs to be exchanged between processes (e.g.,
where connection information is required and the host environment does not
support the \refterm{instant on} option) or where the host environment does not
provide a required piece of data. Beyond the restriction on name prefix,
non-reserved keys are required to be unique across conflicting \emph{scopes} as defined in Section \ref{api:nres:scope} - e,g., a non-reserved key cannot be posted by the same process in both the \refconst{PMIX_LOCAL} and \refconst{PMIX_REMOTE} scopes (note that posting the key in the \refconst{PMIX_GLOBAL} scope would have met the desired objective).

\ac{PMIx} provides support for two methods of exchanging non-reserved keys:

\begin{itemize}
    \item Global, collective exchange of the information prior to retrieval. This is accomplished by executing a barrier operation that includes collection and exchange of the data provided by each process such that each process has access to the full set of data from all participants once the operation has completed. \ac{PMIx} provides the \refapi{PMIx_Fence} function (or its non-blocking equivalent) for this purpose, accompanied by the \refattr{PMIX_COLLECT_DATA} qualifier.
    \item Direct, on-demand retrieval of the information. No barrier or global exchange is conducted in this case. Instead, information is retrieved from the host where that process is executing upon request - i.e., a call to \refapi{PMIx_Get} results in a data exchange with the \ac{PMIx} server on the remote host. Various caching strategies may be employed by the host environment and/or \ac{PMIx} implementation to reduce the number of retrievals. Note that this method requires that the host environment both know the location of the posting process and support direct information retrieval.
\end{itemize}

Both of the above methods are based on retrieval from a specific process -
i.e., the \refarg{proc} argument to \refapi{PMIx_Get} must include both the
namespace and the rank of the process that posted the information. However, in
some cases, non-reserved keys are provided on a globally unique basis and the
retrieving process has no knowledge of the identity of the process posting the
key. This is typically found in legacy applications (where the originating
process identifier is often embedded in the key itself) and in unstructured
applications that lack rank-related behavior. In these cases, the key remains
associated with the namespace of the process that posted it, but is retrieved
by use of the \refconst{PMIX_RANK_UNDEF} rank. In addition, the keys must be
globally exchanged prior to retrieval as there is no way for the host to
otherwise locate the source for the information.

Note that the retrieval rules for non-reserved keys (detailed in Section \ref{chap:nrkeys:retrules}) differ significantly from those used for reserved keys.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Posting Key/Value Pairs}
\label{chap:api_kv_mgmt:set}

\ac{PMIx} clients can post non-reserved key-value pairs associated with themselves by using \refapi{PMIx_Put}. Alternatively, \ac{PMIx} clients can cache arbitrary key-value pairs accessible only by the caller via the \refapi{PMIx_Store_internal} \ac{API}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_Put}}
\declareapi{PMIx_Put}

%%%%
\summary

Post a key/value pair for distribution.

%%%%
\format

\copySignature{PMIx_Put}{1.0}{
pmix_status_t \\
PMIx_Put(pmix_scope_t scope, \\
\hspace*{9\sigspace}const pmix_key_t key, \\
\hspace*{9\sigspace}pmix_value_t *val);
}

\begin{arglist}
\argin{scope}{Distribution scope of the provided value (handle)}
\argin{key}{key (\refstruct{pmix_key_t})}
\argin{value}{Reference to a \refstruct{pmix_value_t} structure (handle)}
\end{arglist}

\returnstart
\begin{itemize}
\item \refconst{PMIX_ERR_BAD_PARAM} indicating a reserved key is provided in the \refarg{key} argument.
\end{itemize}
\returnend

%%%%
\descr

Post a key-value pair for distribution. Depending upon the \ac{PMIx} implementation, the posted value may be locally cached in the client's \ac{PMIx} library until \refapi{PMIx_Commit} is called.

The provided \refarg{scope} determines the ability of other processes to access the posted data, as defined in \specrefstruct{pmix_scope_t}.
Specific implementations may support different scope values, but all implementations must support at least \refconst{PMIX_GLOBAL}.

The \refstruct{pmix_value_t} structure supports both string and binary values.
\ac{PMIx} implementations are required to support heterogeneous environments by properly converting binary values between host architectures, and will copy the provided \refarg{value} into internal memory prior to returning from \refapi{PMIx_Put}.

\adviceuserstart
Note that keys starting with a string of ``\code{pmix}'' must not be used in calls to \refapi{PMIx_Put}. Thus, applications should never use a defined ``PMIX'' attribute as the key in a call to \refapi{PMIx_Put}.
\adviceuserend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Scope of Put Data}
\declarestruct{pmix_scope_t}
\label{api:nres:scope}

\versionMarker{1.0}
The \refstruct{pmix_scope_t} structure is a \code{uint8_t} type that defines the availability of data passed to \refapi{PMIx_Put}.
The following constants can be used to set a variable of the type \refstruct{pmix_scope_t}. All definitions were introduced in version 1 of the standard unless otherwise marked.

Specific implementations may support different scope values, but all implementations must support at least \refconst{PMIX_GLOBAL}.
If a specified scope value is not supported, then the \refapi{PMIx_Put} call must return \refconst{PMIX_ERR_NOT_SUPPORTED}.

\begin{constantdesc}
%
\declareconstitem{PMIX_SCOPE_UNDEF}
Undefined scope.
%
\declareconstitem{PMIX_LOCAL}
The data is intended only for other application processes on the same node.
Data marked in this way will not be included in data packages sent to remote requesters - i.e., it is only available to processes on the local node.
%
\declareconstitem{PMIX_REMOTE}
The data is intended solely for applications processes on remote nodes.
Data marked in this way will not be shared with other processes on the same node - i.e., it is only available to  processes on remote nodes.
%
\declareconstitem{PMIX_GLOBAL}
The data is to be shared with all other requesting processes, regardless of location.
%
\versionMarker{2.0}
\declareconstitem{PMIX_INTERNAL}
The data is intended solely for this process and is not shared with other processes.
%
\end{constantdesc}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_Store_internal}}
\declareapi{PMIx_Store_internal}

%%%%
\summary

Store some data locally for retrieval by other areas of the process.

%%%%
\format

\copySignature{PMIx_Store_internal}{1.0}{
pmix_status_t \\
PMIx_Store_internal(const pmix_proc_t *proc, \\
\hspace*{20\sigspace}const pmix_key_t key, \\
\hspace*{20\sigspace}pmix_value_t *val);
}

\begin{arglist}
\argin{proc}{process reference (handle)}
\argin{key}{key to retrieve (string)}
\argin{val}{Value to store (handle)}
\end{arglist}

\returnstart
\begin{itemize}
\item \refconst{PMIX_ERR_BAD_PARAM} indicating a reserved key is provided in the \refarg{key} argument.
\end{itemize}
\returnend

%%%%
\descr

Store some data locally for retrieval by other areas of the process.
This is data that has only internal scope - it will never be posted externally. Typically used to cache data obtained by means outside of \ac{PMIx} so that it can be accessed by various areas of the process.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_Commit}}
\declareapi{PMIx_Commit}

%%%%
\summary

Post all previously \refapi{PMIx_Put} values for distribution.

%%%%
\format

\copySignature{PMIx_Commit}{1.0}{
pmix_status_t PMIx_Commit(void);
}

\returnsimple

%%%%
\descr

\ac{PMIx} implementations may choose to locally cache non-reserved keys prior to submitting them for distribution. Accordingly, \ac{PMIx} provides a second \ac{API} specifically to stage all previously posted data for distribution - e.g., by transmitting the entire collection of data posted by the process to a server in one operation. This is an asynchronous operation that will immediately return to the caller while the data is staged in the background.

\adviceuserstart
Users are advised to always include the call to \refapi{PMIx_Commit} in case the local implementation requires it. Note that posted data will not be circulated during \refapi{PMIx_Commit}. Availability of the data by other processes upon completion of \refapi{PMIx_Commit} therefore still relies upon the exchange mechanisms described at the beginning of this chapter.
\adviceuserend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Retrieval rules for non-reserved keys}
\label{chap:nrkeys:retrules}

Since non-reserved keys cannot, by definition, have been provided by the host
environment, their retrieval follows significantly different rules than those
defined for reserved keys (as detailed in Section \ref{chap:rkeys:retrules}).
\refapi{PMIx_Get} for a non-reserved key will obey the
following precedence search:

\begin{enumerate}
    \item If the \refattr{PMIX_GET_REFRESH_CACHE} attribute is given, then the
    request is first forwarded to the local \ac{PMIx} server which will then
    update the client's cache. Note that this may not, depending upon
    implementation details, result in any action.

    \item Check the local \ac{PMIx} client cache for the requested key - if not found and either the \refattr{PMIX_OPTIONAL} or \refattr{PMIX_GET_REFRESH_CACHE} attribute was given, the search will stop at this point and return the \refconst{PMIX_ERR_NOT_FOUND} status.

    \item Request the information from the local \ac{PMIx} server. The server
    will check its cache for the specified key within the appropriate scope as
    defined by the process that originally posted the key. If the value exists
    in a scope that contains the requesting process, then the value shall be
    returned. If the value exists, but in a scope that excludes the requesting
    process, then the server shall immediately return the
    \refconst{PMIX_ERR_EXISTS_OUTSIDE_SCOPE}.

    If the value still isn't found and the \refattr{PMIX_IMMEDIATE} attribute
    was given, then the library shall return the \refconst{PMIX_ERR_NOT_FOUND}
    error constant to the requester. Otherwise, the \ac{PMIx} server library
    will take one of the following actions:
    \begin{compactitemize}
        \item If the target process has a rank of \refconst{PMIX_RANK_UNDEF},
        then this indicates that the key being requested is globally unique
        and \emph{not} associated with a specific process. In this case, the
        server shall hold the request until either the data appears at the
        server or, if given, the \refattr{PMIX_TIMEOUT} is reached. In the
        latter case, the server will return the \refconst{PMIX_ERR_TIMEOUT}
        status. Note that the server may, depending on \ac{PMIx}
        implementation, never respond if the caller failed to specify a
        \refattr{PMIX_TIMEOUT} and the requested key fails to arrive at the
        server.

        \item If the target process is \emph{local} (i.e., attached to the
        same \ac{PMIx} server), then the server will hold the request until
        either the target process provides the data or, if given, the
        \refattr{PMIX_TIMEOUT} is reached. In the latter case, the server will
        return the \refconst{PMIX_ERR_TIMEOUT} status. Note that data which is
        posted via \refapi{PMIx_Put} but not staged with \refapi{PMIx_Commit}
        may, depending upon implementation, never appear at the server.

        \item If the target process is \emph{remote} (i.e., not attached to
        the same \ac{PMIx} server), the server will either:
        \begin{compactitemize}
            \item If the host has provided the
            \refapi{pmix_server_dmodex_req_fn_t} module function
            interface, then the server
            shall pass the request to its host for servicing. The host is
            responsible for determining the location of the target process and
            passing the request to the \ac{PMIx} server at that location.

            When the remote data request is received, the target \ac{PMIx}
            server will check its cache for the specified key. If the key is
            not present, the request shall be held until either the target
            process provides the data or, if given, the \refattr{PMIX_TIMEOUT}
            is reached. In the latter case, the server will return the
            \refconst{PMIX_ERR_TIMEOUT} status. The host shall convey the
            result back to the originating \ac{PMIx} server, which will reply
            to the requesting client with the result of the request when the
            host provides it.

            Note that the target server may, depending on \ac{PMIx}
            implementation, never respond if the caller failed to specify a
            \refattr{PMIX_TIMEOUT} and the target process fails to post the
            requested key.

            \item if the host does not support the
            \refapi{pmix_server_dmodex_req_fn_t} interface, then
            the server will immediately respond to the client with the
            \refconst{PMIX_ERR_NOT_FOUND} status
        \end{compactitemize}
    \end{compactitemize}
\end{enumerate}

\adviceimplstart
While there is no requirement that all \ac{PMIx} implementations follow the
client-server paradigm used in the above description, implementers are
required to provide behaviors consistent with the described search pattern.
\adviceimplend

\adviceuserstart
Users are advised to always specify the \refattr{PMIX_TIMEOUT} value when
retrieving non-reserved keys to avoid potential deadlocks should the specified
key not become available.
\adviceuserend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%