v4.2 GDS issue with singleton spawn

[jjhursey]

Background information

What version of the PMIx Reference Library are you using?

• OpenPMIx v4.2 branch at 8cb6f58
• PRRTE master branch at openpmix/prrte@8d735f5 and v3.0 branch at openpmix/prrte@b29abde
• Open MPI main branch at open-mpi/ompi@5575c86 (with both Fix Singletons and Singleton Spawn open-mpi/ompi#10688 and Fix the dpm to fwd IO to the spawning parent open-mpi/ompi#10695 cherry-picked into the branch since they have not been merged in yet)

Testing note on this comment

Describe how PMIx was installed

Built with Open MPI main and manually adjusted the submodule pointers.

Please describe the system on which you are running

• Operating system/version: RHEL 8.4
• Computer hardware: ppc64le
• Network type: Single node shared memory

---

Details of the problem

Using PRRTE master and OpenPMIx master we have been able to get singleton MPI_Comm_spawn working. However, if we move to OpenPMix v4.2 then it fails. See open-mpi/ompi#10688

The MPI tests I'm using are located here

shell$ mpirun --np 1 ./simple_spawn ./simple_spawn
Spawning './simple_spawn' ... OK
shell$ ./simple_spawn ./simple_spawn
[f5n18:3788975] PMIX ERROR: PROC-ENTRY-NOT-FOUND in file server/pmix_server.c at line 3588
[f5n18:3788914] pml_ucx.c:191  Error: Failed to receive UCX worker address: Take next option (-46)
[f5n18:3788914] OPAL ERROR: Error in file dpm/dpm.c at line 480

I noticed that if I force the hash GDS component that it works correctly

shell$ export PMIX_MCA_gds=hash
shell$ ./simple_spawn ./simple_spawn
Spawning './simple_spawn' ... OK

So this seems to only impact the singleton spawn case, and is related to the GDS component (verbose output indicates that it is using ds21)

[rhc54]

Is it the singleton or the child that is complaining?

[rhc54]

Oh, and what key are they looking for?

[rhc54]

Just off the top of my head: I'd guess that the error comes from the child and that it is looking for a modex key. The problem looks to be in the dpm code - once you have spawned the prte server, you have to PMIx_Commit the singleton's modex data to it so the children can look it up.

[jjhursey]

It looks like modex information (if I force OMPI_MCA_pml=ob1 it ends up failing in the modex in the btls instead of UCX). From what I can tell it is the child that is complaining (in addition to the server).

[jjhursey]

It looks like we are calling PMIx_Commit() just after the server starts:

• https://github.com/open-mpi/ompi/blob/main/ompi/dpm/dpm.c#L2138-L2139

[rhc54]

Okay, here's the easiest solution. Once you detect that you are a singleton, push PMIX_MCA_gds=hash into your environment. This will make no difference to the singleton as that is all it can use, but it will cause prte to restrict itself to the hash, which will force all child procs to use their hash components.

See if that works.

[jjhursey]

Yeah, that works. I'll post a PR to OMPI to make that change.

It bugged me that it worked with OpenPMIx master but not the v4.2 branch. Is that because dstore is not in the main branch anymore so I'm effectively just using hash there?

[rhc54]

Correct. The problem with dstore is that it is way behind in how to handle anything other than the original job info. The hash component is far more versatile. Once we get to PMIx v5, dstore will go away and we will have a shmem hash component instead - will solve many problems.