Command line issues

[awlauria]

Here's an overview of issues @gpaulsen found when testing ompi command line options.

NOTE: Box's should be checked as complete when fixes find their way to their release branches (openpmix v4.2/prrte v3.0).
As they come in feel free to edit in links to PR's in this comment.

• --display-devel-allocation, should map to --display devel-map
• --display-topo/--display topo, does nothing. Should be removed? --display-topo should map to --display topo
• --merge-stderr-to-stdout does not seem to work as expected. fixes in master: iof: Fix merging of stderr to stdout. openpmix/openpmix#2709 + schizo/ompi: Add translation for --merge-stderr-to-stdout. openpmix/prrte#1462
• --do-not-launch does not work as expected (processes are launched) Fixed in main: schizo/ompi: Fix --do-not-launch. openpmix/prrte#1459
• --show-progress, doesn't seem to do anything. Removal candidate?
• --stream-buffering - Can't tell if this is working as expected
  • PRRTE master: stream-buffering is an OMPI option, so make it an MCA openpmix/prrte#1489
  • PRRTE v3.0: Multiple commits openpmix/prrte#1491
  • Open MPI main: Make stream buffing an MCA option again #10746
  • Open MPI v5.0.x: Make stream buffing an MCA option again #10756
• --app, deprecated translation is not working
• -N doesn't appear in --help, possible removal candidate. Maps to --map-by ppr:1:node, so when used with --map-by this is confusing.
• --get-stack-traces seems to fail at larger scales via a hang or crash
• --stop-in-app shouldn't require an argument. Fixed in main: Fix minor things openpmix/prrte#1458
• --mca mca_base_env_list does not work on recent prrte updates
• --ppr - not well documented. Slot detection when used in conjuction with --host foo (without specifying slots)
  seems broken, for example:

      mpirun --ppr 1:core --host hostA ./hw.x                        
      --------------------------------------------------------------------------
      There are not enough slots available in the system to satisfy the 44
      slots that were requested by the application:

        ./hw.x

Removing the --host and it works, or specifying hostA:44.

• rankfile - some known issues already reported, needs verification
• --cpu-set/--cpu-list Works - somewhat. If you ask for more ranks than cpus
  you get:

  PRTE ERROR: Unable to map job in file rmaps_rr.c at line 184
   even with --oversubscribe
  

[rhc54]

--display-devel-allocation, should map to --display devel-map

Actually, it should map to --display alloc-devel. The simpler allocation output is --display alloc

--display-topo should map to --display topo

Correct - should work. I can check it

--merge-stderr-to-stdout does not seem to work as expected.

I'll check it - it was working last I checked. Maps to --output merge

--do-not-launch does not work as expected (processes are launched)

Hmmm...I've been using it extensively and it works fine, so probably just a translation issue. Maps to --map-by :donotlaunch, but may get moved to the new --controls xxx option as that is a better fit.

--show-progress, doesn't seem to do anything. Removal candidate?

The code is still there, but I'll have to look at why it isn't working

--stream-buffering - Can't tell if this is working as expected

Code is unchanged, so I assume it is still working. Perhaps @jjhursey can check (since he wrote it)?

--ppr - not well documented.

Yeah, that option is way old. I suspect it isn't being mapped correctly.

--cpu-set/--cpu-list Works - somewhat. If you ask for more ranks than cpus you get...

Actually, cpu-list seems to be working just fine if correctly translated. I suspect the problem is in translation. Remember, --cpu-list is now mapped to --map-by pe-list=a,b,c and there is the ordered qualifier to take into account. Don't know about cpu-set - I'm not sure PRRTE has an equivalent to that one other than pe-list. Perhaps the difference lies in the qualifier (cpu-set => nonordered and cpu-list => ordered)?

[awlauria]

--display-devel-allocation, should map to --display devel-map

Actually, it should map to --display alloc-devel. The simpler allocation output is --display alloc

It appears this option no longer exists:

--------------------------------------------------------------------------
The specified display directive is not recognized:

  Directive: alloc-devel
  Valid directives: allocation:map:bind:map-devel:topo

does it need to be re-implemented or should it just be removed?

[jjhursey]

We should also double check the $HOME/.prte/mca-params.conf and $HOME/.openmpi/mca-params.conf per the note below:

• comm_split_type HW_GUIDED fix MPI_UNDEFINED handling #10702 (comment)

[rhc54]

does it need to be re-implemented or should it just be removed?

I'll take a look - might just be an error.

[rhc54]

We should also double check the $HOME/.prte/mca-params.conf and $HOME/.openmpi/mca-params.conf

The PRRTE default param file is definitely working as I am using it and all get set as specified. I cannot speak to the OMPI one so that might need checking - it should get picked up by the pml/ompi component, but it might not be implemented there yet? Probably should first check that the OMPI personality is getting passed to PMIx (pretty sure that is happening, but worth checking).

[rhc54]

For the problem listed in #10702 (comment), what MCA param is he trying to set? I suspect it is either an incorrect param (due to it being PRRTE and not ORTE) or something that no longer exists as a param due to PRRTE treating things as job-specific instead (so maybe it needs to be restored as setting a default policy that is overridden by cmd line directives)

[rhc54]

For the problem listed in #10702 (comment), what MCA param is he trying to set? I suspect it is either an incorrect param (due to it being PRRTE and not ORTE) or something that no longer exists as a param due to PRRTE treating things as job-specific instead (so maybe it needs to be restored as setting a default policy that is overridden by cmd line directives)

I looked further at the description there and saw where he set the param - it looks okay. I suspect the problem is the same as was just reported to me offlist by the DDT folks - he isn't hitting a slot limit, but rather he is hitting the overload limit. Setting oversubscribe isn't turning off the check on CPUs. I'll have to fix that separately.

[rhc54]

Should be fixed in openpmix/prrte#1463

[rhc54]

It appears this option no longer exists:

---

The specified display directive is not recognized:

Directive: alloc-devel
Valid directives: allocation:map:bind:map-devel:topo

does it need to be re-implemented or should it just be removed?

Fixed in openpmix/prrte#1464

[rhc54]

I'm sensing some confusion over the user-facing options vs what PRRTE accepts, particularly with regard to the --map-by CLI. After looking into it a bit, I can understand why that would be happening as we overloaded --map-by with display options - probably a bad idea.

So please give me a bit to split all this out. Should be done by end of week, but I need some flexibility on the timing as my back has really flared up and so my time-at-keyboard is becoming hit/miss.