Release updating

[gq1]

I follow GitHub for your latest release: https://github.com/singularityhub/singularity-hpc/releases

It is [0.0.37] now, you manually added this tag when I requested last time?

But you have the newer version in PyPi: https://pypi.org/project/singularity-hpc/

Are you planning to keep them consistent automatically?

Thanks.

[vsoch]

I normally just release to pypi. If you want to add a release workflow to trigger here as well I’d be happy to review it!

[gq1]

Thanks. I will check how to do it and change my updating procedure as well.

Just saw an waring message:
WARNING: While bind mounting '/lustre:/lustre': destination is already in the mount point list
WARNING: While bind mounting '/nfs:/nfs': destination is already in the mount point list

When I have bindpaths: /lustre,/nfs

Any idea this could be?

[vsoch]

If your system (where your admins installed singularity) already have these as default mounts (most will have some defaults) then adding them again via shpc would be redundant and you'd see that message.

[gq1]

I have checked with the system bind path, these two are not included.

If I don't add them to shpc, the software installed by shpc will not be able to read from /nfs or /lustre.

Another thing I want to mention to you, I said shpc module loading was very slow last time. It is OK now after I separated the module base and container base.

[vsoch]

I have checked with the system bind path, these two are not included.

Okay so let's try a few things! First remove the bindpaths from your settings.yml and confirm that nothing is bound.
Then my thinking is that Singularity already honors the envar SINGULARITY_BINDPATH so the issue comes from us exporting that AND explicitly added them to the command. And in this case, we just need to decide which is best to use.

Another thing I want to mention to you, I said shpc module loading was very slow last time. It is OK now after I separated the module base and container base.

This is good feedback - I'm thinking perhaps we should have the default be that modules and containers are separate. What say you @marcodelapierre ?

[vsoch]

@gq1 I will have a PR in tonight for you to test my bind mount theory! And After we hear from @marcodelapierre (he is in Australia so the opposite timezone) I can also do a PR to make the containers / modules separate by default.

[vsoch]

here you go! Please test @gq1 ! #489

[vsoch]

@gq1 @marcodelapierre here is a suggestion as I mentioned above to make the default containers directory alongside modules! #490. For singularity users I think this will make loading of modules much faster. For Podman/Docker it doesn't matter since the blobs live elsewhere!

[vsoch]

And here is a release workflow that should allow me to release on GitHub and then upload to pypi. I am also adding a wheel (not sure if that's a good idea for this package but maybe worth a shot?) I suspect most people install via clone.

#491

[gq1]

@gq1 I will have a PR in tonight for you to test my bind mount theory! And After we hear from @marcodelapierre (he is in Australia so the opposite timezone) I can also do a PR to make the containers / modules separate by default.

Thanks. I have tested this branch and the warning message disappeared now.

[vsoch]

Going to re-open - @gq1 the release work is done and the bug is fixed, and the last thing this issue needs to address is the default container path, see #490. Also ping @marcodelapierre for feedback on this (I think it's an easy yes but want to double check with both of you!)

[marcodelapierre]

Hi @vsoch , @gq1, sorry for the silence, another full-on work week!

So:

1. containers and modules distinct by default : I second that, it's the first SHPC configuration I edit when I deploy it!

2. github and pypi release versions in sync: I think this can be quite important for reproducible deployments. Even if you pip install shpc, you will still need to clone the registry from github, and so it's good to always have the option to keep versions in sync. This is only one of the possible ways people may choose to enforce reproducibility control for their software deployments, and one which I think is good to provide.

3. bind path behaviour
   Yes, that looked to me like a duplicate bind mount, too. It does not cause issues, just a warning.
   In general, I would think it's up to the specific system setup, and one can make informed choices by knowing that SHPC depends on Singularity but not vice versa.
   Then, the Singularity configuration should probably be decided first, considering it will impact everyone using containers, not only SHPC users.
   Then, moving to SHPC, I would avoid specifying in the bindpath setting any path that is already taken care of by the Singularity configuration.

I hope it makes sense, I have noticed recently that my Friday afternoon Github comments are often a bit nebulous ...and I had no beers yet, I swear 😄

[vsoch]

okay all these things are fixed, the last release was a success: https://github.com/singularityhub/singularity-hpc/runs/5159558661?check_suite_focus=true thanks @gq1 and @marcodelapierre !