Feature/spack env #18

adrienbernede · 2020-01-18T01:01:52Z

Goal:

Here is an attempt to use Spack environments to enforce the configuration.

Benefits:

No need to patch Spack, no need to override Spack configuration.

Painful points:

Provided environments have to exhaustively specify Spack configuration (configure.yaml, compilers.yaml, package).
Forgetting :: at the beginning of config, compilers and packages sections would results in potential change of behavior.
Each providers have to be overridden as buildable: false to ensure perfect control it (ex: MPI). Note: Spack now allows to mark directly mpi as buildable false.
When installing a spec (uberenv.py --install), this spec gets added to the environment file...

white238 · 2020-01-28T18:11:08Z

uberenv.py

-        # copy in "defaults" config.yaml
-        config_yaml = os.path.abspath(pjoin(self.uberenv_path,"spack_configs","config.yaml"))
-        sexe("cp {} {}/".format(config_yaml, spack_etc_defaults_dir ), echo=True)
+        self.spack_cmd = "{} -e {}".format(self.spack_cmd,self.config_dir())


Is there a reason you went with -e instead of loading the environment? My concern is if you need to debug your build you would need to remember to add the -e to your spack commands.

Unfortunately there is a reason:
I tried first with loading the environment, but loading from a script is not reliable/persistent in my experience. It seems to be because it would load the environment in a subprocess and then kill it (the subprocess).
I asked @becker33. The other solution would be to use "spack python" as an interpreter, but it’s not compatible with the fact uberenv clones spack.

I think the "-e" method isn’t stupid though, the way I coded it I mean. Maybe we could make it more explicit what is the environment and the command to use to reproduce the behavior.

Like an output
"Uberenv: Now using spack environment scripts/uberenv/spack-config/toss3..."
"Ubernev: load environment with : "
"Uberenv: > spack env activate scripts/uberenv/spack-config/toss3..."
"Uberenv: or"
"Uberenv: > spack -e scripts/uberenv/spack-config/toss3... <action>"

As long as there is a reason, this is the whole purpose of uberenv is to encapsulate what we feel is the best way to use spack for us. If loading the environment doesn't work right now then we have to go with the -e option. Just make sure that it prints out the extra options so we can copy and paste the command. Maybe when it errors out re-print the command to the screen so it is easy to find.

I would try to get them to fix the env command while moving forward with this solution though.

What do you mean by "fix the env command"?

@cyrush I’m not sure I understand the question.

be persistent beyond the shell instance

Would you like the "activation" or the "configuration" to persist ?

yes, our current setup modifies spack so that the compiler and package settings persist.

Any further spack calls (driven by uberenv or manually) will respect those changes without having to know to invoke extra commands.

Now I understand. And it’s true that it is not the case with the present PR.

What is the exact usage currently, because you still need a way to call Spack:

always using the binary (uberenv-libs/spack/bin/spack) ?

loading spack shell setup : . ubernev-libs/spack/share/spack/setup.sh ?

@cyrush -- maybe we could generate a small script run_spack in the uberenv_libs directory that runs spack with the desired persistent parameters.

This is similar to what axom does in its config_build script to use the same version of cmake that was used to configure axom:
https://github.com/LLNL/axom/blob/61cd97c5aad8c96fd5d3defe16f5f72667ee1b99/config-build.py#L217-L230

A side benefit would be that you no longer need to run spack/bin/spack

After a bit more thought, I think it is crucial to export a run_spack script (and document this) to help users know how to setup their spack environment, as per my previous comment.

cyrush · 2020-01-28T21:34:29Z

usually bin/spack

I haven't used spack shell much b/c there was a tricky bug with an alias to a shell command in my environment that prevented me from using for years (until we figured it out :-), but old habits are hard to get rid of)

adrienbernede · 2020-01-28T22:19:04Z

Let’s summarize:

Need:

A local Spack instance configured for the project, automated, isolated, stable.

Root problem:

Spack needs to be configured for the project and the "platform" (what external product to use, where to place cache, what compiler, etc).
Spack standard way of configuration cannot be trusted because user scope is prevalent.

First way to solve it:

Patch Spack
Force configuration

This method was OK. I think the patch was a bit "violent" but it’s working. The root problem remains.

Another way with environments?

Now, with environments we can do something somewhat equivalent without patching Spack, but the configuration is not forced anymore, the goal of environments being to easily switch from one to another. We could imagine ways to force the use of a configuration, like a script to call the local Spack binary with the adequate environment, could even be encapsulated in Uberenv, but it gets a bit twisted IMO.

However, when using environment, we begin to envision a new possibility: using the same Spack instance to manage multiple projects. In fact, each project provides the desired configuration. Everything gets installed in the same place. And I think that’s what Spack is aiming at. Then, "you wanna work on conduit? Just load conduit environment." (currently we use anonymous environment, but we could name them).

If the goal is to get rid of patching of Spack and not to use a single Spack instance, maybe environment is not the perfect answer precisely because we loose the "enforced configuration".

@becker33 I’m just mentioning you so that you get an idea of what we need.

kennyweiss

Thanks @adrienbernede

I think this branch is almost there, but needs a bit more work before we can merge it.

We need to make sure the script won't crash if a user does not provide a spack config dir. E.g. if they are setting this up on their laptop and are happy with the default environment.
We should document the change in behavior in the sphinx docs, since this fix, while elegant will take the user from "it just works" (using the spack environment hotcopy), to "it works if I pass in the right parameter (and btw, there is no error message to let you know that this parameter is missing)"
We need to export a simple file, named something like run_spack which they can use to run uberenv's spack with the given environment. This script should have permissions for the user to execute it (e.g. chmod u+x).

Some other notes about consistency:

For consistency with the rest of uberenv, please add explicit indices to the String.format lines
I.e. prefer "{0} find -p {1}".format(...) to "{} find -p {}".format(...)
Please also move the entire command into the interpolated string.
I.e. prefer something like sexe("{0} find -p {1}".format(spack_cmd, pkg_name)
to something like sexe("{0} find -p ".format(spack_cmd) + pkg_name)

uberenv.py

kennyweiss · 2020-04-26T18:49:23Z

uberenv.py

@@ -386,82 +389,44 @@ def config_dir(self):
        return spack_config_dir


When the spack-config option is not provided, this will return None, right?

kennyweiss · 2020-04-26T18:51:03Z

uberenv.py

-        # copy in "defaults" config.yaml
-        config_yaml = os.path.abspath(pjoin(self.uberenv_path,"spack_configs","config.yaml"))
-        sexe("cp {} {}/".format(config_yaml, spack_etc_defaults_dir ), echo=True)
+        self.spack_cmd = "{} -e {}".format(self.spack_cmd,self.config_dir())


What happens if there is no spack config dir? Will this error out?
Perhaps we need to only add the -e <config-dir> option when it is available?

@kennyweiss an environment is treated the same way config dir were.
It means that Uberenv will try to retrieve the environment corresponding to the current OS/system type.
The behavior when no environment is found is then consistent with what it was before: it throws and error.

This means projects must add a new environment (spack.yaml file) in each config dir.
Adding a basic one is fairly straightforward though, and consists in:

spack: include: - ../config.yaml - compilers.yaml - packages.yaml specs: [] view: false concretization: separately

and in compilers.yaml and packages.yaml, changing respectively:

compilers: -> compilers::

packages:-> packages::

In order to override and scoping (can be done for config too, but I think we will need to discuss the config case separately: overriding scopes in config is not safe).

kennyweiss · 2020-04-26T18:52:55Z

uberenv.py

-        # copy in "defaults" config.yaml
-        config_yaml = os.path.abspath(pjoin(self.uberenv_path,"spack_configs","config.yaml"))
-        sexe("cp {} {}/".format(config_yaml, spack_etc_defaults_dir ), echo=True)
+        self.spack_cmd = "{} -e {}".format(self.spack_cmd,self.config_dir())


After a bit more thought, I think it is crucial to export a run_spack script (and document this) to help users know how to setup their spack environment, as per my previous comment.

adrienbernede · 2020-09-02T23:15:19Z

@kennyweiss I added a very simple script generation to encapsulate the spack command.

This branch is working fine on Umpire CI.

adrienbernede · 2020-09-02T23:17:09Z

I should have merged instead of force pushing the rebased branch. Old habits.

adrienbernede · 2021-04-02T18:10:26Z

@cyrush @kennyweiss @white238 what do you guys want for this branch?
It seems you are interested in using environments, yet this is stale, and now out-dated.
I will only update if I can be confident it will be merged. (update will come with some improvements).

Please express your requirements.

cyrush · 2021-04-02T18:53:59Z

Since I have been using envs a bit more, I have a few general suggestions:

env support should be supported along with existing paths.

The fact that dev build support replaced the fake package support delayed me being able to upgrade for quite some time.

From this experience -- I think its really important that env support can be adopted and vetted as a parallel or composable option. I also think this will help avoid stalled PRs like we have here.
The env should be created inside the prefix dir.

If folks want to control where the assets are built (instead in default spack dir) spack envs and configs already provide a path to do that - nothing special is needed for uberenv. But keeping things contained to the prefix dir is a priority.
We need to figure out the implications for python packages.

If I can use spack env with the fake package approach, most of my concerns about envs having to be active go away. The final hostconfig result can be easily disconnected from spack. This may be the case for dev build as well, but not sure.

The only thing I am not sure about is how activating python packages is impacted.

white238 reviewed Jan 28, 2020

View reviewed changes

tgamblin mentioned this pull request Jan 29, 2020

feature: sharing a spack instance spack/spack#11871

Open

adrienbernede force-pushed the feature/spack-env branch from 471b6bf to 657ea71 Compare March 2, 2020 18:44

kennyweiss mentioned this pull request Mar 11, 2020

Fixes, Improvements and CI #21

Merged

kennyweiss requested changes Apr 26, 2020

View reviewed changes

adrienbernede added 4 commits September 2, 2020 11:12

WIP: moving to spack environments

0a53c59

spack: dropping env creation for anonymous env

ee47f36

Replace env loading with spack -e command

2331521

Generate a proxy script for spack with last environment used

5b804cd

adrienbernede force-pushed the feature/spack-env branch from 657ea71 to 5b804cd Compare September 2, 2020 23:13

adrienbernede marked this pull request as ready for review September 2, 2020 23:21

adrienbernede added 2 commits September 2, 2020 16:27

Cleaning and applying review suggestions

487c247

Fix and format

3cadf65

adrienbernede mentioned this pull request Nov 26, 2020

allow mirror url to be set in project.json file #32

Open

Merge branch 'master' into feature/spack-env

3dbfcbb

kennyweiss mentioned this pull request Apr 20, 2021

Add documentation about using uberenv's copy of spack #61

Open

adrienbernede mentioned this pull request Apr 23, 2021

Feature/environments #63

Closed

4 tasks

white238 changed the base branch from master to main October 19, 2022 21:36

chapman39 closed this Mar 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/spack env #18

Feature/spack env #18

adrienbernede commented Jan 18, 2020 •

edited

Loading

white238 Jan 28, 2020

adrienbernede Jan 28, 2020

adrienbernede Jan 28, 2020

white238 Jan 28, 2020

adrienbernede Jan 28, 2020

adrienbernede Jan 28, 2020

cyrush Jan 28, 2020

adrienbernede Jan 28, 2020

kennyweiss Jan 30, 2020

kennyweiss Apr 26, 2020

cyrush commented Jan 28, 2020

adrienbernede commented Jan 28, 2020 •

edited

Loading

kennyweiss left a comment

kennyweiss Apr 26, 2020 •

edited

Loading

kennyweiss Apr 26, 2020

adrienbernede Sep 2, 2020

kennyweiss Apr 26, 2020

adrienbernede commented Sep 2, 2020 •

edited

Loading

adrienbernede commented Sep 2, 2020

adrienbernede commented Apr 2, 2021

cyrush commented Apr 2, 2021

		@@ -386,82 +389,44 @@ def config_dir(self):
		return spack_config_dir

Feature/spack env #18

Feature/spack env #18

Conversation

adrienbernede commented Jan 18, 2020 • edited Loading

Goal:

Benefits:

Painful points:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cyrush commented Jan 28, 2020

adrienbernede commented Jan 28, 2020 • edited Loading

Need:

Root problem:

First way to solve it:

Another way with environments?

kennyweiss left a comment

Choose a reason for hiding this comment

kennyweiss Apr 26, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrienbernede commented Sep 2, 2020 • edited Loading

adrienbernede commented Sep 2, 2020

adrienbernede commented Apr 2, 2021

cyrush commented Apr 2, 2021

adrienbernede commented Jan 18, 2020 •

edited

Loading

adrienbernede commented Jan 28, 2020 •

edited

Loading

kennyweiss Apr 26, 2020 •

edited

Loading

adrienbernede commented Sep 2, 2020 •

edited

Loading