-
Notifications
You must be signed in to change notification settings - Fork 202
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add support for reverse dependencies of installed modules (WIP) #2783
base: develop
Are you sure you want to change the base?
add support for reverse dependencies of installed modules (WIP) #2783
Conversation
easybuild/tools/docs.py
Outdated
""" | ||
reverse_dependencies = {} | ||
|
||
module_avail_cmd = "module -t avail 2>&1 | grep -v ':$'" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How would this work with a hierarchical scheme (or any scheme that involved extending the module path)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That shouldn't be a problem when using modules_tool().available()
, which spits out a list of full module names in a hierarchy (i.e. things like Core/GCC/6.4.0
rather than just GCC/6.4.0
), so you'll see all modules regardless of where they are in the hierarchy...
easybuild/tools/docs.py
Outdated
available_modules, _ = run_cmd(module_avail_cmd) | ||
|
||
for module in available_modules.splitlines(): | ||
dependencies, _ = run_cmd(module_purge_load_list_cmd % (module,module)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@migueldiascosta probably a good idea to use trace=False
here, otherwise you get very amusing output when using eb --trace
(which is default for me)
easybuild/tools/docs.py
Outdated
reverse_dependencies = {} | ||
|
||
module_avail_cmd = "module -t avail 2>&1 | grep -v ':$'" | ||
module_purge_load_list_cmd = "module purge; module load %s; module -t list 2>&1 | grep -v %s" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using purge
is troublesome on some systems, and fairly expensive with Lmod...
How about just running this in a subshell, since we only care about the output, not the changes to the environment?
Or use ml show
(which only gives direct deps though), that would also make it significantly faster...
I was thinking about this recently but with a very different approach, you can generate a full dependency graph for your installation tree with
but it only works with certain caveats
|
I tried it for a software in a stage of JURECA, and it (currently) doesn't work, graphdot throws errors like
|
Sorry for bombing your PR but with a small tweak to tools.py
my method succeeds and I get
|
@ocaisa One of the design principles here, which we also briefly discussed during the user meeting, is that the existing module files present the current "state" of things, and that's really all the matters from a users perspective. Your approach is interesting, but not with the feature we have in mind that will be implemented on top of this (i.e. |
But it would be easy to prune out non-installed easyconfigs (of which there would be none unless you manually removed the module files, note I'm only talking about easyconfigs in the easybuild directory of installed software here) since you have this info from the functionality of |
@ocaisa bomb away :), this WIP PR was created for precisely for discussing the issue My original approach a while back was also based on easyconfigs in the robot path, and I still think there's a use case for that, but since then I came to agree with @boegel that for uninstall purposes the installed modules are the source of truth from the perspective of the users. For instance, there may be modules that were not created by easybuild but depend on modules created by easybuild, and we should know about those before removing a module created by easybuild, or even modules may have been modified after being created by easybuild... Maybe there's a case for having both "easyconfig reverse dependencies", e.g. for removing easyconfigs from a repository, and "module reverse dependencies", e.g. for removing installed software? As to this particular implementation of "module reverse dependencies", it's very slow, I'll look into using module show instead of module load + list to speed it up |
…m ModuleGenerator and (modified, to optionally use MODULEPATH instead of calling module show, if using full module names) modulefile_path from ModuleTool
@boegel by caching dependencies and, crucially, using |
if mod_name not in alldeps: | ||
alldeps[mod_name] = [] | ||
|
||
mod_filepath = modtool.modulefile_path(mod_name, full_module_names=full_module_names) | ||
modtxt = read_file(mod_filepath) | ||
loadregex = module_load_regex(mod_filepath) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@migueldiascosta What about depends_on
and prereq
?
Travis test report: 1/8 runs failed - see https://travis-ci.org/easybuilders/easybuild-framework/builds/502458945 Only showing partial log for 1st failed test suite run 2867.7;
*bleep, bloop, I'm just a bot (boegelbot v20180813.01)*Please talk to my owner @boegel if you notice you me acting stupid),or submit a pull request to https://github.com/boegel/boegelbot fix the problem. |
This currently works for a flat module naming scheme, but not for one that extends the module path. For HierarchicalMNS, it is possible to recurse through the hierarchy levels, keeping track of extended module paths, short and full module names for each dependency, but it's messy and even then I'm not sure it would work for other naming schemes... |
No description provided.