v0.21.0
is a major feature release.
-
Better error messages with condition chaining
In v0.18, we added better error messages that could tell you what problem happened, but they couldn't tell you why it happened.
0.21
adds condition chaining to the solver, and Spack can now trace back through the conditions that led to an error and build a tree of causes potential causes and where they came from. For example:$ spack solve hdf5 ^cmake@3.0.1 ==> Error: concretization failed for the following reasons: 1. Cannot satisfy 'cmake@3.0.1' 2. Cannot satisfy 'cmake@3.0.1' required because hdf5 ^cmake@3.0.1 requested from CLI 3. Cannot satisfy 'cmake@3.18:' and 'cmake@3.0.1 required because hdf5 ^cmake@3.0.1 requested from CLI required because hdf5 depends on cmake@3.18: when @1.13: required because hdf5 ^cmake@3.0.1 requested from CLI 4. Cannot satisfy 'cmake@3.12:' and 'cmake@3.0.1 required because hdf5 depends on cmake@3.12: required because hdf5 ^cmake@3.0.1 requested from CLI required because hdf5 ^cmake@3.0.1 requested from CLI
More details in #40173.
-
OCI build caches
You can now use an arbitrary OCI registry as a build cache:
$ spack mirror add my_registry oci://user/image # Dockerhub $ spack mirror add my_registry oci://ghcr.io/haampie/spack-test # GHCR $ spack mirror set --push --oci-username ... --oci-password ... my_registry # set login creds $ spack buildcache push my_registry [specs...]
And you can optionally add a base image to get runnable images:
$ spack buildcache push --base-image ubuntu:23.04 my_registry python Pushed ... as [image]:python-3.11.2-65txfcpqbmpawclvtasuog4yzmxwaoia.spack $ docker run --rm -it [image]:python-3.11.2-65txfcpqbmpawclvtasuog4yzmxwaoia.spack
This creates a container image from the Spack installations on the host system, without the need to run
spack install
from aDockerfile
orsif
file. It also addresses the inconvenience of losing binaries of dependencies whenRUN spack install
fails insidedocker build
.Further, the container image layers and build cache tarballs are the same files. This means that
spack install
anddocker pull
use the exact same underlying binaries. If you previously usedspack install
inside ofdocker build
, this feature helps you save storage by a factor two.More details in #38358.
-
Multiple versions of build dependencies
Increasingly, complex package builds require multiple versions of some build dependencies. For example, Python packages frequently require very specific versions of
setuptools
,cython
, and sometimes different physics packages require different versions of Python to build. The concretizer enforced that every solve was unified, i.e., that there only be one version of every package. The concretizer now supports "duplicate" nodes for build dependencies, but enforces unification through transitive link and run dependencies. This will allow it to better resolve complex dependency graphs in ecosystems like Python, and it also gets us very close to modeling compilers as proper dependencies.This change required a major overhaul of the concretizer, as well as a number of performance optimizations. See #38447, #39621.
-
Cherry-picking virtual dependencies
You can now select only a subset of virtual dependencies from a spec that may provide more. For example, if you want
mpich
to be yourmpi
provider, you can be explicit by writing:hdf5 ^[virtuals=mpi] mpich
Or, if you want to use, e.g.,
intel-parallel-studio
forblas
along with an externallapack
likeopenblas
, you could write:strumpack ^[virtuals=mpi] intel-parallel-studio+mkl ^[virtuals=lapack] openblas
The
virtuals=mpi
is an edge attribute, and dependency edges in Spack graphs now track which virtuals they satisfied. More details in #17229 and #35322.Note for packaging: in Spack 0.21
spec.satisfies("^virtual")
is true if and only if the package specifiesdepends_on("virtual")
. This is different from Spack 0.20, where depending on a provider implied depending on the virtual provided. See #41002 for an example where^mkl
was being used to test for severalmkl
providers in a package that did not depend onmkl
. -
License directive
Spack packages can now have license metadata, with the new
license()
directive:license("Apache-2.0")
Licenses use SPDX identifiers, and you can use SPDX expressions to combine them:
license("Apache-2.0 OR MIT")
Like other directives in Spack, it's conditional, so you can handle complex cases like Spack itself:
license("LGPL-2.1", when="@:0.11") license("Apache-2.0 OR MIT", when="@0.12:")
More details in #39346, #40598.
-
spack deconcretize
commandWe are getting close to having a
spack update
command for environments, but we're not quite there yet. This is the next best thing.spack deconcretize
gives you control over what you want to update in an already concrete environment. If you have an environment built with, say,meson
, and you want to update yourmeson
version, you can run:spack deconcretize meson
and have everything that depends on
meson
rebuilt the next time you runspack concretize
. In a future Spack version, we'll handle all of this in a single command, but for now you can use this to drop bits of your lockfile and resolve your dependencies again. More in #38803. -
UI Improvements
The venerable
spack info
command was looking shabby compared to the rest of Spack's UI, so we reworked it to have a bit more flair.spack info
now makes much better use of terminal space and shows variants, their values, and their descriptions much more clearly. Conditional variants are grouped separately so you can more easily understand how packages are structured. More in #40998.spack checksum
now allows you to filter versions from your editor, or by version range. It also notifies you about potential download URL changes. See #40403. -
Environments can include definitions
Spack did not previously support using
include:
with The definitions section of an environment, but now it does. You can use this to curate lists of specs and more easily reuse them across environments. See #33960. -
Aliases
You can now add aliases to Spack commands in
config.yaml
, e.g. this might enshrine your favorite args tospack find
asspack f
:config: aliases: f: find -lv
See #17229.
-
Improved autoloading of modules
Spack 0.20 was the first release to enable autoloading of direct dependencies in module files.
The downside of this was that
module avail
andmodule load
tab completion would show users too many modules to choose from, and many users disabled generating modules for dependencies throughexclude_implicits: true
. Further, it was necessary to keep hashes in module names to avoid file name clashes.In this release, you can start using
hide_implicits: true
instead, which exposes only explicitly installed packages to the user, while still autoloading dependencies. On top of that, you can safely usehash_length: 0
, as this config now only applies to the modules exposed to the user -- you don't have to worry about file name clashes for hidden dependencies.
Note: for tcl
this feature requires Modules 4.7 or higher
-
Updated container labeling
Nightly Docker images from the
develop
branch will now be tagged as:develop
and:nightly
. The:latest
tag is no longer associated with:develop
, but with the latest stable release. Releases will be tagged with:{major}
,:{major}.{minor}
and:{major}.{minor}.{patch}
.ubuntu:18.04
has also been removed from the list of generated Docker images, as it is no longer supported. See #40593.
spack env activate
without arguments now loads adefault
environment that you do not have to create (#40756).spack find -H
/--hashes
: a new shortcut for pipingspack find
output to other commands (#38663)- Add
spack checksum --verify
, fix--add
(#38458) - New
default_args
context manager factors out common args for directives (#39964) spack compiler find --[no]-mixed-toolchain
lets you easily mixclang
andgfortran
on Linux (#40902)
spack external find
execution is now much faster (#39843)spack location -i
now much faster on success (#40898)- Drop redundant rpaths post install (#38976)
- ASP-based solver: avoid cycles in clingo using hidden directive (#40720)
- Fix multiple quadratic complexity issues in environments (#38771)
- archspec: update to v0.2.2, support for Sapphire Rapids, Power10, Neoverse V2 (#40917)
- Propagate variants across nodes that don't have that variant (#38512)
- Implement fish completion (#29549)
- Can now distinguish between source/binary mirror; don't ping mirror.spack.io as much (#34523)
- Improve status reporting on install (add [n/total] display) (#37903)
This release has the best Windows support of any Spack release yet, with numerous improvements and much larger swaths of tests passing:
- MSVC and SDK improvements (#37711, #37930, #38500, #39823, #39180)
- Windows external finding: update default paths; treat .bat as executable on Windows (#39850)
- Windows decompression: fix removal of intermediate file (#38958)
- Windows: executable/path handling (#37762)
- Windows build systems: use ninja and enable tests (#33589)
- Windows testing (#36970, #36972, #36973, #36840, #36977, #36792, #36834, #34696, #36971)
- Windows PowerShell support (#39118, #37951)
- Windows symlinking and libraries (#39933, #38599, #34701, #38578, #34701)
- User-specified flags take precedence over others in Spack compiler wrappers (#37376)
- Improve setup of build, run, and test environments (#35737, #40916)
make
is no longer a required system dependency of Spack (#40380)- Support Python 3.12 (#40404, #40155, #40153)
- docs: Replace package list with packages.spack.io (#40251)
- Drop Python 2 constructs in Spack (#38720, #38718, #38703)
- e4s arm stack: duplicate and target neoverse v1 (#40369)
- Add macOS ML CI stacks (#36586)
- E4S Cray CI Stack (#37837)
- e4s cray: expand spec list (#38947)
- e4s cray sles ci: expand spec list (#39081)
- ASP: targets, compilers and providers soft-preferences are only global (#31261)
- Parser: fix ambiguity with whitespace in version ranges (#40344)
- Module file generation is disabled by default; you'll need to enable it to use it (#37258)
- Remove deprecated "extra_instructions" option for containers (#40365)
- Stand-alone test feature deprecation postponed to v0.22 (#40600)
- buildcache push: make
--allow-root
the default and deprecate the option (#38878)
- Bugfix: propagation of multivalued variants (#39833)
- Allow
/
in git versions (#39398) - Fetch & patch: actually acquire stage lock, and many more issues (#38903)
- Environment/depfile: better escaping of targets with Git versions (#37560)
- Prevent "spack external find" to error out on wrong permissions (#38755)
- lmod: allow core compiler to be specified with a version range (#37789)
- 7,469 total packages, 303 new since
v0.20.0
- 150 new Python packages
- 34 new R packages
- 353 people contributed to this release
- 336 committers to packages
- 65 committers to core
- Fix a bug where
spack mirror set-url
would drop configured connection info (reverts #34210) - Fix a minor issue with package hash computation for Python 3.12 (#40328)
Spack now supports Python 3.12 (#40155)
- Improve escaping in Tcl module files (#38375)
- Make repo cache work on repositories with zero mtime (#39214)
- Ignore errors for newer, incompatible buildcache version (#40279)
- Print an error when git is required, but missing (#40254)
- Ensure missing build dependencies get installed when using
spack install --overwrite
(#40252) - Fix an issue where Spack freezes when the build process unexpectedly exits (#39015)
- Fix a bug where installation failures cause an unrelated
NameError
to be thrown (#39017) - Fix an issue where Spack package versions would be incorrectly derived from git tags (#39414)
- Fix a bug triggered when file locking fails internally (#39188)
- Prevent "spack external find" to error out when a directory cannot be accessed (#38755)
- Fix multiple performance regressions in environments (#38771)
- Add more ignored modules to
pyproject.toml
formypy
(#38769)
- Spec removed from an environment where not actually removed if
--force
was not given (#37877) - Speed-up module file generation (#37739)
- Hotfix for a few recipes that treat CMake as a link dependency (#35816)
- Fix re-running stand-alone test a second time, which was getting a trailing spurious failure (#37840)
- Fixed reading JSON manifest on Cray, reporting non-concrete specs (#37909)
- Fixed a few bugs when generating Dockerfiles from Spack (#37766,#37769)
- Fixed a few long-standing bugs when generating module files (#36678,#38347,#38465,#38455)
- Fixed issues with building Python extensions using an external Python (#38186)
- Fixed compiler removal from command line (#38057)
- Show external status as [e] (#33792)
- Backported
archspec
fixes (#37793) - Improved a few error messages (#37791)
v0.20.0
is a major feature release.
-
requires()
directive and enhanced package requirementsWe've added some more enhancements to requirements in Spack (#36286).
There is a new
requires()
directive for packages.requires()
is the opposite ofconflicts()
. You can use it to impose constraints on this package when certain conditions are met:requires( "%apple-clang", when="platform=darwin", msg="This package builds only with clang on macOS" )
More on this in the docs.
You can also now add a
when:
clause torequires:
in yourpackages.yaml
configuration or in an environment:packages: openmpi: require: - any_of: ["%gcc"] when: "@:4.1.4" message: "Only OpenMPI 4.1.5 and up can build with fancy compilers"
More details can be found here
-
Exact versions
Spack did not previously have a way to distinguish a version if it was a prefix of some other version. For example,
@3.2
would match3.2
,3.2.1
,3.2.2
, etc. You can now match exactly3.2
with@=3.2
. This is useful, for example, if you need to patch only the3.2
version of a package. The new syntax is described in the docs.Generally, when writing packages, you should prefer to use ranges like
@3.2
over the specific versions, as this allows the concretizer more leeway when selecting versions of dependencies. More details and recommendations are in the packaging guide.See #36273 for full details on the version refactor.
-
New testing interface
Writing package tests is now much simpler with a new test interface.
Writing a test is now as easy as adding a method that starts with
test_
:class MyPackage(Package): ... def test_always_fails(self): """use assert to always fail""" assert False def test_example(self): """run installed example""" example = which(self.prefix.bin.example) example()
You can use Python's native
assert
statement to implement your checks -- no more need to fiddle withrun_test
or other test framework methods. Spack will introspect the class and runtest_*
methods when you runspack test
, -
More stable concretization
-
Now,
spack concretize
will only concretize the new portions of the environment and will not change existing parts of an environment unless you specify--force
. This has always been true forunify:false
, but not forunify:true
andunify:when_possible
environments. Now it is true for all of them (#37438, #37681). -
The concretizer has a new
--reuse-deps
argument that only reuses dependencies. That is, it will always treat the roots of your environment as it would with--fresh
. This allows you to upgrade just the roots of your environment while keeping everything else stable (#30990).
-
-
Weekly develop snapshot releases
Since last year, we have maintained a buildcache of
develop
at https://binaries.spack.io/develop, but the cache can grow to contain so many builds as to be unwieldy. When we get a stabledevelop
build, we snapshot the release and add a corresponding tag the Spack repository. So, you can use a stack from a specific day. There are now tags in the spack repository like:develop-2023-05-14
develop-2023-05-18
that correspond to build caches like:
We plan to store these snapshot releases weekly.
-
Specs in buildcaches can be referenced by hash.
- Previously, you could run
spack buildcache list
and see the hashes in buildcaches, but referring to them by hash would fail. - You can now run commands like
spack spec
andspack install
and refer to buildcache hashes directly, e.g.spack install /abc123
(#35042)
- Previously, you could run
-
New package and buildcache index websites
Our public websites for searching packages have been completely revamped and updated. You can check them out here:
- Package Index: https://packages.spack.io
- Buildcache Index: https://cache.spack.io
Both are searchable and more interactive than before. Currently major releases are shown; UI for browsing
develop
snapshots is coming soon. -
Default CMake and Meson build types are now Release
Spack has historically defaulted to building with optimization and debugging, but packages like
llvm
can be enormous with debug turned on. Our default build type for all Spack packages is nowRelease
(#36679, #37436). This has a number of benefits:- much smaller binaries;
- higher default optimization level; and
- defining
NDEBUG
disables assertions, which may lead to further speedups.
You can still get the old behavior back through requirements and package preferences.
spack checksum
can automatically add new versions to package (#24532)- new command:
spack pkg grep
to easily search package files (#34388) - New
maintainers
directive (#35083) - Add
spack buildcache push
(alias tobuildcache create
) (#34861) - Allow using
-j
to control the parallelism of concretization (#37608) - Add
--exclude
option to 'spack external find' (#35013)
- editing: add higher-precedence
SPACK_EDITOR
environment variable - Many YAML formatting improvements from updating
ruamel.yaml
to the latest version supporting Python 3.6. (#31091, #24885, #37008). - Requirements and preferences should not define (non-git) versions (#37687, #37747)
- Environments now store spack version/commit in
spack.lock
(#32801) - User can specify the name of the
packages
subdirectory in repositories (#36643) - Add container images supporting RHEL alternatives (#36713)
- make version(...) kwargs explicit (#36998)
- buildcache create: reproducible tarballs (#35623)
- Bootstrap most of Spack dependencies using environments (#34029)
- Split
satisfies(..., strict=True/False)
into two functions (#35681) - spack install: simplify behavior when inside environments (#35206)
- Major simplification of CI boilerplate in stacks (#34272, #36045)
- Many improvements to our CI pipeline's reliability
- Module file generation is disabled by default; you'll need to enable it to use it (#37258)
- Support for Python 2 was deprecated in
v0.19.0
and has been removed.v0.20.0
only supports Python 3.6 and higher. - Deprecated target names are no longer recognized by Spack. Use generic names instead:
graviton
is nowcortex_a72
graviton2
is nowneoverse_n1
graviton3
is nowneoverse_v1
blacklist
andwhitelist
in module configuration were deprecated inv0.19.0
and are removed in this release. Useexclude
andinclude
instead.- The
ignore=
parameter of theextends()
directive has been removed. It was not used by any builtin packages and is no longer needed to avoid conflicts in environment views (#35588). - Support for the old YAML buildcache format has been removed. It was deprecated in
v0.19.0
(#34347). spack find --bootstrap
has been removed. It was deprecated inv0.19.0
. Usespack --bootstrap find
instead (#33964).spack bootstrap trust
andspack bootstrap untrust
are now removed, having been deprecated inv0.19.0
. Usespack bootstrap enable
andspack bootstrap disable
.- The
--mirror-name
,--mirror-url
, and--directory
options to buildcache and mirror commands were deprecated inv0.19.0
and have now been removed. They have been replaced by positional arguments (#37457). - Deprecate
env:
as top level environment key (#37424) - deprecate buildcache create --rel, buildcache install --allow-root (#37285)
- Support for very old perl-like spec format strings (e.g.,
$_$@$%@+$+$=
) has been removed (#37425). This was deprecated in inv0.15
(#10556).
- bugfix: don't fetch package metadata for unknown concrete specs (#36990)
- Improve package source code context display on error (#37655)
- Relax environment manifest filename requirements and lockfile identification criteria (#37413)
installer.py
: drop build edges of installed packages by default (#36707)- Bugfix: package requirements with git commits (#35057, #36347)
- Package requirements: allow single specs in requirement lists (#36258)
- conditional variant values: allow boolean (#33939)
- spack uninstall: follow run/link edges on --dependents (#34058)
- 7,179 total packages, 499 new since
v0.19.0
- 329 new Python packages
- 31 new R packages
- 336 people contributed to this release
- 317 committers to packages
- 62 committers to core
buildcache create
: make "file exists" less verbose (#35019)spack mirror create
: don't change paths to urls (#34992)- Improve error message for requirements (#33988)
- uninstall: fix accidental cubic complexity (#34005)
- scons: fix signature for
install_args
(#34481) - Fix
combine_phase_logs
text encoding issues (#34657) - Use a module-like object to propagate changes in the MRO, when setting build env (#34059)
- PackageBase should not define builder legacy attributes (#33942)
- Forward lookup of the "run_tests" attribute (#34531)
- Bugfix for timers (#33917, #33900)
- Fix path handling in prefix inspections (#35318)
- Fix libtool filter for Fujitsu compilers (#34916)
- Bug fix for duplicate rpath errors on macOS when creating build caches (#34375)
- FileCache: delete the new cache file on exception (#34623)
- Propagate exceptions from Spack python console (#34547)
- Tests: Fix a bug/typo in a
config_values.py
fixture (#33886) - Various CI fixes (#33953, #34560, #34560, #34828)
- Docs: remove monitors and analyzers, typos (#34358, #33926)
- bump release version for tutorial command (#33859)
v0.19.0
is a major feature release.
-
Package requirements
Spack's traditional package preferences are soft, but we've added hard requriements to
packages.yaml
andspack.yaml
(#32528, #32369). Package requirements use the same syntax as specs:packages: libfabric: require: "@1.13.2" mpich: require: - one_of: ["+cuda", "+rocm"]
More details in the docs.
-
Environment UI Improvements
-
Fewer surprising modifications to
spack.yaml
(#33711):-
spack install
in an environment will no longer add to thespecs:
list; you'll need to either usespack add <spec>
orspack install --add <spec>
. -
Similarly,
spack uninstall
will not remove from your environment'sspecs:
list; you'll need to usespack remove
orspack uninstall --remove
.
This will make it easier to manage an environment, as there is clear separation between the stack to be installed (
spack.yaml
/spack.lock
) and which parts of it should be installed (spack install
/spack uninstall
). -
-
concretizer:unify:true
is now the default mode for new environments (#31787)We see more users creating
unify:true
environments now. Users who needunify:false
can add it to their environment to get the old behavior. This will concretize every spec in the environment independently. -
Include environment configuration from URLs (#29026, docs)
You can now include configuration in your environment directly from a URL:
spack: include: - https://github.com/path/to/raw/config/compilers.yaml
-
-
Multiple Build Systems
An increasing number of packages in the ecosystem need the ability to support multiple build systems (#30738, docs), either across versions, across platforms, or within the same version of the software. This has been hard to support through multiple inheritance, as methods from different build system superclasses would conflict.
package.py
files can now define separate builder classes with installation logic for different build systems, e.g.:class ArpackNg(CMakePackage, AutotoolsPackage): build_system( conditional("cmake", when="@0.64:"), conditional("autotools", when="@:0.63"), default="cmake", ) class CMakeBuilder(spack.build_systems.cmake.CMakeBuilder): def cmake_args(self): pass class Autotoolsbuilder(spack.build_systems.autotools.AutotoolsBuilder): def configure_args(self): pass
-
Compiler and variant propagation
Currently, compiler flags and variants are inconsistent: compiler flags set for a package are inherited by its dependencies, while variants are not. We should have these be consistent by allowing for inheritance to be enabled or disabled for both variants and compiler flags.
Example syntax:
package ++variant
: enabled variant that will be propagated to dependenciespackage +variant
: enabled variant that will NOT be propagated to dependenciespackage ~~variant
: disabled variant that will be propagated to dependenciespackage ~variant
: disabled variant that will NOT be propagated to dependenciespackage cflags==-g
:cflags
will be propagated to dependenciespackage cflags=-g
:cflags
will NOT be propagated to dependencies
Syntax for non-boolan variants is similar to compiler flags. More in the docs for variants and compiler flags.
-
Enhancements to git version specifiers
-
v0.18.0
added the ability to use git commits as versions. You can now use thegit.
prefix to specify git tags or branches as versions. All of these are valid git versions inv0.19
(#31200):foo@abcdef1234abcdef1234abcdef1234abcdef1234 # raw commit foo@git.abcdef1234abcdef1234abcdef1234abcdef1234 # commit with git prefix foo@git.develop # the develop branch foo@git.0.19 # use the 0.19 tag
-
v0.19
also gives you more control over how Spack interprets git versions, in case Spack cannot detect the version from the git repository. You can suffix a git version with=<version>
to force Spack to concretize it as a particular version (#30998, #31914, #32257):# use mybranch, but treat it as version 3.2 for version comparison foo@git.mybranch=3.2 # use the given commit, but treat it as develop for version comparison foo@git.abcdef1234abcdef1234abcdef1234abcdef1234=develop
More in the docs
-
-
Changes to Cray EX Support
Cray machines have historically had their own "platform" within Spack, because we needed to go through the module system to leverage compilers and MPI installations on these machines. The Cray EX programming environment now provides standalone
craycc
executables and propermpicc
wrappers, so Spack can treat EX machines like Linux with extra packages (#29392).We expect this to greatly reduce bugs, as external packages and compilers can now be used by prefix instead of through modules. We will also no longer be subject to reproducibility issues when modules change from Cray PE release to release and from site to site. This also simplifies dealing with the underlying Linux OS on cray systems, as Spack will properly model the machine's OS as either SuSE or RHEL.
-
Improvements to tests and testing in CI
-
spack ci generate --tests
will generate a.gitlab-ci.yml
file that not only does builds but also runs tests for built packages (#27877). Public GitHub pipelines now also run tests in CI. -
spack test run --explicit
will only run tests for packages that are explicitly installed, instead of all packages.
-
-
Experimental binding link model
You can add a new option to
config.yaml
to make Spack embed absolute paths to needed shared libraries in ELF executables and shared libraries on Linux (#31948, docs):config: shared_linking: type: rpath bind: true
This can improve launch time at scale for parallel applications, and it can make installations less susceptible to environment variables like
LD_LIBRARY_PATH
, even especially when dealing with external libraries that useRUNPATH
. You can think of this as a faster, even higher-precedence version ofRPATH
.
spack spec
prints dependencies more legibly. Dependencies in the output now appear at the earliest level of indentation possible (#33406)- You can override
package.py
attributes likeurl
, directly inpackages.yaml
(#33275, docs) - There are a number of new architecture-related format strings you can use in Spack configuration files to specify paths (#29810, docs)
- Spack now supports bootstrapping Clingo on Windows (#33400)
- There is now support for an
RPATH
-like library model on Windows (#31930)
- Major performance improvements for installation from binary caches (#27610, #33628, #33636, #33608, #33590, #33496)
- Test suite can now be parallelized using
xdist
(used in GitHub Actions) (#32361) - Reduce lock contention for parallel builds in environments (#31643)
- We now build nearly all of E4S with
oneapi
in our buildcache (#31781, #31804, #31804, #31803, #31840, #31991, #32117, #32107, #32239) - Added 3 new machine learning-centric stacks to binary cache:
x86_64_v3
, CUDA, ROCm (#31592, #33463)
-
Support for Python 3.5 is dropped (#31908). Only Python 2.7 and 3.6+ are officially supported.
-
This is the last Spack release that will support Python 2 (#32615). Spack
v0.19
will emit a deprecation warning if you run it with Python 2, and Python 2 support will soon be removed from thedevelop
branch. -
LD_LIBRARY_PATH
is no longer set by default byspack load
or module loads.Setting
LD_LIBRARY_PATH
in Spack environments/modules can cause binaries from outside of Spack to crash, and Spack's own builds useRPATH
and do not needLD_LIBRARY_PATH
set in order to run. If you still want the old behavior, you can run these commands to configure Spack to setLD_LIBRARY_PATH
:spack config add modules:prefix_inspections:lib64:[LD_LIBRARY_PATH] spack config add modules:prefix_inspections:lib:[LD_LIBRARY_PATH]
-
The
spack:concretization:[together|separately]
has been removed after being deprecated inv0.18
. Useconcretizer:unify:[true|false]
. -
config:module_roots
is no longer supported after being deprecated inv0.18
. Use configuration in module sets instead (#28659, docs). -
spack activate
andspack deactivate
are no longer supported, having been deprecated inv0.18
. Use an environment with a view instead of activating/deactivating (docs). -
The old YAML format for buildcaches is now deprecated (#33707). If you are using an old buildcache with YAML metadata you will need to regenerate it with JSON metadata.
-
spack bootstrap trust
andspack bootstrap untrust
are deprecated in favor ofspack bootstrap enable
andspack bootstrap disable
and will be removed inv0.20
. (#33600) -
The
graviton2
architecture has been renamed toneoverse_n1
, andgraviton3
is nowneoverse_v1
. Buildcaches using the old architecture names will need to be rebuilt. -
The terms
blacklist
andwhitelist
have been replaced withinclude
andexclude
in all configuration files (#31569). You can usespack config update
to automatically fix your configuration files.
- Permission setting on installation now handles effective uid properly (#19980)
buildable:true
for an MPI implementation now overridesbuildable:false
formpi
(#18269)- Improved error messages when attempting to use an unconfigured compiler (#32084)
- Do not punish explicitly requested compiler mismatches in the solver (#30074)
spack stage
: add missing --fresh and --reuse (#31626)- Fixes for adding build system executables like
cmake
to package scope (#31739) - Bugfix for binary relocation with aliased strings produced by newer
binutils
(#32253)
- 6,751 total packages, 335 new since
v0.18.0
- 141 new Python packages
- 89 new R packages
- 303 people contributed to this release
- 287 committers to packages
- 57 committers to core
- Fix several bugs related to bootstrapping (#30834,#31042,#31180)
- Fix a regression that was causing spec hashes to differ between Python 2 and Python 3 (#31092)
- Fixed compiler flags for oneAPI and DPC++ (#30856)
- Fixed several issues related to concretization (#31142,#31153,#31170,#31226)
- Improved support for Cray manifest file and
spack external find
(#31144,#31201,#31173,#31186) - Assign a version to openSUSE Tumbleweed according to the GLIBC version in the system (#19895)
- Improved Dockerfile generation for
spack containerize
(#29741,#31321) - Fixed a few bugs related to concurrent execution of commands (#31509,#31493,#31477)
- WarpX: add v22.06, fixed libs property (#30866,#31102)
- openPMD: add v0.14.5, update recipe for @develop (#29484,#31023)
v0.18.0
is a major feature release.
-
Concretizer now reuses by default
spack install --reuse
was introduced inv0.17.0
, and--reuse
is now the default concretization mode. Spack will try hard to resolve dependencies using installed packages or binaries (#30396).To avoid reuse and to use the latest package configurations, (the old default), you can use
spack install --fresh
, or add configuration like this to your environment orconcretizer.yaml
:concretizer: reuse: false
-
Finer-grained hashes
Spack hashes now include
link
,run
, andbuild
dependencies, as well as a canonical hash of package recipes. Previously, hashes only includedlink
andrun
dependencies (thoughbuild
dependencies were stored by environments). We coarsened the hash to reduce churn in user installations, but the new default concretizer behavior mitigates this concern and gets us reuse and provenance. You will be able to see the build dependencies of new installations withspack find
. Old installations will not change and their hashes will not be affected. (#28156, #28504, #30717, #30861) -
Improved error messages
Error handling with the new concretizer is now done with optimization criteria rather than with unsatisfiable cores, and Spack reports many more details about conflicting constraints. (#30669)
-
Unify environments when possible
Environments have thus far supported
concretization: together
orconcretization: separately
. These have been replaced by a new preference inconcretizer.yaml
:concretizer: unify: [true|false|when_possible]
concretizer:unify:when_possible
will try to resolve a fully unified environment, but if it cannot, it will create multiple configurations of some packages where it has to. For large environments that previously had to be concretized separately, this can result in a huge speedup (40-50x). (#28941) -
Automatically find externals on Cray machines
Spack can now automatically discover installed packages in the Cray Programming Environment by running
spack external find
(orspack external read-cray-manifest
to only query the PE). Packages from the PE (e.g.,cray-mpich
are added to the database with full dependency information, and compilers from the PE are added tocompilers.yaml
. Available with the June 2022 release of the Cray Programming Environment. (#24894, #30428) -
New binary format and hardened signing
Spack now has an updated binary format, with improvements for security. The new format has a detached signature file, and Spack verifies the signature before untarring or decompressing the binary package. The previous format embedded the signature in a
tar
file, which required the client to runtar
before verifying (#30750). Spack can still install from build caches using the old format, but we encourage users to switch to the new format going forward.Production GitLab pipelines have been hardened to securely sign binaries. There is now a separate signing stage so that signing keys are never exposed to build system code, and signing keys are ephemeral and only live as long as the signing pipeline stage. (#30753)
-
Bootstrap mirror generation
The
spack bootstrap mirror
command can automatically create a mirror for bootstrapping the concretizer and other needed dependencies in an air-gapped environment. (#28556) -
Nascent Windows support
Spack now has initial support for Windows. Spack core has been refactored to run in the Windows environment, and a small number of packages can now build for Windows. More details are in the documentation (#27021, #28385, many more)
-
Makefile generation
spack env depfile
can be used to generate aMakefile
from an environment, which can be used to build packages the environment in parallel on a single node. e.g.:spack -e myenv env depfile > Makefile make
Spack propagates
gmake
jobserver information to builds so that their jobs can share cores. (#30039, #30254, #30302, #30526) -
New variant features
In addition to being conditional themselves, variants can now have conditional values that are only possible for certain configurations of a package. (#29530)
Variants can be declared "sticky", which prevents them from being enabled or disabled by the concretizer. Sticky variants must be set explicitly by users on the command line or in
packages.yaml
. (#28630)
- Allow conditional possible values in variants
- Add a "sticky" property to variants
- Environment views can optionally link only
run
dependencies withlink:run
(#29336) spack external find --all
finds library-only packages in addition to build dependencies (#28005)- Customizable
config:license_dir
option (#30135) spack external find --path PATH
takes a custom search path (#30479)spack spec
has a new--format
argument likespack find
(#27908)spack concretize --quiet
skips printing concretized specs (#30272)spack info
now has cleaner output and displays test info (#22097)- Package-level submodule option for git commit versions (#30085, #30037)
- Using
/hash
syntax to refer to concrete specs in an environment now works even if/hash
is not installed. (#30276)
- full hash (see above)
- new develop versioning scheme
0.19.0-dev0
- Allow for multiple dependencies/dependents from the same package (#28673)
- Splice differing virtual packages (#27919)
- Concretization of large environments with
unify: when_possible
is much faster than concretizing separately (#28941, see above) - Single-pass view generation algorithm is 2.6x faster (#29443)
oneapi
anddpcpp
flag support (#30783)- better support for
M1
anda64fx
(#30683)
-
Spack no longer supports Python
2.6
(#27256) -
Removed deprecated
--run-tests
option ofspack install
; usespack test
(#30461) -
Removed deprecated
spack flake8
; usespack style
(#27290) -
Deprecate
spack:concretization
config option; useconcretizer:unify
(#30038) -
Deprecate top-level module configuration; use module sets (#28659)
-
spack activate
andspack deactivate
are deprecated in favor of environments; will be removed in0.19.0
(#29430; see alsolink:run
in #29336 above)
- Fix bug that broke locks with many parallel builds (#27846)
- Many bugfixes and consistency improvements for the new concretizer
and
--reuse
(#30357, #30092, #29835, #29933, #28605, #29694, #28848)
CMakePackage
usesCMAKE_INSTALL_RPATH_USE_LINK_PATH
(#29703)- Refactored
lua
support:lua-lang
virtual supports bothlua
andluajit
via newLuaPackage
build system(#28854) - PythonPackage: now installs packages with
pip
(#27798) - Python: improve site_packages_dir handling (#28346)
- Extends: support spec, not just package name (#27754)
find_libraries
: search for both .so and .dylib on macOS (#28924)- Use stable URLs and
?full_index=1
for all github patches (#29239)
- 6,416 total packages, 458 new since
v0.17.0
- 219 new Python packages
- 60 new R packages
- 377 people contributed to this release
- 337 committers to packages
- 85 committers to core
- Fix missing chgrp on symlinks in package installations (#30743)
- Allow having non-existing upstreams (#30744, #30746)
- Fix
spack stage
with custom paths (#30448) - Fix failing call for
spack buildcache save-specfile
(#30637) - Fix globbing in compiler wrapper (#30699)
- Fix --reuse with upstreams set in an environment (#29680)
- config add: fix parsing of validator error to infer type from oneOf (#29475)
- Fix spack -C command_line_scope used in conjunction with other flags (#28418)
- Use Spec.constrain to construct spec lists for stacks (#28783)
- Fix bug occurring when searching for inherited patches in packages (#29574)
- Fixed a few bugs when manipulating symlinks (#28318, #29515, #29636)
- Fixed a few minor bugs affecting command prompt, terminal title and argument completion (#28279, #28278, #28939, #29405, #29070, #29402)
- Fixed a few bugs affecting the spack ci command (#29518, #29419)
- Fix handling of Intel compiler environment (#29439)
- Fix a few edge cases when reindexing the DB (#28764)
- Remove "Known issues" from documentation (#29664)
- Other miscellaneous bugfixes (0b72e070583fc5bcd016f5adc8a84c99f2b7805f, #28403, #29261)
- Allow locks to work under high contention (#27846)
- Improve errors messages from clingo (#27707 #27970)
- Respect package permissions for sbang (#25764)
- Fix --enable-locks behavior (#24675)
- Fix log-format reporter ignoring install errors (#25961)
- Fix overloaded argparse keys (#27379)
- Allow style commands to run with targets other than "develop" (#27472)
- Log lock messages to debug level, instead of verbose level (#27408)
- Handle invalid unicode while logging (#21447)
- spack audit: fix API calls to variants (#27713)
- Provide meaningful message for empty environment installs (#28031)
- Added opensuse leap containers to spack containerize (#27837)
- Revert "patches: make re-applied patches idempotent" (#27625)
- MANPATH can use system defaults (#21682)
- Add "setdefault" subcommand to
spack module tcl
(#14686) - Regenerate views when specs already installed (#28113)
- Fix external package detection for OpenMPI (#27255)
- Update the UPC++ package to 2021.9.0 (#26996)
- Added py-vermin v1.3.2 (#28072)
v0.17.0
is a major feature release.
-
New concretizer is now default The new concretizer introduced as an experimental feature in
v0.16.0
is now the default (#25502). The new concretizer is based on the clingo logic programming system, and it enables us to do much higher quality and faster dependency solving The old concretizer is still available via theconcretizer: original
setting, but it is deprecated and will be removed inv0.18.0
. -
Binary Bootstrapping To make it easier to use the new concretizer and binary packages, Spack now bootstraps
clingo
andGnuPG
from public binaries. If it is not able to bootstrap them from binaries, it installs them from source code. With these changes, you should still be able to clone Spack and start using it almost immediately. (#21446, #22354, #22489, #22606, #22720, #22720, #23677, #23946, #24003, #25138, #25607, #25964, #26029, #26399, #26599). -
Reuse existing packages (experimental) The most wanted feature from our 2020 user survey and the most wanted Spack feature of all time (#25310).
spack install
,spack spec
, andspack concretize
now have a--reuse
option, which causes Spack to minimize the number of rebuilds it does. The--reuse
option will try to find existing installations and binary packages locally and in registered mirrors, and will prefer to use them over building new versions. This will allow users to build from source far less than in prior versions of Spack. This feature will continue to be improved, with configuration options and better CLI expected inv0.17.1
. It will become the default concretization mode inv0.18.0
. -
Better error messages We have improved the error messages generated by the new concretizer by using unsatisfiable cores. Spack will now print a summary of the types of constraints that were violated to make a spec unsatisfiable (#26719).
-
Conditional variants Variants can now have a
when="<spec>"
clause, allowing them to be conditional based on the version or other attributes of a package (#24858). -
Git commit versions In an environment and on the command-line, you can now provide a full, 40-character git commit as a version for any package with a top-level
git
URL. e.g.,spack install hdf5@45bb27f58240a8da7ebb4efc821a1a964d7712a8
. Spack will compare the commit to tags in the git repository to understand what versions it is ahead of or behind. -
Override local config and cache directories You can now set
SPACK_DISABLE_LOCAL_CONFIG
to disable the~/.spack
and/etc/spack
configuration scopes.SPACK_USER_CACHE_PATH
allows you to move caches out of~/.spack
, as well (#27022, #26735). This addresses common problems where users could not isolate CI environments from local configuration. -
Improvements to Spack Containerize For added reproducibility, you can now pin the Spack version used by
spack containerize
(#21910). The container build will only build with the Spack version pinned at build recipe creation instead of the latest Spack version. -
New commands for dealing with tags The
spack tags
command allows you to list tags on packages (#26136), and you can list tests and filter tags withspack test list
(#26842).
- Copy and relocate environment views as stand-alone installations (#24832)
spack diff
command can diff two installed specs (#22283, #25169)spack -c <config>
can set one-off config parameters on CLI (#22251)spack load --list
is an alias forspack find --loaded
(#27184)spack gpg
can export private key with--secret
(#22557)spack style
automatically bootstraps dependencies (#24819)spack style --fix
automatically invokesisort
(#24071)- build dependencies can be installed from build caches with
--include-build-deps
(#19955) spack audit
command for checking package constraints (#23053)- Spack can now fetch from
CVS
repositories (yep, really) (#23212) spack monitor
lets you upload analysis about installations to a spack monitor server (#23804, #24321, #23777, #25928))spack python --path
shows whichpython
Spack is using (#22006)spack env activate --temp
can create temporary environments (#25388)--preferred
and--latest
options forspack checksum
(#25830)cc
is now pure posix and runs on Alpine (#26259)SPACK_PYTHON
environment variable sets whichpython
spack uses (#21222)SPACK_SKIP_MODULES
lets you sourcesetup-env.sh
faster if you don't need modules (#24545)
spec.yaml
files are nowspec.json
, yielding a large speed improvement (#22845)- Splicing allows Spack specs to store mixed build provenance (#20262)
- More extensive hooks API for installations (#21930)
- New internal API for getting the active environment (#25439)
- Parallelize separate concretization in environments; Previously 55 min E4S solve now takes 2.5 min (#26264)
- Drastically improve YamlFilesystemView file removal performance via batching (#24355)
- Speed up spec comparison (#21618)
- Speed up environment activation (#25633)
- support for new generic
x86_64_v2
,x86_64_v3
,x86_64_v4
targets (see archspec#31) spack arch --generic
lets you get the best generic architecture for your node (#27061)- added support for aocc (#20124),
arm
compiler ongraviton2
(#24904) and ona64fx
(#24524),
- Add support for GCS Bucket Mirrors (#26382)
- Add
spackbot
to help package maintainers with notifications. See spack.github.io/spackbot - Reproducible pipeline builds with
spack ci rebuild
(#22887) - Removed redundant concretizations from GitLab pipeline generation (#26622)
- Spack CI no longer generates jobs for unbuilt specs (#20435)
- Every pull request pipeline has its own buildcache (#25529)
--no-add
installs only specified specs and only if already present in… (#22657)- Add environment-aware
spack buildcache sync
command (#25470) - Binary cache installation speedups and improvements (#19690, #20768)
spack setup
was deprecated in v0.16.0, and has now been removed. Usespack develop
andspack dev-build
.- Remove unused
--dependencies
flag fromspack load
(#25731) - Remove stubs for
spack module [refresh|find|rm|loads]
, all of which were deprecated in 2018.
- Deactivate previous env before activating new one (#25409)
- Many fixes to error codes from
spack install
(#21319, #27012, #25314) - config add: infer type based on JSON schema validation errors (#27035)
spack config edit
now works even ifspack.yaml
is broken (#24689)
- Allow non-empty version ranges like
1.1.0:1.1
(#26402) - Remove
.99
's from many version ranges (#26422) - Python: use platform-specific site packages dir (#25998)
CachedCMakePackage
for using *.cmake initial config files (#19316)lua-lang
allows swappinglua
andluajit
(#22492)- Better support for
ld.gold
andld.lld
(#25626) - build times are now stored as metadata in
$prefix/.spack
(#21179) - post-install tests can be reused in smoke tests (#20298)
- Packages can use
pypi
attribute to inferhomepage
/url
/list_url
(#17587) - Use gnuconfig package for
config.guess
file replacement (#26035) - patches: make re-applied patches idempotent (#26784)
- 5969 total packages, 920 new since
v0.16.0
- 358 new Python packages, 175 new R packages
- 513 people contributed to this release
- 490 committers to packages
- 105 committers to core
- Lots of GPU updates:
- ~77 CUDA-related commits
- ~66 AMD-related updates
- ~27 OneAPI-related commits
- 30 commits from AMD toolchain support
spack test
usage in packages is increasing- 1669 packages with tests (mostly generic python tests)
- 93 packages with their own tests
- clang/llvm: fix version detection (#19978)
- Fix use of quotes in Python build system (#22279)
- Cray: fix extracting paths from module files (#23472)
- Use AWS CloudFront for source mirror (#23978)
- Ensure all roots of an installed environment are marked explicit in db (#24277)
- Fix fetching for Python 3.8 and 3.9 (#24686)
- locks: only open lockfiles once instead of for every lock held (#24794)
- Remove the EOL centos:6 docker image
- Major performance improvement for
spack load
and other commands. (#23661) spack fetch
is now environment-aware. (#19166)- Numerous fixes for the new,
clingo
-based concretizer. (#23016, #23307, #23090, #22896, #22534, #20644, #20537, #21148) - Supoprt for automatically bootstrapping
clingo
from source. (#20652, #20657 #21364, #21446, #21913, #22354, #22444, #22460, #22489, #22610, #22631) - Python 3.10 support:
collections.abc
(#20441) - Fix import issues by using
__import__
instead of Spack package importe. (#23288, #23290) - Bugfixes and
--source-dir
argument forspack location
. (#22755, #22348, #22321) - Better support for externals in shared prefixes. (#22653)
spack build-env
now prefers specs defined in the active environment. (#21642)- Remove erroneous warnings about quotes in
from_sourcing_files
. (#22767) - Fix clearing cache of
InternalConfigScope
. (#22609) - Bugfix for active when pkg is already active error. (#22587)
- Make
SingleFileScope
able to repopulate the cache after clearing it. (#22559) - Channelflow: Fix the package. (#22483)
- More descriptive error message for bugs in
package.py
(#21811) - Use package-supplied
autogen.sh
. (#20319) - Respect
-k/verify-ssl-false
in_existing_url
method. (#21864)
This minor release includes a new feature and associated fixes:
- intel-oneapi support through new packages (#20411, #20686, #20693, #20717, #20732, #20808, #21377, #21448)
This release also contains bug fixes/enhancements for:
- HIP/ROCm support (#19715, #20095)
- concretization (#19988, #20020, #20082, #20086, #20099, #20102, #20128, #20182, #20193, #20194, #20196, #20203, #20247, #20259, #20307, #20362, #20383, #20423, #20473, #20506, #20507, #20604, #20638, #20649, #20677, #20680, #20790)
- environment install reporting fix (#20004)
- avoid import in ABI compatibility info (#20236)
- restore ability of dev-build to skip patches (#20351)
- spack find -d spec grouping (#20028)
- spack smoke test support (#19987, #20298)
- macOS fixes (#20038, #21662)
- abstract spec comparisons (#20341)
- continuous integration (#17563)
- performance improvements for binary relocation (#19690, #20768)
- additional sanity checks for variants in builtin packages (#20373)
- do not pollute auto-generated configuration files with empty lists or dicts (#20526)
plus assorted documentation (#20021, #20174) and package bug fixes/enhancements (#19617, #19933, #19986, #20006, #20097, #20198, #20794, #20906, #21411).
v0.16.0
is a major feature release.
-
New concretizer (experimental) Our new backtracking concretizer is now in Spack as an experimental feature. You will need to install
clingo@master+python
and setconcretizer: clingo
inconfig.yaml
to use it. The original concretizer is not exhaustive and is not guaranteed to find a solution if one exists. We encourage you to use the new concretizer and to report any bugs you find with it. We anticipate making the new concretizer the default and including all required dependencies for it in Spackv0.17
. For more details, see #19501. -
spack test (experimental) Users can add
test()
methods to their packages to run smoke tests on installations with the newspack test
command (the oldspack test
is nowspack unit-test
).spack test
is environment-aware, so you canspack install
an environment andspack test run
smoke tests on all of its packages. Historical test logs can be perused withspack test results
. Generic smoke tests for MPI implementations, C, C++, and Fortran compilers as well as specific smoke tests for 18 packages. This is marked experimental because the test API (self.run_test()
) is likely to be change, but we encourage users to upstream tests, and we will maintain and refactor any that are added to mainline packages (#15702). -
spack develop New
spack develop
command allows you to develop several packages at once within a Spack environment. Runningspack develop foo@v1
andspack develop bar@v2
will check out specific versions offoo
andbar
into subdirectories, which you can then build incrementally withspack install
(#15256). -
More parallelism Spack previously installed the dependencies of a single spec in parallel. Entire environments can now be installed in parallel, greatly accelerating builds of large environments. get parallelism from individual specs. Spack now parallelizes entire environment builds (#18131).
-
Customizable base images for spack containerize
spack containerize
previously only output aDockerfile
based onubuntu
. You may now specify any base image of your choosing (#15028). -
more external finding
spack external find
was added inv0.15
, but onlycmake
had support.spack external find
can now findbison
,cuda
,findutils
,flex
,git
,lustre
m4
,mpich
,mvapich2
,ncurses
,openmpi
,perl
,spectrum-mpi
,tar
, andtexinfo
on your system and add them automatically topackages.yaml
. -
Support aocc, nvhpc, and oneapi compilers We are aggressively pursuing support for the newest vendor compilers, especially those for the U.S. exascale and pre-exascale systems. Compiler classes and auto-detection for
aocc
,nvhpc
,oneapi
are now in Spack (#19345, #19294, #19330).
- New
spack mark
command can be used to designate packages as explicitly installed, so thatspack gc
will not garbage-collect them (#16662). install_tree
can be customized with Spack's projection format (#18341)sbang
now lives in theinstall_tree
so that all users can access it (#11598)csh
andtcsh
users no longer need to setSPACK_ROOT
before sourcingsetup-env.csh
(#18225)- Spec syntax now supports
variant=*
syntax for finding any package that has a particular variant (#19381). - Spack respects
SPACK_GNUPGHOME
variable for custom GPG directories (#17139) - Spack now recognizes Graviton chips
- Use spawn instead of fork on Python >= 3.8 on macOS (#18205)
- Use indexes for public build caches (#19101, #19117, #19132, #19141, #19209)
sbang
is an external package now (https://github.com/spack/sbang, #19582)archspec
is an external package now (https://github.com/archspec/archspec, #19600)
spack bootstrap
was deprecated in v0.14.0, and has now been removed.spack setup
is deprecated as of v0.16.0.- What was
spack test
is now calledspack unit-test
.spack test
is now the smoke testing feature in (2) above.
Some of the most notable bugfixes in this release include:
- Better warning messages for deprecated syntax in
packages.yaml
(#18013) buildcache list --allarch
now works properly (#17827)- Many fixes and tests for buildcaches and binary relcoation (#15687, *#17455, #17418, #17455, #15687, #18110)
Spack now has 5050 total packages, 720 of which were added since v0.15
.
- ROCm packages (
hip
,aomp
, more) added by AMD (#19957, #19832, others) - Many improvements for ARM support
llvm-flang
,flang
, andf18
removed, asllvm
has realflang
support since Flang was merged to LLVM mainline- Emerging support for
spack external find
andspack test
in packages.
- Major infrastructure improvements to pipelines on
gitlab.spack.io
- Support for testing PRs from forks (#19248) is being enabled for all
forks to enable rolling, up-to-date binary builds on
develop
This release contains one feature addition:
- Users can set
SPACK_GNUPGHOME
to override Spack's GPG path (#17139)
Several bugfixes for CUDA, binary packaging, and spack -V
:
- CUDA package's
.libs
method searches forlibcudart
instead oflibcuda
(#18000) - Don't set
CUDAHOSTCXX
in environments that contain CUDA (#17826) buildcache create
:NoOverwriteException
is a warning, not an error (#17832)- Fix
spack buildcache list --allarch
(#17884) spack -V
works withreleases/latest
tag and shallow clones (#17884)
And fixes for GitHub Actions and tests to ensure that CI passes on the release branch (#15687, #17279, #17328, #17377, #17732).
This release contains the following bugfixes:
- Fix handling of relative view paths (#17721)
- Fixes for binary relocation (#17418, #17455)
- Fix redundant printing of error messages in build environment (#17709)
It also adds a support script for Spack tutorials:
- Add a tutorial setup script to share/spack (#17705, #17722)
This minor release includes two new features:
- Spack install verbosity is decreased, and more debug levels are added (#17546)
- The $spack/share/spack/keys directory contains public keys that may be optionally trusted for public binary mirrors (#17684)
This release also includes several important fixes:
- MPICC and related variables are now cleand in the build environment (#17450)
- LLVM flang only builds CUDA offload components when +cuda (#17466)
- CI pipelines no longer upload user environments that can contain secrets to the internet (#17545)
- CI pipelines add bootstrapped compilers to the compiler config (#17536)
spack buildcache list
does not exit on first failure and lists later mirrors (#17565)- Apple's "gcc" executable that is an apple-clang compiler does not generate a gcc compiler config (#17589)
- Mixed compiler toolchains are merged more naturally across different compiler suffixes (#17590)
- Cray Shasta platforms detect the OS properly (#17467)
- Additional more minor fixes.
This minor release includes several important fixes:
- Fix shell support on Cray (#17386)
- Fix use of externals installed with other Spack instances (#16954)
- Fix gcc+binutils build (#9024)
- Fixes for usage of intel-mpi (#17378 and #17382)
- Fixes to Autotools config.guess detection (#17333 and #17356)
- Update
spack install
message to prompt user when an environment is not explicitly activated (#17454)
This release also adds a mirror for all sources that are fetched in Spack (#17077). It is expected to be useful when the official website for a Spack package is unavailable.
v0.15.0
is a major feature release.
-
Cray support Spack will now work properly on Cray "Cluster" systems (non XC systems) and after a
module purge
command on Cray systems. See #12989 -
Virtual package configuration Virtual packages are allowed in packages.yaml configuration. This allows users to specify a virtual package as non-buildable without needing to specify for each implementation. See #14934
-
New config subcommands This release adds
spack config add
andspack config remove
commands to add to and remove from yaml configuration files from the CLI. See #13920 -
Environment activation Anonymous environments are no longer automatically activated in the current working directory. To activate an environment from a
spack.yaml
file in the current directory, use thespack env activate .
command. This removes a concern that users were too easily polluting their anonymous environments with accidental installations. See #17258 -
Apple clang compiler The clang compiler and the apple-clang compiler are now separate compilers in Spack. This allows Spack to improve support for the apple-clang compiler. See #17110
-
Finding external packages Spack packages can now support an API for finding external installations. This allows the
spack external find
command to automatically add installations of those packages to the user's configuration. See #15158
- support for using Spack with the fish shell (#9279)
spack load --first
option to load first match (instead of prompting user) (#15622)- support the Cray cce compiler both new and classic versions (#17256, #12989)
spack dev-build
command:- supports stopping before a specified phase (#14699)
- supports automatically launching a shell in the build environment (#14887)
spack install --fail-fast
allows builds to fail at the first error (rather than best-effort) (#15295)- environments: SpecList references can be dereferenced as compiler or dependency constraints (#15245)
spack view
command: new support for a copy/relocate view type (#16480)- ci pipelines: see documentation for several improvements
spack mirror -a
command now supports excluding packages (#14154)spack buildcache create
is now environment-aware (#16580)- module generation: more flexible format for specifying naming schemes (#16629)
- lmod module generation: packages can be configured as core specs for lmod hierarchy (#16517)
The following commands were deprecated in v0.13.0, and have now been removed:
spack configure
spack build
spack diy
The following commands were deprecated in v0.14.0, and will be removed in the next major release:
spack bootstrap
Some of the most notable bugfixes in this release include:
- Spack environments can now contain the string
-h
(#15429) - The
spack install
gracefully handles being backgrounded (#15723, #14682) - Spack uses
-isystem
instead of-I
in cases that the underlying build system does as well (#16077) - Spack no longer prints any specs that cannot be safely copied into a Spack command (#16462)
- Incomplete Spack environments containing python no longer cause problems (#16473)
- Several improvements to binary package relocation
The Spack project is constantly engaged in routine maintenance, bugfixes, and improvements for the package ecosystem. Of particular note in this release are the following:
- Spack now contains 4339 packages. There are 430 newly supported packages in v0.15.0
- GCC now builds properly on ARM architectures (#17280)
- Python: patched to support compiling mixed C/C++ python modules through distutils (#16856)
- improvements to pytorch and py-tensorflow packages
- improvements to major MPI implementations: mvapich2, mpich, openmpi, and others
- Much of the Spack CI infrastructure has moved from Travis to GitHub Actions (#16610, #14220, #16345)
- All merges to the
develop
branch run E4S CI pipeline (#16338) - New
spack debug report
command makes reporting bugs easier (#15834)
This is a minor release on the 0.14
series. It includes performance
improvements and bug fixes:
- Improvements to how
spack install
handles foreground/background (#15723) - Major performance improvements for reading the package DB (#14693, #15777)
- No longer check for the old
index.yaml
database file (#15298) - Properly activate environments with '-h' in the name (#15429)
- External packages have correct
.prefix
in environments/views (#15475) - Improvements to computing env modifications from sourcing files (#15791)
- Bugfix on Cray machines when getting
TERM
env variable (#15630) - Avoid adding spurious
LMOD
env vars to Intel modules (#15778) - Don't output [+] for mock installs run during tests (#15609)
This is a bugfix release on top of v0.14.0
. Specific fixes include:
- several bugfixes for parallel installation (#15339, #15341, #15220, #15197)
spack load
now works with packages that have been renamed (#14348)- bugfix for
suite-sparse
installation (#15326) - deduplicate identical suffixes added to module names (#14920)
- fix issues with
configure_args
during module refresh (#11084) - increased test coverage and test fixes (#15237, #15354, #15346)
- remove some unused code (#15431)
v0.14.0
is a major feature release, with 3 highlighted features:
-
Distributed builds. Multiple Spack instances will now coordinate properly with each other through locks. This works on a single node (where you've called
spack
several times) or across multiple nodes with a shared filesystem. For example, with SLURM, you could buildtrilinos
and its dependencies on 2 24-core nodes, with 3 Spack instances per node and 8 build jobs per instance, withsrun -N 2 -n 6 spack install -j 8 trilinos
. This requires a filesystem with locking enabled, but not MPI or any other library for parallelism. -
Build pipelines. You can also build in parallel through Gitlab CI. Simply create a Spack environment and push it to Gitlab to build on Gitlab runners. Pipeline support is now integrated into a single
spack ci
command, so setting it up is easier than ever. See the Pipelines section in the docs. -
Container builds. The new
spack containerize
command allows you to create a Docker or Singularity recipe from any Spack environment. There are options to customize the build if you need them. See the Container Images section in the docs.
In addition, there are several other new commands, many bugfixes and
improvements, and spack load
no longer requires modules, so you can use
it the same way on your laptop or on your supercomputer.
Spack grew by over 300 packages since our last release in November 2019, and the project grew to over 500 contributors. Thanks to all of you for making yet another great release possible. Detailed notes below.
- Distributed builds: spack instances coordinate and build in parallel (#13100)
- New
spack ci
command to manage CI pipelines (#12854) - Generate container recipes from environments:
spack containerize
(#14202) spack load
now works without using modules (#14062, #14628)- Garbage collect old/unused installations with
spack gc
(#13534) - Configuration files all set environment modifications the same way (#14372, docs)
spack commands --format=bash
auto-generates completion (#14393, #14607)- Packages can specify alternate fetch URLs in case one fails (#13881)
- Improved locking for concurrency with environments (#14676, #14621, #14692)
spack test
sends args topytest
, supports better listing (#14319)- Better support for aarch64 and cascadelake microarch (#13825, #13780, #13820)
- Archspec is now a separate library (see https://github.com/archspec/archspec)
- Many improvements to the
spack buildcache
command (#14237, #14346, #14466, #14467, #14639, #14642, #14659, #14696, #14698, #14714, #14732, #14929, #15003, #15086, #15134)
- Compilers now require an exact match on version (#8735, #14730, #14752)
- Bugfix for patches that specified specific versions (#13989)
spack find -p
now works in environments (#10019, #13972)- Dependency queries work correctly in
spack find
(#14757) - Bugfixes for locking upstream Spack instances chains (#13364)
- Fixes for PowerPC clang optimization flags (#14196)
- Fix for issue with compilers and specific microarchitectures (#13733, #14798)
spack ci
(#12854)spack containerize
(#14202)spack gc
(#13534)spack load
accepts--only package
,--only dependencies
(#14062, #14628)spack commands --format=bash
(#14393)spack commands --update-completion
(#14607)spack install --with-cache
has new option:--no-check-signature
(#11107)spack test
now has--list
,--list-long
, and--list-names
(#14319)spack install --help-cdash
moves CDash help out of the main help (#13704)
spack release-jobs
has been rolled intospack ci
spack bootstrap
will be removed in a future version, as it is no longer needed to set up modules (seespack load
improvements above)
- New section on building container images with Spack (see docs)
- New section on using
spack ci
command to build pipelines (see docs) - Document how to add conditional dependencies (#14694)
- Document how to use Spack to replace Homebrew/Conda (#13083, see docs)
- 3,908 total packages (345 added since 0.13.0)
- Added first cut at a TensorFlow package (#13112)
- We now build R without "recommended" packages, manage them w/Spack (#12015)
- Elpa and OpenBLAS now leverage microarchitecture support (#13655, #14380)
- Fix
octave
compiler wrapper usage (#14726) - Enforce that packages in
builtin
aren't missing dependencies (#13949)
This release contains several bugfixes:
- bugfixes for invoking python in various environments (#14349, #14496, #14569)
- brought tab completion up to date (#14392)
- bugfix for removing extensions from views in order (#12961)
- bugfix for nondeterministic hashing for specs with externals (#14390)
This release contains more major performance improvements for Spack
environments, as well as bugfixes for mirrors and a python
issue with
RHEL8.
- mirror bugfixes: symlinks, duplicate patches, and exception handling (#13789)
- don't try to fetch
BundlePackages
(#13908) - avoid re-fetching patches already added to a mirror (#13908)
- avoid re-fetching already added patches (#13908)
- avoid re-fetching already added patches (#13908)
- allow repeated invocations of
spack mirror create
on the same dir (#13908) - bugfix for RHEL8 when
python
is unavailable (#14252) - improve concretization performance in environments (#14190)
- improve installation performance in environments (#14263)
This release contains major performance improvements for Spack environments, as well as some bugfixes and minor changes.
- allow missing modules if they are blacklisted (#13540)
- speed up environment activation (#13557)
- mirror path works for unknown versions (#13626)
- environments: don't try to modify run-env if a spec is not installed (#13589)
- use semicolons instead of newlines in module/python command (#13904)
- verify.py: os.path.exists exception handling (#13656)
- Document use of the maintainers field (#13479)
- bugfix with config caching (#13755)
- hwloc: added 'master' version pointing at the HEAD of the master branch (#13734)
- config option to allow gpg warning suppression (#13744)
- fix for relative symlinks when relocating binary packages (#13727)
- allow binary relocation of strings in relative binaries (#13724)
This is a bugfix release on top of v0.13.0
. Specific fixes include:
spack find
now displays variants and other spec constraints- bugfix: uninstall should find concrete specs by DAG hash (#13598)
- environments: make shell modifications partially unconditional (#13523)
- binary distribution: relocate text files properly in relative binaries (#13578)
- bugfix: fetch prefers to fetch local mirrors over remote resources (#13545)
- environments: only write when necessary (#13546)
- bugfix: spack.util.url.join() now handles absolute paths correctly (#13488)
- sbang: use utf-8 for encoding when patching (#13490)
- Specs with quoted flags containing spaces are parsed correctly (#13521)
- targets: print a warning message before downgrading (#13513)
- Travis CI: Test Python 3.8 (#13347)
- Documentation: Database.query methods share docstrings (#13515)
- cuda: fix conflict statements for x86-64 targets (#13472)
- cpu: fix clang flags for generic x86_64 (#13491)
- syaml_int type should use int.repr rather than str.repr (#13487)
- elpa: prefer 2016.05.004 until sse/avx/avx2 issues are resolved (#13530)
- trilinos: temporarily constrain netcdf@:4.7.1 (#13526)
v0.13.0
is our biggest Spack release yet, with many new major features.
From facility deployment to improved environments, microarchitecture
support, and auto-generated build farms, this release has features for all of
our users.
Spack grew by over 700 packages in the past year, and the project now has over 450 contributors. Thanks to all of you for making this release possible.
- Chaining: use dependencies from external "upstream" Spack instances
- Environments now behave more like virtualenv/conda
- Each env has a view: a directory with all packages symlinked in
- Activating an environment sets
PATH
,LD_LIBRARY_PATH
,CPATH
,CMAKE_PREFIX_PATH
,PKG_CONFIG_PATH
, etc. to point to this view.
- Spack detects and builds specifically for your microarchitecture
- named, understandable targets like
skylake
,broadwell
,power9
,zen2
- Spack knows which compilers can build for which architectures
- Packages can easily query support for features like
avx512
andsse3
- You can pick a target with, e.g.
spack install foo target=icelake
- named, understandable targets like
- Spack stacks: combinatorial environments for facility deployment
- Environments can now build cartesian products of specs (with
matrix:
) - Conditional syntax support to exclude certain builds from the stack
- Environments can now build cartesian products of specs (with
- Projections: ability to build easily navigable symlink trees environments
- Support no-source packages (BundlePackage) to aggregate related packages
- Extensions: users can write custom commands that live outside of Spack repo
- Support ARM and Fujitsu compilers
spack release-jobs
can detectpackage.py
changes and generate.gitlab-ci.yml
to create binaries for an environment or stack in parallel (initial support -- will change in future release).- Results of build pipelines can be uploaded to a CDash server.
- Spack can now upload/fetch from package mirrors in Amazon S3
spack mirror create --all
downloads all package sources/resources/patchesspack dev-build
runs phases of the install pipeline on the working directoryspack deprecate
permanently symlinks an old, unwanted package to a new onespack verify
chcecks that packages' files match what was originally installedspack find --json
printsJSON
that is easy to parse with, e.g.jq
spack find --format FORMAT
allows you to flexibly print package metadataspack spec --json
prints JSON version ofspec.yaml
- Auto-build requested compilers if they do not exist
- Spack automatically adds
RPATHs
needed to make executables find compiler runtime libraries (e.g., path to newerlibstdc++
inicpc
org++
) - setup-env.sh is now compatible with Bash, Dash, and Zsh
- Spack now caps build jobs at min(16, ncores) by default
spack compiler find
now also throttles number of spawned processes- Spack now writes stage directories directly to
$TMPDIR
instead of symlinking stages within$spack/var/spack/cache
. - Improved and more powerful
spec
format strings - You can pass a
spec.yaml
file anywhere in the CLI you can type a spec. - Many improvements to binary caching
- Gradually supporting new features from Environment Modules v4
spack edit
respectsVISUAL
environment variable- Simplified package syntax for specifying build/run environment modifications
- Numerous improvements to support for environments across Spack commands
- Concretization improvements
- Multi-lingual documentation (Started a Japanese translation)
- Tutorial now has its own site at spack-tutorial.readthedocs.io
- This enables us to keep multiple versions of the tutorial around
- Spack no longer supports dotkit (LLNL's homegrown, now deprecated module tool)
spack build
,spack configure
,spack diy
deprecated in favor ofspack dev-build
andspack install
- 3,563 total packages (718 added since 0.12.1)
- Spack now defaults to Python 3 (previously preferred 2.7 by default)
- Much improved ARM support thanks to Fugaku (RIKEN) and SNL teams
- Support new special versions: master, trunk, and head (in addition to develop)
- Better finding logic for libraries and headers
This is a minor bugfix release, with a minor fix in the tutorial and a flake8
fix.
Bugfixes
- Add
r
back to regex strings in binary distribution - Fix gcc install version in the tutorial
- Spack environments
spack.yaml
andspack.lock
files for tracking dependencies- Custom configurations via command line
- Better support for linking Python packages into view directories
- Packages have more control over compiler flags via flag handlers
- Better support for module file generation
- Better support for Intel compilers, Intel MPI, etc.
- Many performance improvements, improved startup time
- As of this release, all of Spack is permissively licensed under Apache-2.0 or MIT, at the user's option.
- Consents from over 300 contributors were obtained to make this relicense possible.
- Previous versions were distributed under the LGPL license, version 2.1.
Over 2,900 packages (800 added since last year)
Spack would not be possible without our community. Thanks to all of our contributors for the new features and packages in this release!
This release contains the following fixes:
- Fixes for
gfortran
7 compiler detection (#7017) - Fixes for exceptions thrown during module generation (#7173)
This release contains bugfixes for compiler flag handling. There were issues in v0.11.0
that caused some packages to be built without proper optimization.
Fixes:
- Issue #6999: FFTW installed with Spack 0.11.0 gets built without optimisations
Includes:
- PR #6415: Fixes for flag handling behavior
- PR #6960: Fix type issues with setting flag handlers
- 880e319: Upstream fixes to
list_url
in various R packages
Spack v0.11.0 contains many improvements since v0.10.0. Below is a summary of the major features, broken down by category.
- Spack now has 2,178 packages (from 1,114 in v0.10.0)
- Many more Python packages (356) and R packages (471)
- 48 Exascale Proxy Apps (try
spack list -t proxy-app
)
- Relocatable binary packages (
spack buildcache
, #4854) - Spack now fully supports Python 3 (#3395)
- Packages can be tagged and searched by tags (#4786)
- Custom module file templates using Jinja (#3183)
spack bootstrap
command now sets up a basic module environment (#3057)- Simplified and better organized help output (#3033)
- Improved, less redundant
spack install
output (#5714, #5950) - Reworked
spack dependents
andspack dependencies
commands (#4478)
- Multi-valued variants (#2386)
- New
conflicts()
directive (#3125) - New dependency type:
test
dependencies (#5132) - Packages can require their own patches on dependencies (#5476)
depends_on(..., patches=<patch list>)
- Build interface for passing linker information through Specs (#1875)
- Major packages that use blas/lapack now use this interface
- Flag handlers allow packages more control over compiler flags (#6415)
- Package subclasses support many more build systems:
- autotools, perl, qmake, scons, cmake, makefile, python, R, WAF
- package-level support for installing Intel HPC products (#4300)
spack blame
command shows contributors to packages (#5522)spack create
now guesses many more build systems (#2707)- Better URL parsing to guess package version URLs (#2972)
- Much improved
PythonPackage
support (#3367)
- Much faster concretization (#5716, #5783)
- Improved output redirection (redirecting build output works properly #5084)
- Numerous improvements to internal structure and APIs
- Many updates to documentation
- New tutorial material from SC17
- configuration
- build systems
- build interface
- working with module generation
- Documentation on docker workflows and best practices
- No longer build Python eggs -- installations are plain directories (#3587)
- Improved filtering of system paths from build PATHs and RPATHs (#2083, #3910)
- Git submodules are properly handled on fetch (#3956)
- Can now set default number of parallel build jobs in
config.yaml
- Improvements to
setup-env.csh
(#4044) - Better default compiler discovery on Mac OS X (#3427)
- clang will automatically mix with gfortran
- Improved compiler detection on Cray machines (#3075)
- Better support for IBM XL compilers
- Better tab completion
- Resume gracefully after prematurely terminated partial installs (#4331)
- Better mesa support (#5170)
Spack would not be possible without our community. Thanks to all of our contributors for the new features and packages in this release!
This is Spack v0.10.0
. With this release, we will start to push Spack
releases more regularly. This is the last Spack release without
automated package testing. With the next release, we will begin to run
package tests in addition to unit tests.
Spack has grown rapidly from 422 to
1,114 packages,
thanks to the hard work of over 100 contributors. Below is a condensed
version of all the changes since v0.9.1
.
- Grew from 422 to 1,114 packages
- Includes major updates like X11, Qt
- Expanded HPC, R, and Python ecosystems
- Major speed improvements for spack find and concretization
- Completely reworked architecture support
- Platforms can have front-end and back-end OS/target combinations
- Much better support for Cray and BG/Q cross-compiled environments
- Downloads are now cached locally
- Support installations in deeply nested directories: patch long shebangs using
sbang
- Easier global configuration via config.yaml
- customize install, stage, and cache locations
- Hierarchical configuration scopes: default, site, user
- Platform-specific scopes allow better per-platform defaults
- Ability to set
cflags
,cxxflags
,fflags
on the command line - YAML-configurable support for both Lmod and tcl modules in mainline
spack install
supports --dirty option for emergencies
- Support multiple dependency types:
build
,link
, andrun
- Added
Package
base classes for custom build systemsAutotoolsPackage
,CMakePackage
,PythonPackage
, etc.spack create
now guesses many more build systems
- Development environment integration with
spack setup
- New interface to pass linking information via
spec
objects- Currently used for
BLAS
/LAPACK
/SCALAPACK
libraries - Polymorphic virtual dependency attributes:
spec['blas'].blas_libs
- Currently used for
- Unit tests run continuously on Travis CI for Mac and Linux
- Switched from
nose
topytest
for unit tests.- Unit tests take 1 minute now instead of 8
- Massively expanded documentation
- Docs are now hosted on spack.readthedocs.io