CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

[0.18.0] - 2024-02-13

📃 Details

Added

• ProgressBar trait and progress bar for package writing by @wolfv (#525)

Changed

• improved logging in package validation to include package path by @orhun (#521)
• use resolvo 0.4.0 with better error messages by @baszalmstra (#523)

Fixed

• allow multiple clobbers per package by @wolfv (#526)
• remove drop-bomb, move empty folder removal to post_process by @wolfv (#519)
• keep in mind python: noarch packages in clobber calculations by @wolfv (#511)

[0.17.0] - 2024-02-01

✨ Highlights

This release contains some big changes to rattler:

Consistent clobbering

Rattler installs packages in parallel but this was at the cost of not being able to resolve files properly that existed in multiple packages. With this release we fixed this issue by creating a consistent clobbering experience. When a file is clobbered (installed by multiple packages) the last package in the topological ordering wins. This information is also recorded in the prefix itself which means that even if packages are added or removed from the environment the order remains consistent.

reqwest-middleware-client

The AuthenticatedClient has been rewritten by @vlad-ivanov-name. Instead of having a custom client for network requests we now use the reqwest-middleware crate. The rattler implementation adds a middleware that handles authentication. This changes makes it easier to integrate with other crates that use reqwest for network requests, and it allows users to add their own middleware.

Lock-file v4

The lock-file format has been updated to version 4. Originally our implementation was semi-compatible with conda-lock. We wanted to stay as close as possible to this format because it was already an established standard. However, with version 2 and 3 of the format we started to diverge more and more. We felt like the goals between both formats also started to diverge more and more so with version 4 we decided to completely abandon the conda-lock format and create our own. For more information about the lock-file format and the differences between conda-lock you can read the documentation. Note that all old formats (including the original conda-lock format) can still be parsed by rattler.

📃 Details

Added

• Add get_windows_launcher function by @wolfv (#477)
• Expose get_windows_launcher function by @wolfv (#477)
• Consistent clobbering & removal of __pycache__ by @wolfv (#437)
• Add name() to Channel by @ruben-arts (#495)
• Add timeout parameter to the solver by @wolfv (#499)
• Add a very simple basic test to validate that we can at least parse netrc properly by @mariusvniekerk (#503)

Changed

• Allow the full range of compression levels for zstd by @wolfv (#479)
• Make compression conversion functions pub by @wolfv (#480)
• Lock-file v4 by @baszalmstra (#484)
• Convert authenticated client to reqwest middleware by @vlad-ivanov-name (#488)
• Upgrade to latest resolvo main by @tdejager (#497)
• Bump: resolvo 0.3.0 by @baszalmstra (#500)

Fixed

• Copy over file permissions after reflink by @orhun (#485)
• Fix clippy and deprecation warnings by @wolfv (#490)
• Do not unwrap as much in clobberregistry by @wolfv (#489)
• Fix warning for deref on a double reference by @wolfv (#493)
• Fix self-clobbering when updating a package by @wolfv (#494)
• Fix netrc parsing into BasicAuth by @wolfv (#506)

New Contributors

• @vlad-ivanov-name made their first contribution in conda#482

[0.16.2] - 2024-01-11

📃 Details

Fixed

• Reduce tracing level for reflink by @baszalmstra (#475)

[0.16.1] - 2024-01-09

📃 Details

Added

• Add read_package_file function by @wolfv (#472)
• implement Clone for AboutJson by @0xbe7a (#467)
• Allow using str in HashMaps with a PackageName key by @baszalmstra (#468)

Changed

• Reflink files to destination if supported (instead of hardlinking) by @baszalmstra (#463)

Fixed

• Automatic clippy fixes by @wolfv (#470)
• Fix getting credentials from keyring error by @0xbe7a (#474)

[0.15.0] - 2024-01-05

📃 Details

Added

• Add ParseMatchSpecError and ParseMatchSpecError tests by @Johnwillliam (#434)
• Add option to force usage of fallback_auth_store by @0xbe7a (#435)
• New crate (rattler-index) with index functionality including python bindings by @BenjaminLowry (#436)
• Add support for netrc files by @mariusvniekerk (#395)

Changed

• Renamed behaviour to behavior (#428)
• Enabled more clippy lints by @baszalmstra (#462)
• Refactor Version.bump() to accept bumping major/minor/patch/last by @hadim (#452)

Fixed

• Default value for conda_packages in repodata.json by @BenjaminLowry (#441)
• Wildcard expansion for stored credentials of domains by @0xbe7a (#442)
• Use serde default for proper serialization by @ruben-arts (#443)
• Better detection of hardlinks and fallback to copy by @baszalmstra (#461)
• Re-download the repodata cache if is out of sync/corrupt by @orhun (#466)

[0.14.0] - 2023-12-05

📃 Details

Added

• Options to disable bz2 and zstd in fetch_repo_data (#420)
• Support for powerpc64 and s390x (#425)

Changed

• Renamed behaviour to behavior (#428)

Fixed

• Recursive look for parent process name (#424)
• Improve repodata fetch errors (#426)
• Use filelock for authentication fallback storage (#427)
• Improved lockfile version mismatch error (#423)

[0.13.0] - 2023-11-27

📃 Details

Added

• Experimental support for purls in PackageRecord and derived datastructures (#414)

Changed

• Rename pip to pypi in lockfile (#415)

Fixed

• Allow compilation for android (#418)
• Normalize relative-paths with writing to file (#416)

[0.12.3] - 2023-11-23

📃 Details

Fixed

• Expose missing StringMatcherParseError (#410)
• Fix JLAP issue by setting the nominal hash when first downloading repodata (#411)
• Support channel names with slashes (#413)

[0.12.2] - 2023-11-17

📃 Details

Fixed

• fix: make redaction work by using From explicitly (#408)

[0.12.1] - 2023-11-17

📃 Details

Fixed

• fix: redact tokens from urls in errors (#407)

[0.12.0] - 2023-11-14

✨ Highlights

Adds support for strict priority channel ordering, channel-specific selectors,

📃 Details

Added

• Add strict channel priority option (#385)
• Add lock-file forward compatibility error (#389)
• Add channel priority and channel-specific selectors to solver info (#394)

Changed

• Channel in the MatchSpec struct changed to Channel type (#401)

Fixed

• Expose previous python version information in transaction (#384)
• Avoid use of \ in doctest strings, for ide integration (#387)
• Issue with JLAP using the wrong hash (#390)
• Use the correct channel in the reason for exclude (#397)
• Environment activation for windows (#398)

[0.11.0] - 2023-10-17

✨ Highlights

Lock file support has been moved into its own crate (rattler_lock) and support for pip packages has been added.

📃 Details

Changed

• change authentication fallback warnings to debug by @ruben-arts in conda#365
• repodata cache now uses .info.json instead of .state.json by @dholth in conda#377
• lock file now lives in its own crate with pip support by @baszalmstra in conda#378

Fixed

• Nushell fixes by @wolfv in conda#360
• Construct placeholder string at runtime to work around invalid conda prefix replacement by @baszalmstra in conda#371
• xonsh extension by @ruben-arts in conda#375

New Contributors

• @dholth made their first contribution in conda#377

Full Changelog: https://github.com/conda/rattler/compare/v0.10.0...v0.11.0

[0.10.0] - 2023-10-02

✨ Highlights

The solver has been renamed and moved to its own repository: resolvo. With the latest changes to the python bindings you can now download repodata and solve environments! Still no official release of the bindings though, but getting closer every day.

📃 Details

Added

• add initial nushell support by @wolfv in #271

Changed

• the solver has been extracted in its own package: resolvo by @baszalmstra in #349 & #350

Fixed

• Change solver implementation doc comment by @nichmor in #352

🐍 Python

• add more py-rattler types by @Wackyator in #348
• add fetch repo data to py-rattler by @Wackyator in #334
• use SparseRepoData in fetch_repo_data by @Wackyator in #359
• add solver by @Wackyator in #361

🤗 New Contributors

• @nichmor made their first contribution in #352

Full Changelog: https://github.com/conda/rattler/compare/v0.9.0...v0.10.0

[0.9.0] - 2023-09-22

✨ Highlights

This is a pretty substantial release which includes many refactors to the solver (which we will pull out of this repository at some point), initial work on Python bindings, and many many fixes.

📃 Details

Added

• pixi project to make contributing easier by @YeungOnion in #283, #342
• make rattler-package-streaming compile with wasm by @wolfv in #287
• implement base_url cep by @baszalmstra in #322
• use emscripten-wasm32 and wasi-wasm32 by @wolfv in #333
• add build_spec module by @YeungOnion in #340, #346

Changed

• use normalized package names where applicable by @baszalmstra in #285
• new StrictVersion type for VersionSpec ranges. by @tdejager in #296
• refactored ratter_libsolv_rs to be conda agnostic by @tdejager & @baszalmstra in #316, #309, #317, #320, #319, #323, #324, #328, #325, #326, #335, #336, #338, #343, #337
• feat: allow disabling jlap by @baszalmstra in #327
• test: added job to check for lfs links by @tdejager in #331
• hide implementation detail, version_spec::Constraint by @YeungOnion in #341

Fixed

• typo in solver error message by @baszalmstra in #284
• expose ParseMatchSpecError in rattler_conda_types by @Wackyator in #286
• use nvidia-smi on musl targets to detect Cuda by @baszalmstra in #290
• typo in snap file by @wolfv in #291
• Version::is_dev returning false for dev version (fix #289) by @Wackyator in #293
• workaround for PIP_REQUIRE_VIRTUALENV env variable by @tusharsadhwani in #294
• ensure consistent sorting of locked packages by @baszalmstra in #295
• updates to NamelessMatchSpec to allow deserializing by @travishathaway in #299
• update all dependencies and fix chrono deprecation by @wolfv in #302
• shell improvements for powershell env-var escaping and xonsh detection by @wolfv in #307
• also export strict version by @wolfv in #312
• make FetchRepoDataOptions clonable by @Wackyator in #321
• bump json-patch 1.1.0 to fix stack overflow by @baszalmstra in #332
• emscripten is a unix variant by @wolfv in #339
• authentication fallback storage location by @ruben-arts in #347

🐍 Python

Although this release doesn't include a formal release of the python bindings yet, a lot of work has been done to work towards a first version.

• initial version of rattler python bindings by @baszalmstra in #279
• bind Version, MatchSpec, NamelessMatchSpec by @Wackyator in #292
• add more tests, hash and repr changes by @baszalmstra in #300
• add license by @Wackyator in #301
• bind channel types to py-rattler by @wolfv in #313
• bind everything necessary for shell activation by @wolfv in #298
• add mypy checks by @baszalmstra in #314
• bind AuthenticatedClient by @Wackyator in #315
• add py.typed file by @baszalmstra in #318
• bind VersionWithSource by @Wackyator in #304

🤗 New Contributors

• @Wackyator made their first contribution in #286
• @YeungOnion made their first contribution in #283
• @tusharsadhwani made their first contribution in #294

To all contributors, thank you for your amazing work on Rattler. This project wouldn't exist without you! 🙏

[0.8.0] - 2023-08-22

Highlights

This release contains bug fixes.

Details

Added

• retry behavior when downloading package archives by @baszalmstra in (#281)

Fixed

• parsing of local versions in Constraints by @baszalmstra in (#280)

[0.7.0] - 2023-08-11

Highlights

This release mostly contains bug fixes.

Details

Added

• Rattler is now also build for Linux aarch64 in CI by @pavelzw in (#272)
• FromStr for ShellEnum by @ruben-arts in (#258)

Changed

• Run activation scripts and capture their output by @baszalmstra in (#239)
• If memory mapping fails during installation the entire file is read instead by @baszalmstra in (#273)
• Constraints parsing to improve performance and error messages by @baszalmstra in (#254)
• Added explicit error in case repodata does not exist on the server by @ruben-arts in (#256)
• Code signing on apple platform now uses codesign binary instead of apple-codesign crate by @wolfv in (#259)

Fixed

• Shell::run_command ends with a newline by @baszalmstra in (#262)
• Formatting of environment variable in fish by @ruben-arts in (#264)
• Suppress stderr and stdout from codesigning by @wolfv in (#265)
• All crates have at least basic documentation by @wolfv in (#268)
• Use default_cache_dir in the rattler binary by @wolfv in (#269)
• Corrupted tar files generated by rattler-package-streaming by @johnhany97 in (#276)
• Superfluous quotes in the content-hash of conda lock files by @baszalmstra in (#277)
• subdir and arch fields when converting RepoDataRecord to a LockedDependency in conda-lock format by @wolfv in (#255)

[0.6.0] - 2023-07-07

Highlights

New rust based solver implementation

This version of rattler includes a new solver implementation! @aochagavia worked hard on porting libsolv to rust and integrating that with rattler_solve. The port performs slightly faster or similar to the original C code and does not contain unsafe code, is well documented, and thread-safe. Our implementation (rattler_libsolv_rs) is specific to solving conda packages by leveraging rattler_conda_types for matching and parsing.

Some performance benchmarks taken on Apple M2 Max.

	libsolv-c	libsolv-rs
python=3.9	7.3734 ms	4.5831 ms
xtensor, xsimd	5.7521 ms	2.7643 ms
tensorflow	654.38 ms	371.59 ms
quetz	1.2577 s	1.3807 s
tensorboard=2.1.1, grpc-cpp=1.39.1	474.76 ms	132.79 ms

Run cargo bench libsolv to check the results on your own machine.

Besides the much improved implementation the new solver also provides much better error messages based on the work from mamba. When a conflict is detected the incompatibilities are analyzed and displayed with a more user-friendly error message.

The following packages are incompatible
|-- asdf can be installed with any of the following options:
    |-- asdf 1.2.3 would require
        |-- C >1, which can be installed with any of the following options:
            |-- C 2.0.0
|-- C 1.0.0 is locked, but another version is required as reported above

rattler-solve has also been refactored to accommodate this change. It is now more easily possible to switch between solvers add runtime by writing functions that are generic on the solver. The solvers now live in a separate module rattler_solve::libsolv_c for the original libsolv C implementation and rattler_solve::libsolv_rs for the rust version. Both solvers can be enabled with feature flags. The default features only select libsolv_c.

Caching of activation scripts

This release contains code to execute an activation script and capture the changes it made to the environment. Caching the result of an activation script can be useful if you need to invoke multiple executables from the same environment.

Details

Added

• Run activation scripts and capture their output by @baszalmstra in (#239)
• Support for sha256 and md5 field in matchspec by @0xbe7a in (#241)
• A rust port of libsolv as an additional solver backend by @aochagavia, @baszalmstra in (#243 & #253)
• Test cases and benchmarks for solver implementations by @baszalmstra in (#250 & #250)
• The ability to add a dependency from python on pip while loading repodata @wolfv in (#238)

Changed

• Completely refactored version parsing by @baszalmstra in (#240)
• Refactored solver interface to allow generic use of solver implementations by @baszalmstra in (#245)
• Also check if credentials stored under wildcard host by @wolfv in (#252)

Fixed

• Compilation issues by @wolfv in (#244)
• Add missing From<VersionWithSource> for Version by @baszalmstra in (#246)
• Optimized libsolv port by removing redundant MatchSpec parsing by @baszalmstra in (#246)
• Optimized libsolv port by caching matching Solvables by @baszalmstra in (#251)

[0.5.0] - 2023-06-30

Highlights

A bug fix release

Details

Added

• More control over how the PATH is altered during activation (#232)

Fixed

• Reconstructing of RepoData from conda lock files for local channels (#231)
• Powershell on Linux (#234)
• Proper parsing of >2.10* as >=2.10 (#237)

[0.4.0] - 2023-06-23

Highlights

A new algorithm was introduced to sort PackageRecords in a topological order based on their dependencies. Sorting in this way provides a deterministic way of sorting packages in the order in which they should be installed to avoid clobbering. The original algorithm was extracted from rattler-server.

Experimental extensions to the conda lock file format have also been introduced to make it possible to completely reproduce the original RepoDataRecords from a lock file.

Fixes were made to the MatchSpec and Version implementation to catch some corner cases and detecting the current shell has become more robust.

Details

Added

• PackageRecord::sort_topologically to perform a topological sort of PackageRecords (#218)
• Experimental fields to be able to reconstruct RepoDataRecord from conda lock files. (#221)
• Methods to manipulate Versions (#229)

Changed

• Refactored shell detection code using $SHELL or parent process name (#219)
• The error message that is thrown when parsing a Platform now includes possible options (#222)
• Completely refactored Version implementation to reduce memory footprint and increase readability (#227)

Fixed

• Issue with parsing matchspecs that contain epochs (#220)
• Zsh activation scripts invoke .sh scripts (#223)
• Detect the proper powershell parent process (#224)

[0.3.0] - 2023-06-15

Highlights

This release contains lots of fixes, small (breaking) changes, and new features. The biggest highlights are:

JLAP support

JLAP is a file format to incrementally update a cached repodata.json without downloading the entire file. This can save a huge amount of bandwidth for large repodatas that change often (like those from conda-forge). If you have a previously cached repodata.json on your system only small JSON patches are downloaded to bring your cache up to date. The format was initially proposed through a CEP and has been available in conda as an experimental feature since 23.3.0.

When using rattler you get JLAP support out of the box. No changes are needed.

Support for local file://

file:// based urls are now supported for all functions that use a Url to download certain data.

rattler_networking

A new crate has been added to facilitate authentication when downloading repodata or packages called rattler_networking.

Details

Added

• Support for detecting more platforms (#135)
• RepoData is now clonable (#138)
• RunExportsJson is now clonable (#169)
• file:// urls are now supported for package extraction functions (#157)
• file:// urls are now supported for repodata fetching (#158)
• Getting started with rattler using micromamba (#163)
• Add Platform::arch function to return the architecture of a given platform (#166)
• Extracted Python style JSON formatting into a separate crate (#163)
• Added feature to use rustls with rattler_package_streaming and rattler_repodata_gateway (#179 & #181)
• Expose version_spec module (#183)
• NamelessMatchSpec a variant of MatchSpec that does not include a package name #185)
• ShellEnum - a dynamic shell type for dynamic discovery #187)
• Exposed the python_entry_point_template function (#190)
• Enable deserializing virtual packages (#198)
• Refactored CI to add macOS arm64 (#201)
• Support for JLAP when downloading repodata (#197 & #214)
• Clone, Debug, PartialEq, Eq implementations for conda lock types (#213)
• rattler_networking to enable accessing repodata.json and packages that require authentication (#191)

Changed

• FileMode is now included with prefix_placeholder is set (#136)
• rattler_digest now re-exports commonly used hash types and typed hashes are now used in more placed (instead of strings) [#137 & #153]
• Use Platform in to detect running operating system (#144)
• paths.json is now serialized in a deterministic fashion (#147)
• Determine the subdir for the platform and arch fields when creating a PackageRecord from an index.json (#145 & #152)
• Activator::activation now returns the new PATH in addition to the script (#151)
• Use properly typed chrono::DateTime<chrono::Utc> for timestamps instead of u64 (#157)
• Made ParseError public and reuse ArchiveType (#167)
• Allow setting timestamps when creating package archive (#171)
• about.json and index.json are now serialized in a deterministic fashion (#180)
• SHA256 and MD5 hashes are computed on the fly when extracting packages (#176
• Change blake2 hash to use blake2b instead of blake2s (#192
• LibSolv error messages are now passed through (#202 & #210)
• VersionTree parsing now uses nom instead of a complex regex (#206
• libc version detection now uses lld --version to properly detect the libc version on the host (#209
• Improved version parse error messages (#211
• Parsing of some complex MatchSpecs (#217

Fixed

• MatchSpec bracket list parsing can now handle quoted values (#157)
• Typos and documentation (#164 & #188)
• Allow downloading of repodata.json to fail in some cases (only noarch is a required subdir) (#174)
• Missing feature when using the sparse index (#182)
• Several small issues or missing functionality (#184)
• Loosened strictness of comparing packages in Transactions (#186
• Missing noarch: generic parsing in links.json (#189
• Ignore trailing .0 in version comparison (#196

[0.2.0] - 2023-03-24

Added

• Construction methods for NoArchType (#130)
• Function to create package record from index.json + size and hashes (#126)
• Serialization for repodata (#124)
• Functions to apply patches to repodata (#127)
• More tests and evict removed packages from repodata (#128)
• First version of package writing functions (#112)

Changed

• Removed dependency on clang-sys during build (#131)

0.1.0 - 2023-03-16

First release