All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
ProgressBartrait and progress bar for package writing by @wolfv (#525)
- improved logging in package validation to include package path by @orhun (#521)
- use resolvo 0.4.0 with better error messages by @baszalmstra (#523)
- allow multiple clobbers per package by @wolfv (#526)
- remove drop-bomb, move empty folder removal to
post_processby @wolfv (#519) - keep in mind python: noarch packages in clobber calculations by @wolfv (#511)
This release contains some big changes to rattler:
Rattler installs packages in parallel but this was at the cost of not being able to resolve files properly that existed in multiple packages. With this release we fixed this issue by creating a consistent clobbering experience. When a file is clobbered (installed by multiple packages) the last package in the topological ordering wins. This information is also recorded in the prefix itself which means that even if packages are added or removed from the environment the order remains consistent.
The AuthenticatedClient has been rewritten by @vlad-ivanov-name.
Instead of having a custom client for network requests we now use the reqwest-middleware crate.
The rattler implementation adds a middleware that handles authentication.
This changes makes it easier to integrate with other crates that use reqwest for network requests, and it allows users to add their own middleware.
The lock-file format has been updated to version 4. Originally our implementation was semi-compatible with conda-lock. We wanted to stay as close as possible to this format because it was already an established standard. However, with version 2 and 3 of the format we started to diverge more and more. We felt like the goals between both formats also started to diverge more and more so with version 4 we decided to completely abandon the conda-lock format and create our own. For more information about the lock-file format and the differences between conda-lock you can read the documentation. Note that all old formats (including the original conda-lock format) can still be parsed by rattler.
- Add
get_windows_launcherfunction by @wolfv (#477) - Expose
get_windows_launcherfunction by @wolfv (#477) - Consistent clobbering & removal of
__pycache__by @wolfv (#437) - Add
name()toChannelby @ruben-arts (#495) - Add timeout parameter to the solver by @wolfv (#499)
- Add a very simple basic test to validate that we can at least parse netrc properly by @mariusvniekerk (#503)
- Allow the full range of compression levels for zstd by @wolfv (#479)
- Make compression conversion functions
pubby @wolfv (#480) - Lock-file v4 by @baszalmstra (#484)
- Convert authenticated client to reqwest middleware by @vlad-ivanov-name (#488)
- Upgrade to latest resolvo main by @tdejager (#497)
- Bump: resolvo 0.3.0 by @baszalmstra (#500)
- Copy over file permissions after reflink by @orhun (#485)
- Fix clippy and deprecation warnings by @wolfv (#490)
- Do not unwrap as much in clobberregistry by @wolfv (#489)
- Fix warning for deref on a double reference by @wolfv (#493)
- Fix self-clobbering when updating a package by @wolfv (#494)
- Fix netrc parsing into BasicAuth by @wolfv (#506)
- @vlad-ivanov-name made their first contribution in conda#482
- Reduce tracing level for reflink by @baszalmstra (#475)
- Add
read_package_filefunction by @wolfv (#472) - implement
CloneforAboutJsonby @0xbe7a (#467) - Allow using
strinHashMaps with aPackageNamekey by @baszalmstra (#468)
- Reflink files to destination if supported (instead of hardlinking) by @baszalmstra (#463)
- Automatic clippy fixes by @wolfv (#470)
- Fix getting credentials from keyring error by @0xbe7a (#474)
- Add ParseMatchSpecError and ParseMatchSpecError tests by @Johnwillliam (#434)
- Add option to force usage of fallback_auth_store by @0xbe7a (#435)
- New crate (rattler-index) with index functionality including python bindings by @BenjaminLowry (#436)
- Add support for netrc files by @mariusvniekerk (#395)
- Renamed
behaviourtobehavior(#428) - Enabled more clippy lints by @baszalmstra (#462)
- Refactor
Version.bump()to accept bumpingmajor/minor/patch/lastby @hadim (#452)
- Default value for
conda_packagesin repodata.json by @BenjaminLowry (#441) - Wildcard expansion for stored credentials of domains by @0xbe7a (#442)
- Use serde default for proper serialization by @ruben-arts (#443)
- Better detection of hardlinks and fallback to copy by @baszalmstra (#461)
- Re-download the repodata cache if is out of sync/corrupt by @orhun (#466)
- Renamed
behaviourtobehavior(#428)
- Recursive look for parent process name (#424)
- Improve repodata fetch errors (#426)
- Use filelock for authentication fallback storage (#427)
- Improved lockfile version mismatch error (#423)
- Experimental support for purls in PackageRecord and derived datastructures (#414)
- Rename
piptopypiin lockfile (#415)
- Expose missing
StringMatcherParseError(#410) - Fix JLAP issue by setting the nominal hash when first downloading repodata (#411)
- Support channel names with slashes (#413)
- fix: make redaction work by using
Fromexplicitly (#408)
- fix: redact tokens from urls in errors (#407)
Adds support for strict priority channel ordering, channel-specific selectors,
- Add strict channel priority option (#385)
- Add lock-file forward compatibility error (#389)
- Add channel priority and channel-specific selectors to solver info (#394)
- Channel in the
MatchSpecstruct changed toChanneltype (#401)
- Expose previous python version information in transaction (#384)
- Avoid use of \ in doctest strings, for ide integration (#387)
- Issue with JLAP using the wrong hash (#390)
- Use the correct channel in the reason for exclude (#397)
- Environment activation for windows (#398)
Lock file support has been moved into its own crate (rattler_lock) and support for pip packages has been added.
- change authentication fallback warnings to debug by @ruben-arts in conda#365
- repodata cache now uses
.info.jsoninstead of.state.jsonby @dholth in conda#377 - lock file now lives in its own crate with pip support by @baszalmstra in conda#378
- Nushell fixes by @wolfv in conda#360
- Construct placeholder string at runtime to work around invalid conda prefix replacement by @baszalmstra in conda#371
- xonsh extension by @ruben-arts in conda#375
- @dholth made their first contribution in conda#377
Full Changelog: https://github.com/mamba-org/rattler/compare/v0.10.0...v0.11.0
The solver has been renamed and moved to its own repository: resolvo. With the latest changes to the python bindings you can now download repodata and solve environments! Still no official release of the bindings though, but getting closer every day.
- add initial nushell support by @wolfv in #271
- Change solver implementation doc comment by @nichmor in #352
- add more py-rattler types by @Wackyator in #348
- add fetch repo data to py-rattler by @Wackyator in #334
- use SparseRepoData in fetch_repo_data by @Wackyator in #359
- add solver by @Wackyator in #361
- @nichmor made their first contribution in #352
Full Changelog: https://github.com/mamba-org/rattler/compare/v0.9.0...v0.10.0
This is a pretty substantial release which includes many refactors to the solver (which we will pull out of this repository at some point), initial work on Python bindings, and many many fixes.
- pixi project to make contributing easier by @YeungOnion in #283, #342
- make rattler-package-streaming compile with wasm by @wolfv in #287
- implement base_url cep by @baszalmstra in #322
- use emscripten-wasm32 and wasi-wasm32 by @wolfv in #333
- add build_spec module by @YeungOnion in #340, #346
- use normalized package names where applicable by @baszalmstra in #285
- new
StrictVersiontype for VersionSpec ranges. by @tdejager in #296 - refactored ratter_libsolv_rs to be conda agnostic by @tdejager & @baszalmstra in #316, #309, #317, #320, #319, #323, #324, #328, #325, #326, #335, #336, #338, #343, #337
- feat: allow disabling jlap by @baszalmstra in #327
- test: added job to check for lfs links by @tdejager in #331
- hide implementation detail, version_spec::Constraint by @YeungOnion in #341
- typo in solver error message by @baszalmstra in #284
- expose ParseMatchSpecError in rattler_conda_types by @Wackyator in #286
- use nvidia-smi on musl targets to detect Cuda by @baszalmstra in #290
- typo in snap file by @wolfv in #291
- Version::is_dev returning false for dev version (fix #289) by @Wackyator in #293
- workaround for
PIP_REQUIRE_VIRTUALENVenv variable by @tusharsadhwani in #294 - ensure consistent sorting of locked packages by @baszalmstra in #295
- updates to
NamelessMatchSpecto allow deserializing by @travishathaway in #299 - update all dependencies and fix chrono deprecation by @wolfv in #302
- shell improvements for powershell env-var escaping and xonsh detection by @wolfv in #307
- also export strict version by @wolfv in #312
- make FetchRepoDataOptions clonable by @Wackyator in #321
- bump json-patch 1.1.0 to fix stack overflow by @baszalmstra in #332
- emscripten is a unix variant by @wolfv in #339
- authentication fallback storage location by @ruben-arts in #347
Although this release doesn't include a formal release of the python bindings yet, a lot of work has been done to work towards a first version.
- initial version of rattler python bindings by @baszalmstra in #279
- bind
Version,MatchSpec,NamelessMatchSpecby @Wackyator in #292 - add more tests, hash and repr changes by @baszalmstra in #300
- add license by @Wackyator in #301
- bind channel types to py-rattler by @wolfv in #313
- bind everything necessary for shell activation by @wolfv in #298
- add mypy checks by @baszalmstra in #314
- bind
AuthenticatedClientby @Wackyator in #315 - add
py.typedfile by @baszalmstra in #318 - bind
VersionWithSourceby @Wackyator in #304
- @Wackyator made their first contribution in #286
- @YeungOnion made their first contribution in #283
- @tusharsadhwani made their first contribution in #294
To all contributors, thank you for your amazing work on Rattler. This project wouldn't exist without you! 🙏
This release contains bug fixes.
- retry behavior when downloading package archives by @baszalmstra in (#281)
- parsing of local versions in
Constraints by @baszalmstra in (#280)
This release mostly contains bug fixes.
- Rattler is now also build for Linux aarch64 in CI by @pavelzw in (#272)
FromStrforShellEnumby @ruben-arts in (#258)
- Run activation scripts and capture their output by @baszalmstra in (#239)
- If memory mapping fails during installation the entire file is read instead by @baszalmstra in (#273)
- Constraints parsing to improve performance and error messages by @baszalmstra in (#254)
- Added explicit error in case repodata does not exist on the server by @ruben-arts in (#256)
- Code signing on apple platform now uses
codesignbinary instead ofapple-codesigncrate by @wolfv in (#259)
Shell::run_commandends with a newline by @baszalmstra in (#262)- Formatting of environment variable in fish by @ruben-arts in (#264)
- Suppress stderr and stdout from codesigning by @wolfv in (#265)
- All crates have at least basic documentation by @wolfv in (#268)
- Use
default_cache_dirin the rattler binary by @wolfv in (#269) - Corrupted tar files generated by rattler-package-streaming by @johnhany97 in (#276)
- Superfluous quotes in the
content-hashof conda lock files by @baszalmstra in (#277) subdirandarchfields when convertingRepoDataRecordto aLockedDependencyin conda-lock format by @wolfv in (#255)
This version of rattler includes a new solver implementation!
@aochagavia worked hard on porting libsolv to rust and integrating that with rattler_solve.
The port performs slightly faster or similar to the original C code and does not contain unsafe code, is well documented, and thread-safe.
Our implementation (rattler_libsolv_rs) is specific to solving conda packages by leveraging rattler_conda_types for matching and parsing.
Some performance benchmarks taken on Apple M2 Max.
| libsolv-c | libsolv-rs | |
|---|---|---|
| python=3.9 | 7.3734 ms | 4.5831 ms |
| xtensor, xsimd | 5.7521 ms | 2.7643 ms |
| tensorflow | 654.38 ms | 371.59 ms |
| quetz | 1.2577 s | 1.3807 s |
| tensorboard=2.1.1, grpc-cpp=1.39.1 | 474.76 ms | 132.79 ms |
Run
cargo bench libsolvto check the results on your own machine.
Besides the much improved implementation the new solver also provides much better error messages based on the work from mamba. When a conflict is detected the incompatibilities are analyzed and displayed with a more user-friendly error message.
The following packages are incompatible
|-- asdf can be installed with any of the following options:
|-- asdf 1.2.3 would require
|-- C >1, which can be installed with any of the following options:
|-- C 2.0.0
|-- C 1.0.0 is locked, but another version is required as reported above
rattler-solve has also been refactored to accommodate this change.
It is now more easily possible to switch between solvers add runtime by writing functions that are generic on the solver.
The solvers now live in a separate module rattler_solve::libsolv_c for the original libsolv C implementation and rattler_solve::libsolv_rs for the rust version.
Both solvers can be enabled with feature flags. The default features only select libsolv_c.
This release contains code to execute an activation script and capture the changes it made to the environment. Caching the result of an activation script can be useful if you need to invoke multiple executables from the same environment.
- Run activation scripts and capture their output by @baszalmstra in (#239)
- Support for sha256 and md5 field in matchspec by @0xbe7a in (#241)
- A rust port of libsolv as an additional solver backend by @aochagavia, @baszalmstra in (#243 & #253)
- Test cases and benchmarks for solver implementations by @baszalmstra in (#250 & #250)
- The ability to add a dependency from
pythononpipwhile loading repodata @wolfv in (#238)
- Completely refactored version parsing by @baszalmstra in (#240)
- Refactored solver interface to allow generic use of solver implementations by @baszalmstra in (#245)
- Also check if credentials stored under wildcard host by @wolfv in (#252)
- Compilation issues by @wolfv in (#244)
- Add missing
From<VersionWithSource>forVersionby @baszalmstra in (#246) - Optimized libsolv port by removing redundant MatchSpec parsing by @baszalmstra in (#246)
- Optimized libsolv port by caching matching Solvables by @baszalmstra in (#251)
A bug fix release
- More control over how the PATH is altered during activation (#232)
- Reconstructing of RepoData from conda lock files for local channels (#231)
- Powershell on Linux (#234)
- Proper parsing of
>2.10*as>=2.10(#237)
A new algorithm was introduced to sort PackageRecords in a topological order based on their dependencies.
Sorting in this way provides a deterministic way of sorting packages in the order in which they should be installed to avoid clobbering.
The original algorithm was extracted from rattler-server.
Experimental extensions to the conda lock file format have also been introduced to make it possible to completely reproduce the original RepoDataRecords from a lock file.
Fixes were made to the MatchSpec and Version implementation to catch some corner cases and detecting the current shell has become more robust.
PackageRecord::sort_topologicallyto perform a topological sort ofPackageRecords (#218)- Experimental fields to be able to reconstruct
RepoDataRecordfrom conda lock files. (#221) - Methods to manipulate
Versions (#229)
- Refactored shell detection code using
$SHELLor parent process name (#219) - The error message that is thrown when parsing a
Platformnow includes possible options (#222) - Completely refactored
Versionimplementation to reduce memory footprint and increase readability (#227)
- Issue with parsing matchspecs that contain epochs (#220)
- Zsh activation scripts invoke .sh scripts (#223)
- Detect the proper powershell parent process (#224)
This release contains lots of fixes, small (breaking) changes, and new features. The biggest highlights are:
JLAP is a file format to incrementally update a cached repodata.json without downloading the entire file.
This can save a huge amount of bandwidth for large repodatas that change often (like those from conda-forge).
If you have a previously cached repodata.json on your system only small JSON patches are downloaded to bring your cache up to date.
The format was initially proposed through a CEP and has been available in conda as an experimental feature since 23.3.0.
When using rattler you get JLAP support out of the box. No changes are needed.
file:// based urls are now supported for all functions that use a Url to download certain data.
A new crate has been added to facilitate authentication when downloading repodata or packages called rattler_networking.
- Support for detecting more platforms (#135)
RepoDatais now clonable (#138)RunExportsJsonis now clonable (#169)file://urls are now supported for package extraction functions (#157)file://urls are now supported for repodata fetching (#158)- Getting started with rattler using micromamba (#163)
- Add
Platform::archfunction to return the architecture of a given platform (#166) - Extracted Python style JSON formatting into a separate crate (#163)
- Added feature to use
rustlswithrattler_package_streamingandrattler_repodata_gateway(#179 & #181) - Expose
version_specmodule (#183) NamelessMatchSpeca variant ofMatchSpecthat does not include a package name #185)ShellEnum- a dynamic shell type for dynamic discovery #187)- Exposed the
python_entry_point_templatefunction (#190) - Enable deserializing virtual packages (#198)
- Refactored CI to add macOS arm64 (#201)
- Support for JLAP when downloading repodata (#197 & #214)
Clone,Debug,PartialEq,Eqimplementations for conda lock types (#213)rattler_networkingto enable accessingrepodata.jsonand packages that require authentication (#191)
FileModeis now included withprefix_placeholderis set (#136)rattler_digestnow re-exports commonly used hash types and typed hashes are now used in more placed (instead of strings) [#137 & #153]- Use
Platformin to detect running operating system (#144) paths.jsonis now serialized in a deterministic fashion (#147)- Determine the
subdirfor theplatformandarchfields when creating aPackageRecordfrom anindex.json(#145 & #152) Activator::activationnow returns the newPATHin addition to the script (#151)- Use properly typed
chrono::DateTime<chrono::Utc>for timestamps instead ofu64(#157) - Made
ParseErrorpublic and reuseArchiveType(#167) - Allow setting timestamps when creating package archive (#171)
about.jsonandindex.jsonare now serialized in a deterministic fashion (#180)- SHA256 and MD5 hashes are computed on the fly when extracting packages (#176
- Change blake2 hash to use blake2b instead of blake2s (#192
- LibSolv error messages are now passed through (#202 & #210)
VersionTreeparsing now usesnominstead of a complex regex (#206libcversion detection now useslld --versionto properly detect the libc version on the host (#209- Improved version parse error messages (#211
- Parsing of some complex MatchSpecs (#217
- MatchSpec bracket list parsing can now handle quoted values (#157)
- Typos and documentation (#164 & #188)
- Allow downloading of repodata.json to fail in some cases (only noarch is a required subdir) (#174)
- Missing feature when using the sparse index (#182)
- Several small issues or missing functionality (#184)
- Loosened strictness of comparing packages in
Transactions (#186 - Missing
noarch: genericparsing inlinks.json(#189 - Ignore trailing .0 in version comparison (#196
- Construction methods for NoArchType (#130)
- Function to create package record from index.json + size and hashes (#126)
- Serialization for repodata (#124)
- Functions to apply patches to repodata (#127)
- More tests and evict removed packages from repodata (#128)
- First version of package writing functions (#112)
- Removed dependency on
clang-sysduring build (#131)
0.1.0 - 2023-03-16
First release