v0.44.0

Changelog

Garbage Collection! Set retention rules for deleted objects. See issue: #1932

v0.43.0

Note

This release requires running database migration.
The lakeFS service will not run if the migration version isn't compatible with the binary.
Before running the new version you will require to run migrate, with the new version:

$ lakefs migrate up

Changelog

• Server-side support for the upcoming garbage collection functionality (#2069)
• Add lakeFS version to the UI (#2088)
• Update lakectl branch create output to include branch name (#2130)
• Warn about using "local" adapter for use in production (#2159)
• Explicit handling of mismatch in block store config and repository namespace according to a user setting (#2126)
• Initial database connect retry (#2131)
• Web UI Vite build tool upgrade to v2.3.7 (#2108)

Bugfixes

• Fix lakectl read environment variables in absence of lakectl.yml config file, this bugfix unlocks the Kubeflow-lakeFS integration (#2143)
• Add UI error reporting on action failure during a commit (#2120)
• Reset the state of the delete repository dialog when it is dismissed (#2117)

v0.42.0

Changelog

Features

• Support common prefix listing of diffs (uncommitted, commit view, compare view) (#2051)
• Export using spark-submit (#2036)

Bugfixes

• Make authorization errors readable in the UI (#2056)
• Verify paths used in the local block adapter are under its base dir (#2003)
• Fix dump refs JSON output (#2040)
• Fix adapter qualified key format to not use path.Join (#1994)

Breaking Changes

• Object listing API: previously, not specifying a delimiter would default to using "/". The default is now "" (empty string)
• The /config API endpoint that returns storage configuration (used mainly in lakeFS UI), is now under /config/storage
• Accessing the /config/storage endpoint now requires the fs:ReadConfig permission instead of auth:ReadConfig

v0.41.1

Changelog

36eb6ae Bugfix/parquet inventory failure (#1979)
4c9b404 Filesystem: non-atomic files rename (#1972)
37a52ce Fix unexpected merge conflicts bug (#1958)
13de272 Short timeout on fetch AWS account ID (#1987)
2a55322 Support Spark SQL tables in metastore copy (#1997)
4589933 gateway on get without version should return after error (#1990)
7307f5a hadoopfs listFiles (#1922)
7226588 lakectl control logging (#1954)
931be4a lakectl log copy and update metastore operation information (#1986)
4bfead3 use a worker pool to ingest entries faster (#1964)

v0.41.0

Changelog

2e2e005 ingest cmd Importing data from an object store without actually copying it (#1864)
851f02a Don't upload chunks smaller than 8192 bytes while streaming (#1885)
1a81228 Fix multipart upload failure in Azure (#1896)
9ebab7a Fix broken Clipboard copy button (#1904)
984cc68 lakeFS filesystem add "create" method (#1907)
5996edf lakeFS filesystem add "open" method (#1895)
7c22cd5 Add Java OpenAPI client up-to-date validation (#1870)
4ea6571 Add conflict response and use it when creating an existing resource (#1900)
0fad6a9 Use React with Vite (#1874)
ad188b6 Implement lakefsFS delete method (#1920)
ca96271 Mark operations unsupported by LakeFS filesystem (#1881)
f592298 Publish Java API client to Sonatype repository (#1869)
be53f58 Remove unsupported reset-to-commit option (#1946)

v0.40.3

Changelog

• New Java and Scala API client (#1837)
• Metastore operations between Glue and Hive (#1838)
• Fix bug where unchanged files appear as changed (#1841)
• Upgrade NodeJS (#1801)
• Added put-if-absent operation (#1823)

v0.40.2

Changelog

• New repository default branch from master to main (#1800)
• Fix auth middleware - check only required security providers (#1795)
• Entry's identity to not contain physical address path (#1805)
• Fix logger broken caller and function detection (#1807)

v0.40.1

Changelog

4206fcd UI: missing setup page (#1793)
0505c79 UI: use first parent for diff of merge commit (#1787)

v0.40.0

Changelog

This is a big release for lakeFS with many notable improvements.

Some of these are breaking changes. It's always a tough decision to introduce a change that isn't backwards compatible,
but we felt that at this stage they represent a significant enough benefit to be worth it.

Going forward, our goal is to make as few of those as possible, as we near a 1.0.0 release.

Here are the most notable changes:

lakeFS is now OpenAPI 3.0 compliant ✨

The lakeFS API has been migrated from OpenAPI 2.0 to OpenAPI 3.0.

OpenAPI 3.0 includes many improvements over the previous version: Cookie based authentication, reusable query parameters, better JSON Schema support and more.

While homegrown clients that simply use the lakeFS API as a REST inteface will continue to work,
client that relied on OpenAPI 2.0 specific behaviors will stop working.

This includes the previously recommended bravado based client for Python. For that reason, we're also releasing an officially supported Python client:

lakeFS now ships with a native Python client ✨

It's now as simple as:

$ pip install lakefs-client~=0.40.0

And then:

import lakefs_client
from lakefs_client.client import LakeFSClient

lakefs = LakeFSClient(lakefs_client.Configuration(
    username='AKIAIOSFODNN7EXAMPLE', 
    password='...', 
    host='http://lakefs.example.com'))
    
lakefs.branches.list_branches(repository='my-repo')  # Or any other API action

This client is officially supported and distributed by the lakeFS team, and will be released in conjunction with lakeFS releases, so it should always align in capabilities with the latest lakeFS versions.

For more information, see the Python Client Documentation.

Native Spark client, allowing to export a commit (or set of commits) to another object store ✨

Using Apache Spark, lakeFS users can now quickly export the contents of a branch to an external location (say, S3 bucket). Exporting committed data will be parallelized using Spark workers to support copying millions of objects in minutes.

This is the first feature released based on lakeFS' Spark integration (soon to be followed by data retention for stale objects), and a native lakefs:// filesystem support for Spark).

For more information, see the Export Job configuration Documentation.

lakeFS standardized URIs ✨

The lakeFS CLI now supports a standardized URI in the form: lakefs://<repository>/<ref>/<path>.
Additionally, the CLI now allows setting a $LAKECTL_BASE_URI environment variable that, if set, will prefix any relative URI used.

For example, instead of:

$ lakectl diff lakefs://my-repository/my-branch lakefs://my-repository/main
$ lakectl fs ls lakefs://my-repository/my-branch/path/

It's now possible to simply do:

$ export LAKECTL_BASE_URI="lakefs://my-repository/"
$ lakectl diff my-branch main
$ lakectl fs ls mybranch/path/

For more information, see the CLI Command Reference Documentation.

Complete UI Overhaul

Making it faster, more responsive and contains many improvements to pagination, commit browsing and action views.

Full Feature list

• [UI] Complete UI overhaul 💅 (#1766)
• [Spark] Spark client that allows exporting from lakeFS to an object store ✨ (#1658)
• [Metastore] Support metastore copy between two different hive metastores ✨ (#1704)
• [API Gateway] BREAKING: Migrated to OpenAPI 3.0 💣 (#1667)
• [Python SDK] Native lakeFS Python Client ✨ (#1725)
• [Graveler] BREAKING: commit parents order for merge-commits are now [destination, source] instead of [source, destination] 💣 (#1754)
• [CLI] BREAKING: lakefs:// URIs are now standard, replacing @ with / to denote ref 💣 (#1717)
• [CLI] $LAKECTL_BASE_URI prefixes all lakectl URIs for more a human-friendly CLI 🥰 (#1717)
• [CLI] Support non-seekable stdin (- arg) in "fs upload" command 🥰 (#1672)

Bug Fixes

• [S3 Gateway] Avoid logging v2 sigs on failure 🔒 (#1679)
• [Graveler] Limit length of Graveler serialization 🐞 (#1682)
• [Graveler] Fix merge large changes performance (#1652)
• [S3 Gateway] Handle no path for delete objects in gateway 🐞 (#1708)
• [API Gateway] API merge message is optional 🐞 (#1710)
• [API Gateway] Fix auth pagination 🐞 (#1755)
• [API Gateway] List repository actions should not check branch existence 🐞 (#1743)

As always, we hang around at #help on the lakeFS Slack to assist and answer questions!

v0.33.1

Changelog

b253f81 Actions tab UI (hook runs) (#1564)
cd8dbce Adding arm64 to our release binaries (#1585)
7e227b3 Convert merge errors to bad requests (#1555)
9b3ac8e Special warning before delete import branch (#1582)
5b31659 Upgrade to Go 1.16.2 (#1619)
c68123e batch hot paths for a very short duration (#1618)
e053cd3 gateway requests reaching ui handler to fail gracefully (#1578)
759baa8 lakectl validate action file (#1601)