Add documentation for standalone (sparkless) GC #8307
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -33,16 +33,16 @@ Standalone GC is a limited version of the Spark-backed GC that runs without any | |||||
|
|
||||||
| 1. Except for the [Lab tests](./standalone-gc.md#lab-tests) performed, there are no further guarantees about the performance profile of the Standalone GC. | ||||||
| 2. Horizontal scale is not supported - Only a single instance of `lakefs-sgc` can operate at a time on a given repository. | ||||||
| 3. Standalone GC only marks objects and does not delete them - Equivalent to the GC's [mark only mode]({% link howto/garbage-collection/gc.md %}#mark-only-mode). \ | ||||||
| More about that in the [Get the List of Objects Marked for Deletion](./standalone-gc.md#get-the-list-of-objects-marked-for-deletion) section. | ||||||
|
|
||||||
| ### Lab tests | ||||||
|
|
||||||
| Repository spec: | ||||||
|
|
||||||
| - 100k objects | ||||||
| - 250 commits | ||||||
| - 100 branches | ||||||
|
|
||||||
| Machine spec: | ||||||
| - 4GiB RAM | ||||||
|
|
@@ -51,7 +51,7 @@ Machine spec: | |||||
| In this setup, we measured: | ||||||
|
|
||||||
| - Time: < 5m | ||||||
| - Disk space: 123MB | ||||||
|
|
||||||
| ## Installation | ||||||
|
|
||||||
|
|
@@ -61,7 +61,7 @@ If not, contact us at [support@treeverse.io](mailto:support@treeverse.io). | |||||
|
|
||||||
| ### Step 2: Login to Dockerhub with this token | ||||||
| ```bash | ||||||
| docker login -u <token> | ||||||
| ``` | ||||||
|
|
||||||
| ### Step 3: Download the docker image | ||||||
|
|
@@ -72,6 +72,66 @@ docker pull treeverse/lakefs-sgc:<tag> | |||||
|
|
||||||
| ## Usage | ||||||
|
There was a problem hiding this comment. Can you please add these two steps here: There was a problem hiding this comment.
I already added an example - take a look at "Example - docker run command"
Done - in the new "Output" section. Not sure WDYM by a "CTA", I just added a sentence explaining that the user should read the report and delete manually. There was a problem hiding this comment.
Update: I added a dedicated section for "Deleting marked objects" with the same sentence ^ |
||||||
|
|
||||||
| ### Permissions | ||||||
| To run `lakefs-sgc`, you'll need AWS and LakeFS users, with the following permissions: | ||||||
| #### AWS | ||||||
| The minimal required permissions on AWS are: | ||||||
| ```json | ||||||
| { | ||||||
| "Version": "2012-10-17", | ||||||
| "Statement": [ | ||||||
| { | ||||||
| "Effect": "Allow", | ||||||
| "Action": [ | ||||||
| "s3:PutObject", | ||||||
| "s3:GetObject" | ||||||
| ], | ||||||
| "Resource": [ | ||||||
| "arn:aws:s3:::<bucket>/*" | ||||||
|
There was a problem hiding this comment. Too permesive. Should be this, no?
Suggested change
There was a problem hiding this comment. Can't use only |
||||||
| ] | ||||||
| }, | ||||||
| { | ||||||
| "Effect": "Allow", | ||||||
| "Action": [ | ||||||
| "s3:ListBucket" | ||||||
| ], | ||||||
| "Resource": [ | ||||||
| "arn:aws:s3:::<bucket>" | ||||||
| ] | ||||||
| }, | ||||||
| { | ||||||
| "Effect": "Allow", | ||||||
| "Action": [ | ||||||
| "s3:ListAllMyBuckets" | ||||||
| ], | ||||||
| "Resource": [ | ||||||
| "arn:aws:s3:::*" | ||||||
| ] | ||||||
| } | ||||||
| ] | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| #### LakeFS | ||||||
| The minimal required permissions on LakeFS are: | ||||||
| ```json | ||||||
| { | ||||||
| "statement": [ | ||||||
| { | ||||||
| "action": [ | ||||||
| "fs:ReadConfig", | ||||||
| "fs:ReadRepository", | ||||||
| "retention:PrepareGarbageCollectionCommits", | ||||||
| "retention:PrepareGarbageCollectionUncommitted", | ||||||
| "fs:ListObjects", | ||||||
| "fs:ReadConfig" | ||||||
| ], | ||||||
| "effect": "allow", | ||||||
| "resource": "arn:lakefs:fs:::repository/<repository>" | ||||||
| } | ||||||
| ] | ||||||
| } | ||||||
| ``` | ||||||
| ### AWS Credentials | ||||||
| Currently, `lakefs-sgc` does not provide an option to explicitly set AWS credentials. It relies on the hosting machine | ||||||
| to be set up correctly, and reads the AWS credentials from the machine. | ||||||
|
|
@@ -86,16 +146,16 @@ An example setup for working with MinIO: | |||||
| ``` | ||||||
| [profile minio] | ||||||
| region = us-east-1 | ||||||
| endpoint_url = <MinIO URL> | ||||||
| s3 = | ||||||
| signature_version = s3v4 | ||||||
| ``` | ||||||
|
|
||||||
| 2. Add an access and secret keys to your `~/.aws/credentials` file: | ||||||
| ``` | ||||||
| [minio] | ||||||
| aws_access_key_id = <MinIO access key> | ||||||
| aws_secret_access_key = <MinIO secret key> | ||||||
| ``` | ||||||
| 3. Run the `lakefs-sgc` docker image and pass it the `minio` profile - see [example](./standalone-gc.md#mounting-the-aws-directory) below. | ||||||
|
|
||||||
|
|
@@ -110,15 +170,10 @@ The following configuration keys are available: | |||||
| | `cache_dir` | Directory to use for caching data during run | ~/.lakefs-sgc/data | string | | ||||||
| | `aws.max_page_size` | Max number of items per page when listing objects in AWS | 1000 | number | | ||||||
| | `aws.s3.addressing_path_style` | Whether or not to use [path-style](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access) when reading objects from AWS | true | boolean | | ||||||
| | `lakefs.endpoint_url` | The URL to the lakeFS installation - should end with `/api/v1` | NOT SET | URL | | ||||||
| | `lakefs.access_key_id` | Access key to the lakeFS installation | NOT SET | string | | ||||||
| | `lakefs.secret_access_key` | Secret access key to the lakeFS installation | NOT SET | string | | ||||||
|
|
||||||
| These keys can be provided in the following ways: | ||||||
| 1. Config file: Create a YAML file with the keys, each `.` is a new nesting level. \ | ||||||
| For example, `logging.level` will be: | ||||||
|
|
@@ -139,8 +194,8 @@ logging: | |||||
| level: debug | ||||||
| lakefs: | ||||||
| endpoint_url: https://your.url/api/v1 | ||||||
| access_key_id: <lakeFS access key> | ||||||
| secret_access_key: <lakeFS secret key> | ||||||
| ``` | ||||||
|
|
||||||
| ### Command line reference | ||||||
|
|
@@ -159,7 +214,7 @@ Flags: | |||||
| - `--parallelism`: number of parallel downloads for metadataDir (default 10) | ||||||
| - `--presign`: use pre-signed URLs when downloading/uploading data (recommended) (default true) | ||||||
|
|
||||||
| ### How to Run Standalone GC | ||||||
|
|
||||||
| #### Directly passing in credentials parsed from `~/.aws/credentials` | ||||||
|
|
||||||
|
|
@@ -169,9 +224,9 @@ docker run \ | |||||
| -e AWS_SESSION_TOKEN="$(grep 'aws_session_token' ~/.aws/credentials | awk -F' = ' '{print $2}')" \ | ||||||
| -e AWS_ACCESS_KEY_ID="$(grep 'aws_access_key_id' ~/.aws/credentials | awk -F' = ' '{print $2}')" \ | ||||||
| -e AWS_SECRET_ACCESS_KEY="$(grep 'aws_secret_access_key' ~/.aws/credentials | awk -F' = ' '{print $2}')" \ | ||||||
| -e LAKEFS_SGC_LAKEFS_ENDPOINT_URL=<lakefs endpoint URL> \ | ||||||
| -e LAKEFS_SGC_LAKEFS_ACCESS_KEY_ID=<lakefs accesss key> \ | ||||||
| -e LAKEFS_SGC_LAKEFS_SECRET_ACCESS_KEY=<lakefs secret key> \ | ||||||
| -e LAKEFS_SGC_LOGGING_LEVEL=debug \ | ||||||
| treeverse/lakefs-sgc:<tag> run <repository> | ||||||
| ``` | ||||||
|
|
@@ -191,14 +246,14 @@ docker run \ | |||||
| --network=host \ | ||||||
| -v ~/.aws:/home/lakefs-sgc/.aws \ | ||||||
| -e AWS_REGION=us-east-1 \ | ||||||
| -e AWS_PROFILE=<profile> \ | ||||||
| -e LAKEFS_SGC_LAKEFS_ENDPOINT_URL=<lakefs endpoint URL> \ | ||||||
| -e LAKEFS_SGC_LAKEFS_ACCESS_KEY_ID=<lakefs accesss key> \ | ||||||
| -e LAKEFS_SGC_LAKEFS_SECRET_ACCESS_KEY=<lakefs secret key> \ | ||||||
| -e LAKEFS_SGC_LOGGING_LEVEL=debug \ | ||||||
| treeverse/lakefs-sgc:<tag> run <repository> | ||||||
| ``` | ||||||
| ### Get the List of Objects Marked for Deletion | ||||||
| `lakefs-sgc` will write its reports to `<REPOSITORY_STORAGE_NAMESPACE>/_lakefs/retention/gc/reports/<RUN_ID>/`. \ | ||||||
| _RUN_ID_ is generated during runtime by the Standalone GC. You can find it in the logs: | ||||||
| ``` | ||||||
|
|
@@ -207,7 +262,7 @@ _RUN_ID_ is generated during runtime by the Standalone GC. You can find it in th | |||||
|
|
||||||
| In this prefix, you'll find 2 objects: | ||||||
| - `deleted.csv` - Containing all marked objects in a CSV containing one `address` column. Example: | ||||||
|
There was a problem hiding this comment. Not a docs question, but why this file called deleted if it contains objects that are marked for deletion? There was a problem hiding this comment. It's aligned with the GC's output |
||||||
| ``` | ||||||
| address | ||||||
| "data/gcnobu7n2efc74lfa5ug/csfnri7n2efc74lfa69g,_e7P9j-1ahTXtofw7tWwJUIhTfL0rEs_dvBrClzc_QE" | ||||||
| "data/gcnobu7n2efc74lfa5ug/csfnri7n2efc74lfa78g,mKZnS-5YbLzmK0pKsGGimdxxBlt8QZzCyw1QeQrFvFE" | ||||||
|
|
@@ -225,15 +280,17 @@ In this prefix, you'll find 2 objects: | |||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| ### Delete marked objects | ||||||
|
|
||||||
| To delete the objects marked by the GC, you'll need to read the `deleted.csv` file, and manually delete each address from AWS. | ||||||
|
|
||||||
| It is recommended to move all the marked objects to a different bucket instead of deleting them directly. | ||||||
|
|
||||||
| Here's an example bash script to perform this operation: | ||||||
| ```bash | ||||||
| # Change these to your correct values | ||||||
| storage_ns=<storage namespace (s3://...)> | ||||||
| output_bucket=<output bucket (s3://...)> | ||||||
| run_id=<GC run id> | ||||||
|
|
||||||
| # Download the CSV file | ||||||
|
|
||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should add a limitation that says that sgc only implements the mark stage without sweeping, and sweep requires user action
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done - added a bullet to "Limitations", and a new "Output" section describing this.