Add documentation for standalone (sparkless) GC #8307
Conversation
yonipeleg33
added
include-changelog
yonipeleg33
changed the title
|
♻️ PR Preview b44a572 has been successfully destroyed since this PR has been closed.
🤖 By surge-preview |
E2E Test Results - DynamoDB Local - Local Block Adapter |
E2E Test Results - Quickstart |
There was a problem hiding this comment.
Thank you, @yonipeleg33, for implementing and documenting this important feature!
I'm blocking adding the required steps and clarifying the MinIO scenario's behavior.
| > Standalone GC is only available for [lakeFS Enterprise]({% link enterprise/index.md %}). | ||
|
|
||
| {: .note .warning } | ||
| > Standalone GC is experimental and offers limited capabilities compared to the [Spark-backed GC]({% link howto/garbage-collection/gc.md %}). Read through the [limitations](./standalone-gc.md#limitations) carefully and use at your own risk. |
There was a problem hiding this comment.
| > Standalone GC is experimental and offers limited capabilities compared to the [Spark-backed GC]({% link howto/garbage-collection/gc.md %}). Read through the [limitations](./standalone-gc.md#limitations) carefully and use at your own risk. | |
| > Standalone GC is experimental and offers limited capabilities compared to the [Spark-backed GC]({% link howto/garbage-collection/gc.md %}). Read through the [limitations](./standalone-gc.md#limitations) carefully before using it. |
"at your own risk" sounds a bit harsh and unreliable
| {: .note .warning } | ||
| > Standalone GC is experimental and offers limited capabilities compared to the [Spark-backed GC]({% link howto/garbage-collection/gc.md %}). Read through the [limitations](./standalone-gc.md#limitations) carefully and use at your own risk. | ||
|
|
||
| ## About |
There was a problem hiding this comment.
nit; add a whitespace after every markdown heading
| Standalone GC is a limited version of the Spark-backed GC that runs without any external dependencies, as a standalone docker image. | ||
|
|
||
| ## Limitations | ||
| 1. Tested in-lab under the following conditions: <TODO: publish acceptance and performance test results once ready>. |
There was a problem hiding this comment.
This section is going to include the scale in which sgc was tested at?
There was a problem hiding this comment.
Added concrete numbers (left the TODO because I'm waiting to run it on the final version, which is not ready yet).
| As an enterprise customer, you should already have a dockerhub token for the `externallakefs` user. | ||
| If not, contact us at ___ (TODO: add mail/whaterver). | ||
|
|
||
| ### 2. Login to docker with this token |
There was a problem hiding this comment.
| ### 2. Login to docker with this token | |
| ### 2. Login to dockerhub with this token |
|
|
||
| ## Installation | ||
|
|
||
| ### 1. Obtain Dockerhub token |
There was a problem hiding this comment.
nit; I prefer using words rather than bulleted lists in markdown headers but its up to you
| ### 1. Obtain Dockerhub token | |
| ### Step 1: Obtain Dockerhub token |
| logging: | ||
| level: <value> # info,debug... | ||
| ``` | ||
| Then, pass it to the program using the `--config path/to/config.yaml` argument. |
There was a problem hiding this comment.
There isn't a default location it expects for? that's ok, just to clarify
There was a problem hiding this comment.
There is, it's mentioned in the "Command line reference" section:
Flags:
-c, --config: config file to use (default is $HOME/.lakefs-sgc.yaml)
| ## Limitations | ||
| 1. Tested in-lab under the following conditions: <TODO: publish acceptance and performance test results once ready>. | ||
| 2. Horizontal scale is not supported - Only a single instance of `lakefs-sgc` can operate at a time on a given repository. | ||
|
|
There was a problem hiding this comment.
We should add a limitation that says that sgc only implements the mark stage without sweeping, and sweep requires user action
There was a problem hiding this comment.
Done - added a bullet to "Limitations", and a new "Output" section describing this.
| docker pull treeverse/lakefs-sgc:tagname | ||
| ``` | ||
|
|
||
| ## Usage |
There was a problem hiding this comment.
Can you please add these two steps here:
3. running the job with example params
4. How to find the output and guidance for how to read it and a CTA to delete the objects manually
There was a problem hiding this comment.
running the job with example params
I already added an example - take a look at "Example - docker run command"
How to find the output and guidance for how to read it and a CTA to delete the objects manually
Done - in the new "Output" section. Not sure WDYM by a "CTA", I just added a sentence explaining that the user should read the report and delete manually.
There was a problem hiding this comment.
Done - in the new "Output" section. Not sure WDYM by a "CTA", I just added a sentence explaining that the user should read the report and delete manually.
Update: I added a dedicated section for "Deleting marked objects" with the same sentence ^
| Currently, `lakefs-sgc` does not provide an option to explicitly set AWS credentials. It relies on the hosting machine | ||
| to be set up correctly, and reads the AWS credentials from the machine. | ||
|
|
||
| This means, you should set up your machine however AWS expects you to set it. \ |
There was a problem hiding this comment.
How configurations work for on-prem users who use Minio?
There was a problem hiding this comment.
Done - added "S3-compatible clients" section and example (cc @itaiad200)
There was a problem hiding this comment.
Thanks @talSofer!
I addressed all your comments other than the Minio question, which I need to get back to you about.
PTAL
| > Standalone GC is only available for [lakeFS Enterprise]({% link enterprise/index.md %}). | ||
|
|
||
| {: .note .warning } | ||
| > Standalone GC is experimental and offers limited capabilities compared to the [Spark-backed GC]({% link howto/garbage-collection/gc.md %}). Read through the [limitations](./standalone-gc.md#limitations) carefully and use at your own risk. |
| {: .note .warning } | ||
| > Standalone GC is experimental and offers limited capabilities compared to the [Spark-backed GC]({% link howto/garbage-collection/gc.md %}). Read through the [limitations](./standalone-gc.md#limitations) carefully and use at your own risk. | ||
|
|
||
| ## About |
| ## Limitations | ||
| 1. Tested in-lab under the following conditions: <TODO: publish acceptance and performance test results once ready>. | ||
| 2. Horizontal scale is not supported - Only a single instance of `lakefs-sgc` can operate at a time on a given repository. | ||
|
|
There was a problem hiding this comment.
Done - added a bullet to "Limitations", and a new "Output" section describing this.
|
|
||
| ## Installation | ||
|
|
||
| ### 1. Obtain Dockerhub token |
| As an enterprise customer, you should already have a dockerhub token for the `externallakefs` user. | ||
| If not, contact us at ___ (TODO: add mail/whaterver). | ||
|
|
||
| ### 2. Login to docker with this token |
| docker pull treeverse/lakefs-sgc:tagname | ||
| ``` | ||
|
|
||
| ## Usage |
There was a problem hiding this comment.
running the job with example params
I already added an example - take a look at "Example - docker run command"
How to find the output and guidance for how to read it and a CTA to delete the objects manually
Done - in the new "Output" section. Not sure WDYM by a "CTA", I just added a sentence explaining that the user should read the report and delete manually.
| logging: | ||
| level: <value> # info,debug... | ||
| ``` | ||
| Then, pass it to the program using the `--config path/to/config.yaml` argument. |
There was a problem hiding this comment.
There is, it's mentioned in the "Command line reference" section:
Flags:
-c, --config: config file to use (default is $HOME/.lakefs-sgc.yaml)
| Standalone GC is a limited version of the Spark-backed GC that runs without any external dependencies, as a standalone docker image. | ||
|
|
||
| ## Limitations | ||
| 1. Tested in-lab under the following conditions: <TODO: publish acceptance and performance test results once ready>. |
There was a problem hiding this comment.
Added concrete numbers (left the TODO because I'm waiting to run it on the final version, which is not ready yet).
|
|
||
| ### Step 1: Obtain Dockerhub token | ||
| As an enterprise customer, you should already have a dockerhub token for the `externallakefs` user. | ||
| If not, contact us at ___ (TODO: add mail/whaterver). |
There was a problem hiding this comment.
@talSofer what should I put here?
|
|
||
| ### Step 1: Obtain Dockerhub token | ||
| As an enterprise customer, you should already have a dockerhub token for the `externallakefs` user. | ||
| If not, contact us at ___ (TODO: add mail/whaterver). |
| `lakefs-sgc run <repository>` | ||
|
|
||
| Flags: | ||
| - `--cache-dir`: directory to cache read files and metadataDir (default is $HOME/.lakefs-sgc/data/) |
There was a problem hiding this comment.
Isn't this a config value? How can it be both?
There was a problem hiding this comment.
It can 🙂
Using Viper, we bind this config key (from env/file) to a run argument as well
There was a problem hiding this comment.
@itaiad200 thanks, awesome review!
PTAL (also @talSofer regarding MinIO)
|
|
||
| ### Step 1: Obtain Dockerhub token | ||
| As an enterprise customer, you should already have a dockerhub token for the `externallakefs` user. | ||
| If not, contact us at ___ (TODO: add mail/whaterver). |
| Currently, `lakefs-sgc` does not provide an option to explicitly set AWS credentials. It relies on the hosting machine | ||
| to be set up correctly, and reads the AWS credentials from the machine. | ||
|
|
||
| This means, you should set up your machine however AWS expects you to set it. \ |
There was a problem hiding this comment.
Done - added "S3-compatible clients" section and example (cc @itaiad200)
| `lakefs-sgc run <repository>` | ||
|
|
||
| Flags: | ||
| - `--cache-dir`: directory to cache read files and metadataDir (default is $HOME/.lakefs-sgc/data/) |
There was a problem hiding this comment.
It can 🙂
Using Viper, we bind this config key (from env/file) to a run argument as well
| @@ -149,41 +170,75 @@ docker run \ | |||
| -e AWS_ACCESS_KEY_ID="$(grep 'aws_access_key_id' ~/.aws/credentials | awk -F' = ' '{print $2}')" \ | |||
| -e AWS_SECRET_ACCESS_KEY="$(grep 'aws_secret_access_key' ~/.aws/credentials | awk -F' = ' '{print $2}')" \ | |||
| -e LAKEFS_SGC_LAKEFS_ENDPOINT_URL=<your lakefs URL> \ | |||
There was a problem hiding this comment.
| -e LAKEFS_SGC_LAKEFS_ENDPOINT_URL=<your lakefs URL> \ | |
| -e LAKEFS_SGC_LAKEFS_ENDPOINT_URL=<lakeFS Endpoint URL> \ |
| -e LAKEFS_SGC_LAKEFS_ACCESS_KEY_ID=<your lakefs accesss key> \ | ||
| -e LAKEFS_SGC_LAKEFS_SECRET_ACCESS_KEY=<your lakefs secret key> \ |
There was a problem hiding this comment.
| -e LAKEFS_SGC_LAKEFS_ACCESS_KEY_ID=<your lakefs accesss key> \ | |
| -e LAKEFS_SGC_LAKEFS_SECRET_ACCESS_KEY=<your lakefs secret key> \ | |
| -e LAKEFS_SGC_LAKEFS_ACCESS_KEY_ID=<lakeFS accesss key> \ | |
| -e LAKEFS_SGC_LAKEFS_SECRET_ACCESS_KEY=<lakeFS secret key> \ |
| -e LAKEFS_SGC_LAKEFS_ACCESS_KEY_ID=<your lakefs accesss key> \ | ||
| -e LAKEFS_SGC_LAKEFS_SECRET_ACCESS_KEY=<your lakefs secret key> \ |
There was a problem hiding this comment.
Did we mention somewhere which lakeFS user this user is?
If I'm a customer who want to use that, I'd wish to give it the minimal permissions possible
There was a problem hiding this comment.
Done - Added a "Permissions" section
| --network=host \ | ||
| -v ~/.aws:/home/lakefs-sgc/.aws \ | ||
| -e AWS_REGION=us-east-1 \ | ||
| -e AWS_PROFILE=<your profile> \ |
There was a problem hiding this comment.
Nit; Here and elsewhere, drop the your prefix
|
|
||
| To delete the objects marked by the GC, you'll need to read the `deleted.csv` file, and manually delete each address from AWS. | ||
|
|
||
| Example bash command to move all the marked objects to a different bucket on S3: |
There was a problem hiding this comment.
I would add a note about playing it safe and moving the objects instead of deleting them
|
|
||
| 1. Except for the [Lab tests](./standalone-gc.md#lab-tests) performed, there are no further guarantees about the performance profile of the Standalone GC. | ||
| 2. Horizontal scale is not supported - Only a single instance of `lakefs-sgc` can operate at a time on a given repository. | ||
| 3. It only marks objects and does not delete them - Equivalent to the GC's [mark only mode]({% link howto/garbage-collection/gc.md %}#mark-only-mode). \ |
There was a problem hiding this comment.
| 3. It only marks objects and does not delete them - Equivalent to the GC's [mark only mode]({% link howto/garbage-collection/gc.md %}#mark-only-mode). \ | |
| 3. Stand-alone GC only marks objects and does not delete them - Equivalent to the GC's [mark only mode]({% link howto/garbage-collection/gc.md %}#mark-only-mode). \ |
| | `cache_dir` | Directory to use for caching data during run | ~/.lakefs-sgc/data | string | | ||
| | `aws.max_page_size` | Max number of items per page when listing objects in AWS | 1000 | number | | ||
| | `aws.s3.addressing_path_style` | Whether or not to use [path-style](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access) when reading objects from AWS | true | boolean | | ||
| | `objects_min_age`* | Ignore any object that is last modified within this time frame ("cutoff time") | "6h" | duration | |
There was a problem hiding this comment.
Why do we need this if we have retention policy? and if it is risky to change it, why it is configurable?
There was a problem hiding this comment.
Removed.
It's on top of the retention policy, this configuration exists in the GC as well.
But maybe it's worth not documenting it, to prevent mishaps...
| ``` | ||
|
|
||
| In this prefix, you'll find 2 objects: | ||
| - `deleted.csv` - Containing all marked objects in a CSV containing one `address` column. Example: |
There was a problem hiding this comment.
Not a docs question, but why this file called deleted if it contains objects that are marked for deletion?
There was a problem hiding this comment.
It's aligned with the GC's output
(cc @itaiad200 @Jonathan-Rosenberg - right?)
| - `--parallelism`: number of parallel downloads for metadataDir (default 10) | ||
| - `--presign`: use pre-signed URLs when downloading/uploading data (recommended) (default true) | ||
|
|
||
| ### Example run commands |
There was a problem hiding this comment.
| ### Example run commands | |
| ## How to Run Standalone GC | |
| ### Run Commands |
| -e LAKEFS_SGC_LOGGING_LEVEL=debug \ | ||
| treeverse/lakefs-sgc:<tag> run <repository> | ||
| ``` | ||
| ### Output |
There was a problem hiding this comment.
| ### Output | |
| ### Get the List of Objects Marked for Deletion | |
| The output of an SGC job includes the list of objects marked for deletion. it is located at... |
| } | ||
| ``` | ||
|
|
||
| ### Deleting marked objects |
There was a problem hiding this comment.
| ### Deleting marked objects | |
| ### Delete marked objects |
There was a problem hiding this comment.
Thanks @itaiad200 and @talSofer!
Since you approved I'll not ask for your re-review, but you're welcome to take another look
| @@ -149,41 +170,75 @@ docker run \ | |||
| -e AWS_ACCESS_KEY_ID="$(grep 'aws_access_key_id' ~/.aws/credentials | awk -F' = ' '{print $2}')" \ | |||
| -e AWS_SECRET_ACCESS_KEY="$(grep 'aws_secret_access_key' ~/.aws/credentials | awk -F' = ' '{print $2}')" \ | |||
| -e LAKEFS_SGC_LAKEFS_ENDPOINT_URL=<your lakefs URL> \ | |||
| -e LAKEFS_SGC_LAKEFS_ACCESS_KEY_ID=<your lakefs accesss key> \ | ||
| -e LAKEFS_SGC_LAKEFS_SECRET_ACCESS_KEY=<your lakefs secret key> \ |
| -e LAKEFS_SGC_LAKEFS_ACCESS_KEY_ID=<your lakefs accesss key> \ | ||
| -e LAKEFS_SGC_LAKEFS_SECRET_ACCESS_KEY=<your lakefs secret key> \ |
There was a problem hiding this comment.
Done - Added a "Permissions" section
| --network=host \ | ||
| -v ~/.aws:/home/lakefs-sgc/.aws \ | ||
| -e AWS_REGION=us-east-1 \ | ||
| -e AWS_PROFILE=<your profile> \ |
| -e LAKEFS_SGC_LOGGING_LEVEL=debug \ | ||
| treeverse/lakefs-sgc:<tag> run <repository> | ||
| ``` | ||
| ### Output |
| } | ||
| ``` | ||
|
|
||
| ### Deleting marked objects |
|
|
||
| To delete the objects marked by the GC, you'll need to read the `deleted.csv` file, and manually delete each address from AWS. | ||
|
|
||
| Example bash command to move all the marked objects to a different bucket on S3: |
|
|
||
| 1. Except for the [Lab tests](./standalone-gc.md#lab-tests) performed, there are no further guarantees about the performance profile of the Standalone GC. | ||
| 2. Horizontal scale is not supported - Only a single instance of `lakefs-sgc` can operate at a time on a given repository. | ||
| 3. It only marks objects and does not delete them - Equivalent to the GC's [mark only mode]({% link howto/garbage-collection/gc.md %}#mark-only-mode). \ |
| | `cache_dir` | Directory to use for caching data during run | ~/.lakefs-sgc/data | string | | ||
| | `aws.max_page_size` | Max number of items per page when listing objects in AWS | 1000 | number | | ||
| | `aws.s3.addressing_path_style` | Whether or not to use [path-style](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access) when reading objects from AWS | true | boolean | | ||
| | `objects_min_age`* | Ignore any object that is last modified within this time frame ("cutoff time") | "6h" | duration | |
There was a problem hiding this comment.
Removed.
It's on top of the retention policy, this configuration exists in the GC as well.
But maybe it's worth not documenting it, to prevent mishaps...
| - `--parallelism`: number of parallel downloads for metadataDir (default 10) | ||
| - `--presign`: use pre-signed URLs when downloading/uploading data (recommended) (default true) | ||
|
|
||
| ### Example run commands |
| "s3:GetObject" | ||
| ], | ||
| "Resource": [ | ||
| "arn:aws:s3:::<bucket>/*" |
There was a problem hiding this comment.
Too permesive. Should be this, no?
| "arn:aws:s3:::<bucket>/*" | |
| "arn:aws:s3:::<storage_namespace>/_lakefs/*" |
There was a problem hiding this comment.
Can't use only _lakefs as it needs access to the _data prefix as well.
But you're right that it doesn't need permissions for the entire bucket, only the storage namespace prefix.
Changed accordingly.
Closes #8306