Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update objects timestamps on cache hit #542

Merged
merged 4 commits into from
Apr 25, 2022

Conversation

PycKamil
Copy link
Contributor

In this PR I would like to introduce option to leverage S3 storage lifecycle to automatically remove not used objects after specified amount of time.
We can configure S3 bucket to remove objects after certain amount of time from its modification date. To update modification date, on each cache hit we need to copy object in place - that's limitation of s3. We can now pass s3.update_timestamps option in config to use it.

Copy link
Collaborator

@mostynb mostynb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thanks for the PR.

config/config.go Show resolved Hide resolved
Object: object,
}

_, err := c.mcore.ComposeObject(context.Background(), dst, src)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that the copying is done server side- nice. Are there any downsides to having this always on?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question - I don't see any downsides in terms of performance or storage usage but I'm running self hosted s3 storage. As I understand AWS s3 pricing documentation there is some cost for coping object - you pay the S3 charges for storage in the selected destination S3 storage classes, for the primary copy, for replication PUT request. So making this turn on by default could potentially generate costs for people using AWS so maybe it's better to leave it as opt-in?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, let's make it opt-in to be on the safe side.

…it is trying to copy an object to itself without changing the object’s metadata` error
@mostynb mostynb merged commit 6078764 into buchgr:master Apr 25, 2022
@mostynb
Copy link
Collaborator

mostynb commented Apr 25, 2022

Thanks for the contribution- merged.

@PycKamil PycKamil deleted the s3_compose_objects-on-cache-hit branch September 13, 2024 08:46
@PycKamil PycKamil restored the s3_compose_objects-on-cache-hit branch September 13, 2024 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants