Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Storage: Enforce size based retention for topic #2179

Closed
sehz opened this issue Feb 9, 2022 · 4 comments · Fixed by infinyon/fluvio-website#294
Closed

Storage: Enforce size based retention for topic #2179

sehz opened this issue Feb 9, 2022 · 4 comments · Fixed by infinyon/fluvio-website#294
Assignees
@galibey
Labels
SPU SPU related storage
Milestone
0.9.24

Comments

@sehz
Copy link
Contributor

sehz commented Feb 9, 2022 •
edited
Loading

The current retention policy only supports the time-based property. This may result in a incorrect estimate of the total data size. We should add max data size properties as part of retention.

The topic should have new property: 'max_size' to indicate maximum data size will be allowed to accumulate.
@sehz sehz added SPU SPU related storage labels Feb 9, 2022
@nacardin nacardin added this to the 0.10.0 milestone Feb 11, 2022
@ExternalReality
Copy link

ExternalReality commented Mar 4, 2022 •
edited
Loading

Hi @sehz, I'd like to take a look at this issue. At first glace it looks to be a good first issue that is beyond just a simple CLI fix. Would it be a good idea to work this one as a new contributor?

@sehz
Copy link
Contributor Author

sehz commented Mar 4, 2022

Sounds good

@sehz sehz changed the title Storage: Enforce disk hard limit Storage: Enforce size based retention for topic Apr 5, 2022
@galibey galibey self-assigned this Apr 6, 2022
@galibey
Copy link
Contributor

galibey commented Apr 7, 2022

@sehz Do we really want to have max size per topic or per partition will suffice? Per topic size calculation will require communication between nodes whereas per partition can be done locally.

@sehz
Copy link
Contributor Author

sehz commented Apr 7, 2022 •
edited
Loading

Per partition make sense but specified at topic level

@galibey galibey mentioned this issue Apr 13, 2022
Closed
@sehz sehz modified the milestones: 0.10.0, 0.9.24 Apr 14, 2022
bors bot pushed a commit that referenced this issue Apr 14, 2022
Add Max Partition Size property to topic (#2297)
b1b4659 
This PR includes the following changes:
1. Added `max_partition_size` parameter to `fluvio-cli` and topic and partition specs.
2. `Cleaner` in `fluvio-storage` is responsible for enforcing replica size. Replica size is index size + log file size. If the size is exceeded so the first segment is removed.
3. `Cleaner` was moved from segments level to replica level to be able to access replica size.
4. Introduced `Size64` type as the partition size can easily overflow current `Size`.
5. `human_bytes` crate is replaced by `bytesize`. The latter is able to convert bytes to string and from string whereas the former only to string.

Closes #2179
bors bot pushed a commit that referenced this issue Apr 14, 2022
Put #2179 changes to 0.9.24 release notes (#2303)
102349c
bors bot pushed a commit that referenced this issue Apr 14, 2022
Put #2179 changes to 0.9.24 release notes (#2303)
6bb3804
bors bot pushed a commit that referenced this issue Apr 14, 2022
Put #2179 changes to 0.9.24 release notes (#2303)
6311700
bors bot pushed a commit that referenced this issue Apr 14, 2022
Put #2179 changes to 0.9.24 release notes (#2303)
0c719f8
bors bot pushed a commit that referenced this issue Apr 14, 2022
Put #2179 changes to 0.9.24 release notes (#2303)
e4ee0b5
@galibey galibey mentioned this issue Apr 15, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@galibey galibey

Labels
SPU SPU related storage
Projects
None yet
Milestone
0.9.24
4 participants
@sehz @ExternalReality @nacardin @galibey