Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tiered storage - data archiving #290

Closed
spetz opened this issue Nov 12, 2023 · 5 comments · Fixed by #1055
Closed

Tiered storage - data archiving #290

spetz opened this issue Nov 12, 2023 · 5 comments · Fixed by #1055
Assignees
@spetz
Labels
new functionality New functionality server iggy-servere related change

Comments

@spetz
Copy link
Collaborator

spetz commented Nov 12, 2023

To support the different requirements for data archiving e.g. massive amount of records (TBs or even more), automatic backup in multiple environments and so on, it would be great to implement the so-called tiered storage - for example, instead of deleting the expired segments (as currently configurable via retention policy), it should be also possible to archive them, by storing somewhere else than the current server.
@spetz spetz added new functionality New functionality server iggy-servere related change labels Nov 12, 2023
@seroze
Copy link
Contributor

seroze commented Apr 16, 2024

Should we start with AWS S3 here ? background process would pipe the expired segment files directly to s3

@spetz
Copy link
Collaborator Author

spetz commented Apr 16, 2024

If you'd like to contribute, sure thing! I was thinking that instead of the background process (unless you have the actual application in mind), we could simply stick to the background task, similar to the existing message cleaner or saver.

@spetz
Copy link
Collaborator Author

spetz commented Jul 4, 2024

Ok, after the latest release with a new storage implementation, it's high time to start working on this, so I'll focus on it :)

@spetz spetz self-assigned this Jul 4, 2024
spetz added a commit that referenced this issue Jul 5, 2024
@spetz
Add initial archiver and tiered storage #290
a50246f
spetz added a commit that referenced this issue Jul 7, 2024
@spetz
Add logic for archiving and deleting the expired segments and full to…
0e3e138 
…pics, extend options #290
spetz added a commit that referenced this issue Jul 7, 2024
@spetz
Improve archiving segments logic depending on the config #290
b2c2ee7
spetz added a commit that referenced this issue Jul 8, 2024
@spetz
Extend Archiver trait with init method, add archive state command #290
e528e58
spetz added a commit that referenced this issue Jul 9, 2024
@spetz
Refactor archiver config into data maintenance, improve config valida…
eee54a0 
…tion #290
spetz added a commit that referenced this issue Jul 9, 2024
@spetz
Refactor archiver config into data maintenance, improve config valida…
30c3238 
…tion #290
spetz added a commit that referenced this issue Jul 10, 2024
@spetz
Update archiver trait #290
6c58207
spetz added a commit that referenced this issue Jul 10, 2024
@spetz
Complete disk archiver implementation, fix message maintainer #290
2561c5c
spetz added a commit that referenced this issue Jul 11, 2024
@spetz
Finalize disk archiver #290
ef387f9
spetz added a commit that referenced this issue Jul 11, 2024
@spetz
Finalize disk archiver #290
a83f433
spetz added a commit that referenced this issue Jul 12, 2024
@spetz
Disable S3 archiver until implemented #290
badd990
@spetz spetz mentioned this issue Jul 12, 2024
Merged
spetz added a commit that referenced this issue Jul 12, 2024
@spetz
Disable S3 archiver until implemented #290
aa23249
spetz added a commit that referenced this issue Jul 12, 2024
@spetz
Disable S3 archiver until implemented #290
5068dec
@spetz
Copy link
Collaborator Author

spetz commented Jul 12, 2024

Disk archiver has been implemented in #1053, S3 is next on the list.

spetz added a commit that referenced this issue Jul 12, 2024
@spetz
Disable S3 archiver until implemented #290
10f3899
spetz added a commit that referenced this issue Jul 12, 2024
@spetz
Disable S3 archiver until implemented #290
da2d5a0
spetz added a commit that referenced this issue Jul 12, 2024
@spetz
Disable S3 archiver until implemented #290
cba01b8
spetz added a commit that referenced this issue Jul 12, 2024
@spetz
Disable S3 archiver until implemented #290
26a0526
spetz added a commit that referenced this issue Jul 12, 2024
@spetz
Add disk archiver tests #290
80c2b04
spetz added a commit that referenced this issue Jul 12, 2024
@spetz
Add disk archiver tests #290
bd09090
spetz added a commit that referenced this issue Jul 12, 2024
@spetz
Archive tiered storage (#1053)
d6e5a90 
Implement optional and configurable disk archiver for closed segments
(messages) and state log (storage). This is related to #290 and is the first iteration of working data
archiver e.g. for the data backups purposes or just storing the old data in general.
@spetz spetz mentioned this issue Jul 12, 2024
Merged
spetz added a commit that referenced this issue Jul 12, 2024
@spetz
Implement S3 archiver #290
6de9da0
spetz added a commit that referenced this issue Jul 12, 2024
@spetz
Implement S3 archiver #290
22a973c
spetz added a commit that referenced this issue Jul 12, 2024
@spetz
Implement S3 archiver #290
4afb09e
spetz added a commit that referenced this issue Jul 13, 2024
@spetz
Implement S3 archiver #290
6d3ea67
spetz added a commit that referenced this issue Jul 13, 2024
@spetz
Implement S3 archiver #290
22e9d68
spetz added a commit that referenced this issue Jul 13, 2024
@spetz
Add temporary upload directory for S3 archiver #290
a2d59d2
@spetz
Copy link
Collaborator Author

spetz commented Jul 13, 2024

S3 archiver has been implemented in #1055 which closes this feature in its basic form.

spetz added a commit that referenced this issue Jul 13, 2024
@spetz
Implement S3 archiver (#1055)
fd0e80a 
This closes #290 as the second required archiver.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@spetz spetz

Labels
new functionality New functionality server iggy-servere related change
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Implement S3 archiver iggy-rs/iggy
2 participants
@spetz @seroze