Can I quickly import rdb data?

[chunjieyaya]

Feature request type

sample request

Is your feature request related to a problem? Please describe

After saving the existing redis as rdb, I want to import it into garnet

Describe the solution you'd like

You can import them by running commands

Describe alternatives you've considered

No response

Additional context

No response

[badrishc]

Workaround would be to load the rdb into redis, scan the DB, and issue set operations on Garnet.

[chunjieyaya]

Workaround would be to load the rdb into redis, scan the DB, and issue set operations on Garnet.

Can the follow-up plan do this function?

[vazois]

Workaround would be to load the rdb into redis, scan the DB, and issue set operations on Garnet.

Can the follow-up plan do this function?

This a good idea, to have an rdb-importer for Garnet. Unfortunately, it will require constant maintenance to keep up with any updates to the rdb format.
I actually did a quick research and saw that there exist many different parsers for parsing, formatting and analyzing rdb files.
However, all of them do not seem to have support for newer redis versions.
I am adding here some of them for reference.

If we had a robust C# rdb parser then we can use the internal Garnet API to convert the data into upsert commands.
Similar to what rdb-rs is doing with the conversion to RESP protocol.
Bottomline, this is an issue that should be marked as help is needed.
If someone wants to take a stub at it, I am happy to provide some guidance/feedback.

https://github.com/SamuelFisher/CLRdb
https://github.com/badboy/rdb-rs
https://github.com/redis/librdb
https://github.com/HDT3213/rdb

https://rdb.fnordig.de/file_format.html
https://github.com/redis/redis/blob/unstable/src/rdb.h

[chunjieyaya]

Workaround would be to load the rdb into redis, scan the DB, and issue set operations on Garnet.

Can the follow-up plan do this function?

This a good idea, to have an rdb-importer for Garnet. Unfortunately, it will require constant maintenance to keep up with any updates to the rdb format. I actually did a quick research and saw that there exist many different parsers for parsing, formatting and analyzing rdb files. However, all of them do not seem to have support for newer redis versions. I am adding here some of them for reference.

If we had a robust C# rdb parser then we can use the internal Garnet API to convert the data into upsert commands. Similar to what rdb-rs is doing with the conversion to RESP protocol. Bottomline, this is an issue that should be marked as help is needed. If someone wants to take a stub at it, I am happy to provide some guidance/feedback.

https://github.com/SamuelFisher/CLRdb https://github.com/badboy/rdb-rs https://github.com/redis/librdb https://github.com/HDT3213/rdb

https://rdb.fnordig.de/file_format.html https://github.com/redis/redis/blob/unstable/src/rdb.h

Very good

[PaulusParssinen]

I can write a robust and small RDB (de)serialization .NET library as 3rd-party library in upcoming weeks and publish it on NuGet. Would this work for Garnet?

[vazois]

I can write a robust and small RDB (de)serialization .NET library as 3rd-party library in upcoming weeks and publish it on NuGet. Would this work for Garnet?

I think that would be great idea to start with something as a 3rd-party library and work similarly to what the other projects are doing (i.e. have an rdb to RESP protocol parser or json file) but of course being more up to date.
If it is native C# it will be cool to integrate to Garnet and use the internal GarnetAPI and load the rdb file during recovery phase.
Though I would suggest doing the parser first since this is the major ticket item.
The problem I found while researching for this (and admittedly did not spend too much on it) is that there are no official docs for of rdb format.
The only way I see it is to reverse engineer it from the redis codebase.
I posted some relevant link about this also.
As you find more information it will be great to post here

[PaulusParssinen]

Reverse engineering file-formats is right up my alley. This is already much more easier already because I have some source to look at, which normally isn't the case for me 😄 (I can reverse protocols/file-formats out of a black-box if need be)

I will of course only look at Redis souce pre license change (I have a fork frozen at last commit before the license change e64d91c). And yea, parser first of course.

[PaulusParssinen]

By the looks of it, I may have a time slot to tackle this starting next week 👍

UPDATE: gotta postpone a bit. Up for grabs in the meantime.