Datastore copy util #469
Datastore copy util #469
Conversation
Signed-off-by: Sander Pick <sanderpick@gmail.com>
sanderpick
force-pushed
the
sander/datastore-migration
branch
from
75147de
to
bdd8cab
Compare
util/datastore/dscopy/main.go
Outdated
| for r := range results.Next() { | ||
| if err := to.Put(ds.NewKey(r.Key), r.Value); err != nil { |
There was a problem hiding this comment.
Between these two lines we need to add:
if r.Error != nil {
log.Fatal(...)
}
util/datastore/dscopy/main.go
Outdated
| for r := range results.Next() { | ||
| if err := to.Put(ds.NewKey(r.Key), r.Value); err != nil { | ||
| log.Fatalf("copying %s: %v", r.Key, err) | ||
| } |
There was a problem hiding this comment.
I discovered while working and testing Powergate migrations that this is too slow for copying (at least for my taste).
Here is what I did for migrations; which reduced times by some orders of magnitude. Nothing crazy, ignoring the migration logic, basically some rate-limiting to Read-and-Put up to 1000 in concurrently instead of serially.
There was a problem hiding this comment.
Nice, copied your approach there
Signed-off-by: Sander Pick <sanderpick@gmail.com>
Signed-off-by: Sander Pick <sanderpick@gmail.com>
Signed-off-by: Sander Pick <sanderpick@gmail.com>
Signed-off-by: Sander Pick <sanderpick@gmail.com>
|
Sorry the big code update post review here, but I realized using this tool would be much easier if it wasn't a generic datastore copier, but was specific to the structure of |
| }, | ||
| { | ||
| name: "ipfslite", | ||
| files: []string{"key"}, |
There was a problem hiding this comment.
Move the key as well...
| lock.Lock() | ||
| count++ | ||
| lock.Unlock() | ||
| if count%parallel == 0 { |
There was a problem hiding this comment.
Nit: move lock.Unlock() to include the if case since count is being read without lock and might be a data-race with other goroutine owning the lock. (tongue twister)
Signed-off-by: Sander Pick <sanderpick@gmail.com>
Adds a small program for copying a badger/mongo datastore to another badger/mongo datastore.