746 migration #809
746 migration #809
Conversation
|
A command line argument sounds great. |
| log.Error("Error Migrating: ", err) | ||
| return | ||
| } | ||
| // go through in reverse order so most recently created shards will be migrated first |
There was a problem hiding this comment.
I think the directories are sorted using lexicographic order, but we should use numeric sorting according to the comment above. Also, the sort order isn't specified in the documentation. I'm suggesting to explicitly sort the entries.
There was a problem hiding this comment.
In this case, lexigraphical and numerical are the same since the directory names are zero padded. However, I didn't know that the sort order wasn't guaranteed in ReadDir, I just assumed based on what I was getting back.
At the end of the day, it's not super important which get migrated first. The shard id order isn't even a guarantee on which shards have newer data (for instance if they backfill after writing a bunch of current stuff). Ok if I leave as is for now?
- Removed all the deprecated stuff from datastore/shard - Add data_migrator and test - Update server to create and run the migrator - Add the old level_db_shard.go to read old data
* Moved over the data migrator * Moved over the old leveldbshard code and updated datastore/shard so old code could reference everything.
Will write a file into the shard directory. If a migration is called again, it will skip any shard that has a migration marker in the directory.
| return | ||
| } | ||
| atomic.StoreUint32(&self.migrationRunning, uint32(1)) | ||
| defer atomic.StoreUint32(&self.migrationRunning, uint32(0)) |
There was a problem hiding this comment.
I think we should either use compare and swap or a lock but not both, unless I'm missing something
Use CompareAndSwap instead, it's cleaner.
Forces order of shard migrations. Also remove old commented code.
note quite done. Still have to add the flagging that a shard has been migrated so it won't be done again.
I'm also considering making data migrator only run if they start up influx with an argument to do it. That way, people could upgrade, start things up and then create shard spaces before migrating data. Thoughts?