store event data in separate column, no wrapping, #12 #21
Conversation
| row.getInt("ser_id"), | ||
| row.getString("ser_manifest") | ||
| ).get | ||
| } |
There was a problem hiding this comment.
I just duplicated these to 3 places, needs some DRY
|
Moved comment here: @patriknw I had a quick look and started thinking about how to verify that it really works. I'd really like to have a test for it since it would be very time consuming to go back and forth when debugging issues related to the migration if I have to do the migration manually each time. I'm considering the following approach: take one of the snapshot tests and pause it just before it exits, export the data from Cassandra tables and use that as a base for a DB-script that would populate the Cassandra test database at the beginning of the migration test and the proceed to test migration and persisting new events. Would that cover it? suggestions welcome! The only extra scenario I can come up with is that the user moves back and forth between versions after recording data. But I cannot judge yet if that would make any difference, are we forwards compatible as well? |
|
As far as I can see there is no actual schema migration taking place yet. I think the old FYI: I had a look at krasserm/akka-persistence-cassandra#64 to see if I could solve that issue at the same time during schema migration. But contrary to what is stated in the README that migrating from 0.3x to 0.4x will be solved, I don't think that it will be feasible to do that within the scope of this project. The old table used the |
|
I would have expected a one time manual test of the backwards compatibility, but if you think you can pull off an automated test I will not complain (as long as it is not too complicated to maintain). We don't have to care about forward compatibility. We try to be nice about backwards compatibility, i.e. reading data stored with old version. Right now we are only doing it on best effort basis, and some changes will require data migration (ETL) by external tools. Once we have released 1.0 we will be more strict. I'm not sure I understand the implications of keeping old |
|
Aha, you were thinking that the user should manually migrate their schema? I was assuming an automated migration process (perhaps a bit colored after reading the krasserm/akka-persistence-cassandra#64 which talks about automated schema migration). So in that case the user will have to manually apply the schema changes when migrating. About the tests, at least I'll give it a try and you can have a look. I hope it will not be more work than creating a throw away application just for the purpose of testing backwards compatibility. |
* The event data, snapshot data and meta data are stored in a separate columns instead of being wrapped in blob. * This makes it easier to browse the event data with standard tools.
patriknw
force-pushed
the
wip-12-browsable-patriknw
branch
from
6046489 to
f40e251
Compare
patriknw
changed the title
|
@magnusart I have updated this. Did similar for snapshots. Tested migration manually, and added migration guide. |
|
@patriknw is there a schedule for this? I know there is a migration path, but if possible would prefer to avoid doing that and try timing an internal release to production based on this. |
|
@pjan I will release a new version as soon as Akka 2.4.2 is released, and then this will be included. That should be tomorrow, or within a few days. |
patriknw
force-pushed
the
wip-12-browsable-patriknw
branch
from
9a7525f to
f40e251
Compare
|
interesting, JournalPerfSpec failed several times when running on Travis when I tried to use cassandra-all 3.0.3 (worked fine locally), reverted back to 3.0.2 for now |
store event data in separate column, no wrapping, #12
|
@patriknw Very cool. Need to give this a spin now. |
|
@pjan v0.10 is released |
|
@patriknw thank you so much for this. Akka team is nothing short of amazing. |
Not ready, just a starting point for discussion.