New schema and migration for 5.0.0

[octonato]

Events and Snapshots are wrapped in some Akka classe. The goal is to unwrap because those classes were never intended to serialized and saved to DB.

There is a branch in which this was fixed by James (original PR: #180).

However, we may not integrate it into master and start over again. The main reason is that the PR is done as such that allows users to keep the old column and have mixed mode in which old data is kept around using the wrappers and new data is saved unwrapped.

This strategy amounts to the complexity of the code, plus the functionality is enabled/disabled by flags.

Instead, our current goal is to provide a new schema and a migration tool to allow previous journals to be migrated. That means that users moving to 4.0.0 will have to shutdown the system, run the migration and restarting using the new version.

This is broken down in a three main tasks:

• Journal Store refactoring - New Journal Store Schema #315
• Snapshot Store refactoring - New Snapshot Store Schema #316
• Migration tool - Migration tool - 3.5.x => 5.0.0 #317

[WellingR]

I must say I really like the the forwards/backwards compatibiliy, since it allowed the application to read old events, and write them in the new way. The migration could be executed while the application was running.
My idea was to follow the 4.0.0 release with another release (5.0.0?) which does not have all the compatibility code and flags.

I definitely agree that having all the extra code and complexity for the compatibility is undesirable. However on the other hand an intermediate release might make the migration process a lot more bearable for our users...

[octonato]

True, it will be much nicer to be forward/backward compatible, but the cost is also high for us.

Maybe, an alternative approach would be to maintain a branch 3.5.x for a while so users don't need to immediately migrate to 4.0.0. They can decide to do the move on their own pace.

Concerning the issue related to unwrap the BLOB, this is not a major one and I don't think we need to fix in 3.5.x. Users can keep using it until they are ready for 4.0.0.

The major issue is the eventsByTag query, but that's mainly impacting Lagom users because of how we shard the tags. Of course, it can also impact other users. For instance, if you have two different kinds of PersistentActors that tag with UserCreated and AdminUserCreated each. A possible quick fix for this could be a DAO that does a = select instead of LIKE, but has the limitation of one tag per event.

Another possible quick fix: select whatever the user requested using LIKE and filter out in memory using a equals filter. It has the drawback of potentially fetching too much data, but I would expect it to be negligible.

So, here is the alternative path:

• 3.5.x is maintained for a while so users are not in a hurry to upgrade
• Quick fix for eventsByTag (I'm inclined to choose the in-memory filtering)
• No BLOB unwrapping in 3.5.x
• 4.0.0 with the new schemas and migration tool for 3.5.x to 4.0.0

The drawback is that it requires an offline migration, but the advantage is that we simplify the code a lot and we can move faster with the fixes.

WDYT?

[WellingR]

Sounds good.

Since the idea is to rename the table anyway, it would be possible to have both at the same time. Therefore it could be possible to run the migration tooling as part of 3.x.x to prepare for the 4.0.0 upgrade

[octonato]

Correct. It will also require to run the migration once again offline but will run much faster.

[octonato]

We just realised that we can't filter out events based on tags because the event envelop doesn't contain the tags. On the other hand, this is something that we can fix in Lagom instead.

[WellingR]

Actually I guess it could be possible to filter them out in the BaseByteArrayReadJournalDao

[ennru]

#467 introduces the new database schema.
A migration tools is not available at this time.