WIP: Added support for serializing using Akka serialization #199

WellingR · 2018-09-02T17:00:25Z

This is a continuation of #180

Todos (help wanted):

Improve migration script see here and here (in progress: Introduce better logging. Fix for Oracle. Performance fix. #201)
Investigate if we can split out the legacy and new table definitions see here (in progress Investigation if the legacy and new table definitions can be split out #203)
Make sure that the tests use both the new and the legacy configurations.
Re-enable the supportsSerialization compatibility flag (the tests break if this is enabled)
The journal migration tests should work for all supported types of databases. (Perhaps we could read the data from a csv file and convert that to some slick insert statements or something).
The journal migration uses a RowsPerTransaction configuration value which is currently hardcoded. It might be an idea to include rowsPerTransaction as a method parameter when the migration is started.
The changes from Add support for SQL Server #200 have been merged to master, these changes have been merged to this branch too, however there is no schema migration script yet. We should also investigate if additional tests are needed.
Fix oracle sql schema migration bug see here

Added support for serializing using Akka serialization including migration

- Fixed a mistake in the configuration example for custom database providers - Improved documentation on thread/connection pool sizing (specifically by removing the vague 5 * number of cores recommendation and pointing to a better resource)

octonato · 2018-09-03T07:49:40Z

Thanks @WellingR. Good idea to open a serialization branch for this.

patriknw · 2018-09-03T13:43:10Z

src/main/scala/akka/persistence/jdbc/journal/dao/ByteArrayJournalSerializer.scala

+        case _ if serializer.includeManifest =>
+          payload.getClass.getName
+        case _ => ""
+      }


this should be replaced by

val serManifest = akka.serialization.Serializers.manifestFor(serializer, payload)

…ifest to use

skisel · 2018-09-08T14:22:21Z

I'm testing this branch and it looks to me Oracle migration does not work - probably this is something to do with distinguishing empty and NULL blobs. Will try to find out why it happens.

skisel · 2018-09-08T14:50:48Z

It looks to me slick has a bug here: https://github.com/slick/slick/blob/master/slick/src/main/scala/slick/jdbc/OracleProfile.scala#L155

Slick generates something like this for migration code (I shortened) and on non-migrated db it should return ALL rows, but no result returned:
select * from "journal" where dbms_lob.compare("event", NULL) = 0)
This actually works:
select * from "journal" where "event" is null

By changing from .filter(_.event.isEmpty) to .filter(_.serId.isEmpty) it actually works. I will try to improve migration and will submit PR.

skisel · 2018-09-08T19:45:14Z

#201

- scala 2.12.6 - akka 2.5.16 - scalatest 3.0.5 (test only) - postgres 42.2.5 (test only) - mysql 8.0.12 (test only)

…yped

skisel · 2018-09-18T21:08:44Z

Another bug to add here:
sql> ALTER TABLE "journal" MODIFY "message" BLOB DEFAULT NULL [2018-09-18 23:08:08] [99999][22859] ORA-22859: invalid modification of columns

@naferx

Adds SQL Server schema and test updates. I created a derived SQL server [image](https://hub.docker.com/r/topaztechnology/mssql-server-linux/) that allows login and database creation on startup to facilitate testing. Write performance was initially poor vs Postgres, but significantly improved with use of `transactionally` in `BaseByteArrayJournalDao.writeJournalRows` This was based on a closed PR #108, thanks to @naferx

…efinitions. Rows per transaction has been made configurable.

PerWiklander · 2019-09-30T23:40:05Z

It looks like the things left to do here are related to migrations. What is the status of this feature for new projects without old events to support?

WellingR · 2019-10-01T17:05:23Z

@renatocaval @skisel @jroper
This PR has been silent for almost a year, it is now behind master and there are probably some conflicts that may need to be resolved.

Do we still want to make these improvements even given the fact that migrations are needed?
Is any contributor willing to spend the time to work on finishing the remaining work?

I would be happy to contribute myself, but time is an issue.

octonato · 2019-10-02T09:36:43Z

@WellingR, we have talked about this with the Akka team recently. We will need to make time to move one with that. We don't have it yet planned, but it's in our radar.

Not only that but that tagging issue. Actually, tagging is the most important one. And because we need some migration tooling, it will be a good opportunity to tackle that one altogether.

octonato · 2020-03-06T11:03:28Z

Also closing this one as we will be taking a different approach as discussed in #318

jroper and others added 4 commits September 2, 2018 18:20

Added support for serializing using Akka serialization (#180)

48f187e

Added support for serializing using Akka serialization including migration

lowercased field name in postgres schema

e33665b

Re-enabled supportsSerialization compatibility flag and fixed tests

c13636b

Documentation fixes (#197)

1d1b35d

- Fixed a mistake in the configuration example for custom database providers - Improved documentation on thread/connection pool sizing (specifically by removing the vague 5 * number of cores recommendation and pointing to a better resource)

patriknw reviewed Sep 3, 2018

View reviewed changes

Use Serializers.manifestFor(serializer, payload) to determine the man…

ba5ee96

…ifest to use

WellingR added the help wanted label Sep 3, 2018

WellingR mentioned this pull request Sep 3, 2018

Added support for serializing using Akka serialization #180

Merged

WellingR added this to the 4.0.0 milestone Sep 3, 2018

WellingR mentioned this pull request Sep 6, 2018

Add support for SQL Server #200

Merged

skisel mentioned this pull request Sep 9, 2018

Investigation if the legacy and new table definitions can be split out #203

Closed

skisel and others added 3 commits September 15, 2018 15:28

Introduce better logging. Fix for Oracle. Performance fix. (#201)

7966d95

Dependency updates (#202)

06486b0

- scala 2.12.6 - akka 2.5.16 - scalatest 3.0.5 (test only) - postgres 42.2.5 (test only) - mysql 8.0.12 (test only)

switch to using systemActorOf so this library can be used with Akka T…

7c5af74

…yped

skisel mentioned this pull request Sep 18, 2018

#199 WIP. Fixing DDL generation. #205

Merged

jonfox and others added 6 commits September 19, 2018 16:41

Specifies SBT 1.2.3. (#209)

970ff15

Upgrades Scala to 2.12.7 and Akka to 2.5.17. (#208)

3369f14

Adds allowPublicKeyRetrieval=true to all MySQL URLs. (#210)

7618ecf

Fix DDL generation for legacy and non-legacy journal/snapshot table d…

b382e09

…efinitions. Rows per transaction has been made configurable.

Merge branch 'master' into akka-serialization

31641cc

octonato closed this Mar 6, 2020

octonato removed this from the 4.0.0 milestone Mar 6, 2020

ennru deleted the akka-serialization branch November 19, 2020 20:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Added support for serializing using Akka serialization #199

WIP: Added support for serializing using Akka serialization #199

WellingR commented Sep 2, 2018 •

edited

Loading

octonato commented Sep 3, 2018

patriknw Sep 3, 2018

skisel commented Sep 8, 2018

skisel commented Sep 8, 2018 •

edited

Loading

skisel commented Sep 8, 2018

skisel commented Sep 18, 2018

PerWiklander commented Sep 30, 2019

WellingR commented Oct 1, 2019 •

edited

Loading

octonato commented Oct 2, 2019

octonato commented Mar 6, 2020

WIP: Added support for serializing using Akka serialization #199

WIP: Added support for serializing using Akka serialization #199

Conversation

WellingR commented Sep 2, 2018 • edited Loading

octonato commented Sep 3, 2018

patriknw Sep 3, 2018

Choose a reason for hiding this comment

skisel commented Sep 8, 2018

skisel commented Sep 8, 2018 • edited Loading

skisel commented Sep 8, 2018

skisel commented Sep 18, 2018

PerWiklander commented Sep 30, 2019

WellingR commented Oct 1, 2019 • edited Loading

octonato commented Oct 2, 2019

octonato commented Mar 6, 2020

WellingR commented Sep 2, 2018 •

edited

Loading

skisel commented Sep 8, 2018 •

edited

Loading

WellingR commented Oct 1, 2019 •

edited

Loading