Migration tool #501

Tochemey · 2021-02-05T18:08:39Z

Legacy journal migration implementation
Legacy snapshot migration implementation
Event Tag migration
Unit tests
QA Legacy Journal data migration
QA Legacy Snapshot data migration

@octonato, @ennru and @patriknw kindly take a look and advice.

Tochemey · 2021-02-07T20:00:15Z

@octonato , @patriknw for some weird reasons I am getting some type mismatch here https://travis-ci.com/github/akka/akka-persistence-jdbc/jobs/481103765#L368

octonato

@Tochemey, this is looking good.

I like the event adapter idea.

I'm wondering if we should take a similar path as the Cassandra plugin. The Cassandra has a Cleanup tool with different methods that users can combine to achieve what they want.

For instance, do I need to preserve the ordering of events? I may want to do it if I'm using Akka Projections and don't want to replay all my projections.

In that case, preserving the ordering is a must.

If I don't care about the ordering, I may even migrate per persistence id and in parallel.

build.sbt

octonato · 2021-02-11T10:46:38Z

tools/src/main/scala/akka/persistence/jdbc/tools/migration/LegacyJournalDataMigrator.scala

+  private def adaptEvents(repr: PersistentRepr): Seq[PersistentRepr] = {
+    val adapter: EventAdapter = eventAdapters.get(repr.payload.getClass)
+    adapter.fromJournal(repr.payload, repr.manifest).events.map(repr.withPayload)
+  }


This is neat.

This will allow users to eventually migrate their events to some new format.

There is one thing that we should watch-out though. If an adapter produces more events then it takes (1 event split in many), that will change the ordering in the new table.

That's not an issue in itself unless users are tracking the offset using Lagom or Akka Projections.

That's all good, but we need to carefully inform users that their adapters should emit 1:1 in that case or they should reset their offset table and consume from scratch.

There is one more thing on the adapters approach, it that we can re-tag the events.

Interesting.

octonato · 2021-02-11T10:49:53Z

tools/src/main/scala/akka/persistence/jdbc/tools/migration/LegacyJournalDataMigrator.scala

+  private def allEvents(): Source[PersistentRepr, NotUsed] = {
+    legacyReadJournalDao
+      .allPersistenceIdsSource(Long.MaxValue)
+      .flatMapConcat((persistenceId: String) => {


The mapConcat will change the ordering in the new schema. Instead, we should read the events directly without fetching by persistence id.

octonato · 2021-02-11T10:51:44Z

tools/src/main/scala/akka/persistence/jdbc/tools/migration/LegacySnapshotDataMigrator.scala

+  def migrate(): Source[Option[Future[Unit]], NotUsed] = {
+    legacyReadJournalDao.allPersistenceIdsSource(Long.MaxValue).mapAsync(1) { persistenceId: String =>
+      legacySnapshotDao
+        .latestSnapshot(persistenceId)


We may want to offer the option to migrate the last N snapshots. Is common for users to keep more than snapshot and in that case they probably want to migrate them as well.

octonato · 2021-02-12T16:47:50Z

@Tochemey, I have a second thought about the Flyway part. Maybe we should just drop it. There is too much on our side to have something that would work for all databases. Selecting the type, etc. At the end, what is the real added value?

Maybe the best is to let users just pick the schema as we publish in the docs (or use the new SchemaUtils) to create it. Then run the tool using new legacy and new DAOs. And that's it.

The migration doesn't need to take care of creating the schema for the users.

Tochemey · 2021-02-13T08:42:52Z

However the new SchemaUtils need to be made a bit flexible to accommodate the user defined postgres schema. At the moment it is tied to the public schema which can be problematic.

Tochemey · 2021-02-14T18:02:48Z

@octonato
Hello. I am able to move data into the new table for both journal and snapshot. However the event_tag did not get populated. I thought the DefaultJournalDao takes care of that. I am suspecting this https://github.com/akka/akka-persistence-jdbc/blob/master/core/src/main/scala/akka/persistence/jdbc/journal/dao/DefaultJournalDao.scala#L78 which got executed instead of the previous one

Tochemey · 2021-02-14T23:29:54Z

@patriknw some lovely feedback will not be bad.😁

# Conflicts: # migrator/src/main/scala/akka/persistence/jdbc/migrator/LegacySnapshotDataMigrator.scala

Tochemey · 2021-02-23T02:53:36Z

@octonato and @patriknw I am able to migrate all data from snapshot and journal into the new tables. However for the tags migration this is the code I use.

private def events(): Source[PersistentRepr, NotUsed] = {
    Source
      .fromPublisher(
        journaldb.stream(queries.JournalTable.sortBy(_.sequenceNumber).result)
      )
      .via(serializer.deserializeFlow)
      .mapAsync(1)((reprAndOrdNr: Try[(PersistentRepr, Set[String], Long)]) => Future.fromTry(reprAndOrdNr))
      .map { case (repr, tags, _) =>
        repr.withPayload(Tagged(repr.payload, tags))
      }
  }

I don't think we will need an event adapter since the payload is the raw one that is persisted into the old journal table column as well as the tags. I strongly believe we just need to move the data.

octonato · 2021-03-08T16:11:37Z

I don't think we will need an event adapter since the payload is the raw one that is persisted into the old journal table column as well as the tags. I strongly believe we just need to move the data.

@Tochemey, I think you are right on this.

The good part of this approach is that it doesn't leave the door open for adding more events. An event adapter is capable of taking one event and emit more than one. That would add more events on the new table.

octonato

Thanks @Tochemey.

I think we are going in a good direction. Cool that you manage to read the tags and remove the event adapter.

We will need to think of integration tests.
First start the system with old dao, persist some events and snapshots. Stop it, migrate the data and start again using the new DAOs.

octonato · 2021-03-08T16:13:10Z

build.sbt

@@ -4,7 +4,7 @@ lazy val `akka-persistence-jdbc` = project
  .in(file("."))
  .enablePlugins(ScalaUnidocPlugin)
  .disablePlugins(MimaPlugin, SitePlugin)
-  .aggregate(core, migration, docs)
+  .aggregate(core, migration, docs, migrator)


Let's remove the migration project and keep only the migrator.

octonato · 2021-03-08T16:14:36Z

build.sbt

+  .settings(
+    name := "akka-persistence-jdbc-migrator",
+    libraryDependencies ++= Dependencies.Migration ++ Dependencies.Libraries,
+    publish / skip := true)


Add a TODO here so we don't forget to remove it when ready.

We want to publish the tool, but only when ready for use. It doesn't need to be when production ready. We may decide to publish it on a snapshot to test it out on real world projects.

octonato · 2021-03-08T16:15:33Z

migrator/src/main/scala/akka/persistence/jdbc/migrator/LegacyJournalDataMigrator.scala

+      .mapAsync(1)((repr: PersistentRepr) => {
+        defaultJournalDao.asyncWriteMessages(Seq(AtomicWrite(Seq(repr))))
+      })


Suggested change

.mapAsync(1)((repr: PersistentRepr) => {

defaultJournalDao.asyncWriteMessages(Seq(AtomicWrite(Seq(repr))))

})

.mapAsync(1) { repr =>

defaultJournalDao.asyncWriteMessages(Seq(AtomicWrite(Seq(repr))))

}

octonato · 2021-03-08T16:19:08Z

project/Dependencies.scala

@@ -33,7 +33,7 @@ object Dependencies {
    "org.scalatest" %% "scalatest" % ScalaTestVersion % Test) ++ JdbcDrivers.map(_ % Test)

  val Migration: Seq[ModuleID] = Seq(
-    "org.flywaydb" % "flyway-core" % "7.5.1",
+    "org.flywaydb" % "flyway-core" % "7.5.2",


Lets' remove this. We don't need it.

octonato · 2021-03-08T16:21:00Z

migrator/src/main/scala/akka/persistence/jdbc/migrator/LegacyJournalDataMigrator.scala

+        defaultJournalDao.asyncWriteMessages(Seq(AtomicWrite(Seq(repr))))
+      })
+      .limit(Long.MaxValue)
+      .runWith(Sink.seq) // FIXME for performance


Sing.ignore is good enough here and we won't accumulate records in memory.

octonato · 2021-03-08T16:36:33Z

migrator/src/main/scala/akka/persistence/jdbc/migrator/LegacyJournalDataMigrator.scala

+      .mapAsync(1)((reprAndOrdNr: Try[(PersistentRepr, Set[String], Long)]) => Future.fromTry(reprAndOrdNr))
+      .map { case (repr, tags, _) =>
+        repr.withPayload(Tagged(repr.payload, tags))
+      }


We don't need to go from Try to Future and back to unwrapped tuple 3.

I think that whole block can be changed to:

.map { case Success((repr, tags, _)) if tags.nonEmpty => repr.withPayload(Tagged(repr, tags)) // only wrap in `Tagged` if needed case Success((repr, _, _)) => repr // noops map case Failure(exception) => throw exception // blow-up on failure }

Eventually we need to add more logging inside deserializeFlow. If something goes wrong, that will be when deserializing the payload.

octonato · 2021-03-08T16:45:13Z

migrator/src/main/scala/akka/persistence/jdbc/migrator/LegacySnapshotDataMigrator.scala

+  def migrateAll(): Future[Seq[Future[Unit]]] = {
+    for {
+      rows <- snapshotdb.run(queries.SnapshotTable.sortBy(_.sequenceNumber.desc).result)
+    } yield rows.map(toSnapshotData).map { case (metadata, value) =>
+      defaultSnapshotDao.save(metadata, value)
+    }
+  }


Snapshots tend to have payloads much larger then events. I think this query will fetch all snapshots into memory. Better if we stream like we did for the events.

octonato · 2021-03-08T16:46:53Z

migrator/src/main/scala/akka/persistence/jdbc/migrator/LegacySnapshotDataMigrator.scala

+
+  def migrateLatest(): Future[Option[Future[Unit]]] = {
+    for {
+      rows <- snapshotdb.run(queries.SnapshotTable.sortBy(_.sequenceNumber.desc).take(1).result)


This will only take the latest snapshot across all persistence ids.

Insteads, we should use get the distinct list of persistence id (in a stream), run through it and fetch the latest per persistenceId.

octonato · 2021-03-08T16:49:20Z

migrator/src/main/scala/akka/persistence/jdbc/migrator/LegacySnapshotDataMigrator.scala

+  /**
+   * migrate all the legacy snapshot schema data into the new snapshot schema
+   */
+  def migrateAll(): Future[Seq[Future[Unit]]] = {


Future[Seq[Future[Unit]]] this will confuse users. When is the migration done? When the outer future completes I still have a bunch on Futures in flight.

Better if we return a Future[Done] here and that we do it using a stream of snapshots. We consume one by one and we finish with Sink.ignore and return a Future[Done].

octonato · 2021-03-08T16:51:48Z

migrator/src/main/scala/akka/persistence/jdbc/migrator/LegacySnapshotDataMigrator.scala

+  def migrate(offset: Int, limit: Int): Future[Seq[Future[Unit]]] = {
+    for {
+      rows <- snapshotdb.run(queries.SnapshotTable.sortBy(_.sequenceNumber.desc).drop(offset).take(limit).result)
+    } yield rows.map(toSnapshotData).map { case (metadata, value) =>
+      defaultSnapshotDao.save(metadata, value)
+    }
+  }


Maybe this one is less useful. I see the value in migrating only the latest snapshot like in the following method. But migrating a subset is maybe something that we can skip.

It's also one scenario less to test for us.

octonato

@Tochemey, I left a few more comments and a proposal for a test plan.

octonato · 2021-03-22T13:20:55Z

migrator/src/main/scala/akka/persistence/jdbc/migrator/package.scala

+   * This trait will be implemented by the various rdbms supported by the akka persistence jdbc since
+   * the sequence number is handled differently by the various databases
+   */
+  sealed trait JournalOrdering {


Oh! I see it now.

So, the basic issue is that we are forcing new ordering because we are importing the ordering from another table and therefore, when we finished, the new table sequence/auto-increment needs to be reconfigured to start from the last + 1. Correct?

I remember that at some point you mentioned that you were getting the ordering interleaved, but I think the main reason was because when streaming the results you were calling sortBy(_.sequenceNumber).

Wouldn't we just get the right ordering if we read on one side, respecting the ordering, and insert on the other? At the end of the game, the ordering on the new table should be exactly the same. Or am I missing something obvious?

The JournalOrdering you introduced looks neat if we don't have another solution. But I would first try to just ensure that the ordering is preserved by inserting on the same order. As such we will have less code.

octonato · 2021-03-22T13:32:03Z

migrator/src/main/scala/akka/persistence/jdbc/migrator/package.scala

+
+import scala.concurrent.{ ExecutionContext, Future }
+
+package object migrator {


I don't think we need to add a package object. We can just add a DatabaseType.scala and JournalOrdering.scala file.

It makes it easier to find them when just browsing the code in GitHub, text editor or file browser. When we add traits and classes to a package.scala, we need to know they live there in order to find them.

The ordering can never be respected because it is generated by the database. So the only way to ensure the ordering from the old journal to the new is to force insert them during the insertion into the new journal because the ordering field is a database auto-increment field. So after the migration of the data reset the ordering per database on the new journal as done here.

but why wouldn't the ordering be respected if the records are inserted in the same order? True that the ordering is generated by the db, but it is deterministic, no?

If we have a blank slate and we insert the first event, the ordering will be 1, second event ordering 2, etc.

What am I missing?

octonato · 2021-03-22T13:47:47Z

migrator/src/test/scala/akka/persistence/jdbc/migrator/BaseJournalMigratorSpec.scala

+import java.util.UUID
+import scala.concurrent.{ ExecutionContext, Future }
+
+abstract class BaseJournalMigratorSpec(config: String) extends SingleActorSystemPerTestSpec(config) {


I think it will be more convenient (and closer to user experience) if the tests start a EventSourcedBehavior using the legacy tables and fill it by sending commands instead of going low-level and manipulate the queries directly.

When doing so, we can also configure the behavior to save the snapshots every N events (eg: N=5) and to keep 3 snapshots for example.

That's why I was saying that we may need to start two actor systems.

I suggest the following test plan:

start actor system with the legacy config and populate the journal for a few entities, with many events and snapshots. Stop actor system

run migrator for events and snapshots

start another actor system using default config. Reload each entity and verify that their state is as expected.

The Account entity is a good one because we can easily test the balance.

Another tests could be:

start actor system with the legacy config and populate the journal for a few entities, with many events and save snapshot on each event. Stop actor system

run migrator only for snapshots

start another actor system using default config. Reload each entity and verify that they state is as expected.

In that second test, since we save snapshots on each event we don't need to migrate the events and when verifying the state, we prove that we indeed migrated all snapshots.

Finally, we need to test that the tags are correctly migrated.
For that test I think it suffice to create one entity that persist events with tags, migrate the events and consume the tags using eventsByTags from the new table.

In that test, we can simply verify that we consume the same amount of events as we generated through the entity.

WDYT?

flonedo · 2021-09-27T13:51:27Z

Hello, is this still being worked on? Or is it blocked by some underlying issue? We have some production code we would like to upgrade to a newer akka-persistence version, and this PR would be extremely useful for us. We'd be happy to contribute with testing, or help in any other way if necessary. Thanks in advance.

gabriele83 · 2021-09-29T08:16:20Z

We'd be happy to help, there is a way we can contribute?

gabriele83 · 2021-10-01T13:01:40Z

migrator/src/main/scala/akka/persistence/jdbc/migrator/JournalMigrator.scala

+  private val bufferSize: Int = journalConfig.daoConfig.bufferSize
+
+  // get the journal ordering based upon the schema type used
+  private val journalOrdering: JournalOrdering = profile match {


Non-exhaustive pattern matching

gabriele83 · 2021-10-01T13:05:06Z

migrator/src/main/scala/akka/persistence/jdbc/migrator/JournalMigrator.scala

+   *
+   * @param pr the given PersistentRepr
+   */
+  private def unpackPersistentRepr(pr: PersistentRepr): (PersistentRepr, Set[String]) = {


never used declaration

gabriele83 · 2021-10-01T13:16:44Z

migrator/src/main/scala/akka/persistence/jdbc/migrator/JournalMigrator.scala

+      tags: Set[String],
+      ordering: Long): (JournalAkkaSerializationRow, Set[String]) = {
+
+    val serializedPayload: AkkaSerialization.AkkaSerialized =


val serializedPayload: AkkaSerialization.AkkaSerialized =
AkkaSerialization.serialize(serialization, repr.payload).get

gabriele83 · 2021-10-01T13:19:27Z

migrator/src/main/scala/akka/persistence/jdbc/migrator/JournalOrdering.scala

+  /**
+   * the actual event journal table
+   */
+  val tableName: String = s"${journalConfig.eventJournalTableConfiguration.tableName}"


it is enough only:

val tableName: String = journalConfig.eventJournalTableConfiguration.tableName

gabriele83 · 2021-10-01T13:23:31Z

migrator/src/main/scala/akka/persistence/jdbc/migrator/SnapshotMigrator.scala

+  private val legacyJournalDao: ByteArrayReadJournalDao =
+    new ByteArrayReadJournalDao(journaldb, profile, readJournalConfig, SerializationExtension(system))
+
+  private def toSnapshotData(row: SnapshotRow): (SnapshotMetadata, Any) =


private def toSnapshotData(row: SnapshotRow): (SnapshotMetadata, Any) =
serializer.deserialize(row).get

octonato · 2021-10-05T08:51:11Z

@flonedo / @gabriele83, I think the best way to contribute on this PR is to see with @Tochemey as he's the original author.

There are a few paths we can follow:

@Tochemey, gives access to his repo to both of you and you push code to the branch used in this PR (complicated solution)
you fork @Tochemey fork and send PRs to his fork targeting the branch used here. Once merged there, the changes will show up on this PR (also complicated).
I merge this branch as is, but not on master, but on another branch. You guys can continue work sending PRs to this repo, but targeting the new 'feature' branch. Once the branch is ready, we merge it into master.

I think option 3 is the easiest and will allow others to contribute, but first we need to know from @Tochemey if he will continue work on this.

gabriele83 · 2021-10-05T09:02:30Z

@octonato I have already opened a PR 4 days ago on the fork of @Tochemey, I'm waiting ...

Tochemey · 2021-10-05T09:32:42Z

@gabriele83 The migration code has been used in production already and it was successful(we were only using postgres and we did get all relevant tests). However because of the general nature of the lib there might be some edge cases. Honestly I believe it should work for any of the because we migrate the raw data from the old schema unto the new. The only thing that remains that I can think of at the moment are the tests(unit and integration) which I have not got the time to write. Also @octonato will be the best person to give us the edge cases that can be an issue.

I am glad to assist whenever necessary.

Tochemey · 2021-10-05T10:06:27Z

@gabriele83 I have left a comment on you PR.

fix non-exhaustive pattern matching

Tochemey · 2021-10-05T10:56:18Z

@gabriele83 I have added you as collab on my branch. When free kindly fix the conflicts issue and let us merge it to this PR.

flonedo · 2021-10-08T09:00:52Z

@octonato Hello, a while back you suggested to create two actor systems for test purpose (one using legacy journal and one using the new one). How can I set up my tests to do so? Do you have any examples?

octonato · 2021-10-11T19:45:39Z

@octonato Hello, a while back you suggested to create two actor systems for test purpose (one using legacy journal and one using the new one). How can I set up my tests to do so? Do you have any examples?

@flonedo / @gabriele83, we don't have tests using 2 actor systems. That would be the first time we do it. We do have a few tests that starts and stops an actor system on each tests.
Have look at tests extending SingleActorSystemPerTestSpec.

This abstract spec offers some helper methods that you can use, for instance:

def withActorSystem(f: ActorSystem => Unit): Unit = {
    implicit val system: ActorSystem = ActorSystem("test", config)
    f(system)
    system.terminate().futureValue
  }

Nothing fancy. Just a helper that start actor system, runs the f function and stops it.

An alternative for the migration test could be the following:

def withActorSystem(config: Config)(f: ActorSystem => Unit): Unit = {
    implicit val system: ActorSystem = ActorSystem("test", config)
    f(system)
    system.terminate().futureValue
  }

Then the tests:

it should "migrate the event journal" in {

	// start 1st actor system with legacy schema
	withActorSystem(configWithLegacySchema) { sys => 
		// populate db 
	} 

    // start a new actor system using new schema 
    withActorSystem(configWithNewSchema) { sys => 
        // run migration tool

		// instantiate some persistent actors 
        // and verify journal is replayed as expected
	}		
}

That's what I would try out first. And then, following the structure we have in our tests.
We always have an abstract test that contains everything we want to test, then there is a H2 impl that runs as unit tests. The other DBs are run as integration tests against docker containers.

octonato · 2021-10-11T19:52:12Z

I will merge this PR into branch migration-tool. As such others can continue contributing to it.

Note, the code here isn't ready. Tests are not passing and we probably need to run formatting task and other things. I'm doing so because I understood that @Tochemey won't have time to continue working on it for now and others (@flonedo / @gabriele83) would like to pick from here.

So, @flonedo / @gabriele83, few free to open PRs pointing to the new migration-tool branch.

gabriele83 · 2021-10-11T19:53:52Z

Ok 👍

Tochemey added 5 commits February 5, 2021 18:00

WIP

17490ac

WIP

32c7b75

Refactoring 🍻

6fc16ae

Renaming and fix Java collectors 🍻

e824d21

Set explicit type for some weird reasons for scala 2.12 compilation

c49166c

octonato reviewed Feb 11, 2021

View reviewed changes

Tochemey added 3 commits February 13, 2021 11:09

Address some reviews comments

1e3e8e9

Remove schema utils

3cd2e3d

Fix convertion issue

483df26

Tochemey marked this pull request as ready for review February 13, 2021 11:39

Tochemey requested a review from octonato February 13, 2021 11:39

Tochemey changed the title ~~Better SchemaUtils and Data Migration tool~~ Migration tool Feb 13, 2021

Tochemey added 2 commits February 13, 2021 11:45

Rename module name to reflect his job

09f32d1

Fixes

e102ae0

Fix some issue

8aa8542

Tochemey added 5 commits February 15, 2021 08:28

Merge branch 'master' of github.com:Tochemey/akka-persistence-jdbc

712a2fc

# Conflicts: # migrator/src/main/scala/akka/persistence/jdbc/migrator/LegacySnapshotDataMigrator.scala

Merge branch 'master' of github.com:Tochemey/akka-persistence-jdbc

85cfb9b

# Conflicts: # migrator/src/main/scala/akka/persistence/jdbc/migrator/LegacySnapshotDataMigrator.scala

Wow

a39b5e8

Remove event adapter

a228672

Cleanup

0a60fef

octonato reviewed Mar 8, 2021

View reviewed changes

Tochemey added 2 commits March 8, 2021 20:12

Remove the old migration project

9a84ae4

Remove unecessary dependencies

1ac0e5e

Tochemey added 2 commits March 17, 2021 14:32

Basic check

e805b48

🚑

3de5100

octonato reviewed Mar 22, 2021

View reviewed changes

Tochemey added 4 commits March 23, 2021 08:30

refactoring

651e768

add license header

14d0f3d

Merge branch 'master' into master

6095df9

checkin

c169ea7

octonato mentioned this pull request Jul 23, 2021

Change the default for akka-persistence-jdbc.logicalDeletion.enable #557

Open

ennru mentioned this pull request Sep 29, 2021

Migration tool - 3.5.x => 5.0.0 #317

Open

gabriele83 reviewed Oct 1, 2021

View reviewed changes

fix non-exhaustive pattern matching

d5fad3a

Merge pull request #1 from gabriele83/master

f7e483d

fix non-exhaustive pattern matching

Merge branch 'master' into master

3531def

octonato changed the base branch from master to migration-tool October 11, 2021 19:48

octonato merged commit 85880e4 into akka:migration-tool Oct 11, 2021

This was referenced Oct 11, 2021

enable build on branch #602

Merged

Migration tool #603

Merged


		import scala.concurrent.{ ExecutionContext, Future }

		package object migrator {

Migration tool #501

Migration tool #501

Conversation

Tochemey commented Feb 5, 2021 • edited Loading

Tochemey commented Feb 7, 2021

octonato left a comment

Choose a reason for hiding this comment

octonato Feb 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

octonato commented Feb 12, 2021

Tochemey commented Feb 13, 2021

Tochemey commented Feb 14, 2021 • edited Loading

Tochemey commented Feb 14, 2021

Tochemey commented Feb 23, 2021

octonato commented Mar 8, 2021

octonato left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

octonato left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flonedo commented Sep 27, 2021 • edited Loading

gabriele83 commented Sep 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

octonato commented Oct 5, 2021

gabriele83 commented Oct 5, 2021

Tochemey commented Oct 5, 2021

Tochemey commented Oct 5, 2021

Tochemey commented Oct 5, 2021

flonedo commented Oct 8, 2021

octonato commented Oct 11, 2021

octonato commented Oct 11, 2021

gabriele83 commented Oct 11, 2021

Tochemey commented Feb 5, 2021 •

edited

Loading

octonato Feb 11, 2021 •

edited

Loading

Tochemey commented Feb 14, 2021 •

edited

Loading

flonedo commented Sep 27, 2021 •

edited

Loading