Help Users Navigate User Guide Code

[mslinn]

Users who are just getting started with Eventuate, like me, would find it very helpful to work through the code examples in the User Guide. Following are suggestions to make that experience better:

1. The User Guide should say that the code exists at https://github.com/RBMHTechnology/eventuate/blob/master/src/sphinx/code/UserGuideDoc.scala.
2. It would be better if this code was a series of small projects, so the dependencies would be better understood, and so the reader would know that each project is independent.

Following my own suggestions, I made a GitHub repo for UserGuideDoc.scala:

• I refactored the User Guide code into separate Scala files for easier reading. One big file for the entire chapter is painful to figure out.
• Updated to Scala 2.12.1 and SBT 0.13.13.
• Ascribed types where they were not declared.
• Applied minor reformatting, while attempting to not disturb the Sphinx comments.
• Created application.conf for Akka configuration.

In doing the above, I discovered that the mysterious LevelDB mentioned in the docs was actually a Scala port of the Google LevelDB project, originally written in C++. It would be good to tell readers early on that LevelDB is a fast key-value storage, originally developed by Google, which provides ordered mapping from ByteArray keys to ByteArray values.

When the sample project is ready, it might be a good idea to move the repo over to a subproject under RBMHTechnology/eventuate, and replace UserGuideDoc.scala, so others can have a quicker start.

[krasserm]

@mslinn Thanks a lot for your improvement suggestions. They completely make sense to me. I'd highly appreciate if you could make a pull request with your proposed changes and to replace UserGuideDoc.scala with the those developed in your repo and mention them in the user guide.

Updated to Scala 2.12.1 and SBT 0.13.13.

We still need to default to Scala 2.11 because of the dependencies in the Spark Adapter. http://rbmhtechnology.github.io/eventuate/adapters/spark.html. Although we're additionally releasing 2.12 versions of Eventuate without the Spark adapter we can only default to Scala 2.12 when all Spark dependencies support that as well.

In doing the above, I discovered that the mysterious LevelDB mentioned in the docs was actually a Scala port of the Google LevelDB project, originally written in C++.

We're actually using the C++ version of LevelDB with a JNI wrapper and not a Java port of LevelDB.

[mslinn]

Happy to make a PR when ready. I have a lot to learn!

1. I have copied the Sphinx user-guide.rst into my project. I have not used Sphinx before but my intention is to make improvements to the source code and the docs together, then later put the files back to the original project, hopefully as an SBT subproject.
2. What is EventsourcedActorsUpdated.scala for? It is not referenced by anything anywhere.
3. What is EventCommunication for? It is not referenced by anything anywhere.
4. I created application.conf to configure Akka, but encountered the need for a serializer in order to suppress a message: "Using the default Java serializer for class [doc.EventsourcedActors$Appended] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'". I found https://rbmhtechnology.github.io/eventuate/reference/event-sourcing.html#custom-serialization but would appreciate someone taking care of that for me with a PR against my git project.

[mslinn]

Martin, Is there a way to produce a formatted link using Sphinx? I would like to generate: <a href="http://somewhere.com">*<code>*text*</code>*</a> It is easy to produce an unformatted link; I know two ways: Method 1: Here is some `text <http://somewhere.com>`_. Method 2: Here is some `text`_. .. _text: http://somewhere.com I've searched and experimented to produce a formatted link using Sphinx, but failed. The docs would be better if they were properly formatted. Thanks, Mike -- <http://micronauticsresearch.com> *Michael Slinn*, Principal Twitter <http://twitter.com/mslinn> • Blog <http://blog.mslinn.com> • GitHub <https://github.com/mslinn> • LinkedIn <https://www.linkedin.com/in/mslinn> • Resume <https://www.micronauticsresearch.com/#people> Direct: +1-650-560-8771 Cell: +1-650-678-2285 /Get the edge with our online training. Learn Scala today! <https://www.scalacourses.com>/

[krasserm]

... then later put the files back to the original project, hopefully as an SBT subproject.

Making the Sphinx code examples a separate sbt subproject probably won't work as this directory is already an unmanaged source in the settings of the root sbt project. At the moment, Eventuate documentation references code examples at three different kinds of locations:

• in src/sphinx/code. Contains examples that only have core, logLeveldb and logCassandra as dependencies.
• in eventuate-example*: More examples with more dependencies than in src/sphinx/code.
• in src/main of individual modules. References to Eventuate main code directly.

We could either stay with that pattern or decide to move all examples from src/sphinx/code to eventuate-examples. This could make it easier to execute the examples as eventuate-examples is a separate sbt subproject.

EventsourcedActorsUpdated ... EventCommunication ... It is not referenced by anything anywhere.

It is referenced by .rst files to be included as code snippets. Main purpose of the code in src/sphinx/code is to have example code that compiles i.e. if there's a compile error in one of the examples, the documentation build (i.e. sbt makeSite) fails too.

... need for a serializer ...

I think it's completely ok to use Java serialization for the examples together with a akka.actor.warn-about-java-serializer-usage = false configuration setting.

I've searched and experimented to produce a formatted link using Sphinx, but failed.

Me too, didn't find a way to generate formatted links. We therefore followed the convention that links are always unformatted.

[mslinn]

1. I found that the stylesheets were peculiar and suboptimal, so I made the minimum changes necessary for them to be easier to read without destroying the current look and feel. These changes will affect every page of the documentation site.

2. I now understand why the user guide source needs to be bundled alongside the rest of Eventuate. However, the source should not be buried deep within. Lets put all the example projects under one top-level directory, called examples, and put any old code examples under examples/old so people know what to expect. If/when the old code examples get updated they would be graduated up a level, for example from examples/old/interactive-confict-resolution to examples/interactive-confict-resolution. If any other documents currently under code (EventLogDoc.scala, EventRoutingDoc.scala,EventSourcingDoc.scala, and ReliableDeliveryDoc.scala) also contain source code, those should eventually be migrated to the examples/ directory (note I have not volunteered for this task, separate issues would need to be created for each document.)

eventuate/
├── examples
│   ├── adapter
│   │   ├── spark
│   │   ├── stream
│   │   └── vertx
│   ├── old
│   │   └── interactive-confict-resolution
│   └── user-guide
└── other

Notice that the directory names are short, and form a descriptive relative path name that does not contain redundant use of the word eventuate. For example, examples instead of eventuate-examples, and examples/adapter/spark instead of eventuate-examples/eventuate-examples-spark.

3. SBT's multiproject support improved with v0.13.13. The example programs should each be standalone SBT projects, with top-level .sbt files using SBT 0.13.13+. This means users will easily be able to understand the minimal requirements to make each program work, instead of the current undocumented/magical/complex multi-project setup.

4. Not only should the code examples be easy to find and their usage and status be intuitive, they should work without odd errors or output that diminishes a potential user's confidence.

   a. The serializer message falls into this category. It should be addressed. There is only one chance for a first impression.

   b. It is clear that none of the User Guide's Java programs have ever been runnable from a command line. If Java support is a priority then this should be addressed, otherwise Java support should be labeled as being a second priority, and one in need of volunteers.

   c. Most of the Scala programs do not do anything or produce output that helps a user understand the points made in the document. The exception is the first program (ActorExample.scala). I'm not going to take this task on – someone who knows the project could probably do this fairly quickly. I would be happy to discuss the reader's needs from the point of view of the User Guide with that person.

This last point needs to be done before I can complete my task of updating the user guide, so I have made the directory changes described above and merged my work to date into a new branch (newDocs) of my fork of Eventuate. I am now awaiting a volunteer to polish the Scala code examples. If that person, or another volunteer, then moves on to polish the Java code examples, and makes the necessary changes so the Java programs match the corresponding Scala program in terms of functionality, I will rewrite the User Guide to suit.

Update Mar 20/17 11:24am PT: Because I have made changes to the Scala and Java programs, the person/people who assume the task of updating them should extend my work; as of right now the most current version of the sample programs is found in my Eventuate fork in the newDocs branch.

[krasserm]

@mslinn thanks for all the effort you put into it so far. Looks good to me at a first glance and would like to make a more detailed review in several smaller pull requests instead of a single large one. For example PRs could cover:

• restructure examples to have only a single examples root and update the references in the documentation accordingly.
• improve the user guide as you did already in your fork.
• extend the examples to make them self-contained and runnable with all the improvements you suggested. Maybe these can be separate PRs for Scala and Java examples. I cannot promise that I or another volunteer will have the time in the near future to work on that as I'm working on other high-priority tasks at the moment.
• further improve to the user guide based on what was done in the previous PR.
• stylesheet enhancements. Here we need to make sure that the changes are still compatible with RBMH requirements and style guidelines. I will have to check that.
• ...

This procedure will allow us to make progress in several smaller steps and easier for others to review and discuss changes. Regarding the code examples and the separate sbt root: we need ensure that the examples still compile when generating the documentation. Not sure if this is possible if the documentation and the examples use different sbt roots (but I'm not an sbt wizard and that's only a minor issue).

[mslinn]

Makes perfect sense.

• I already updated all references throughout the entire project, including the references in the documentation. My fork has Travis set up to work with the new branch I created and it passes. The badge is shown here and this means the root project still compiles.
• You can check the style sheet changes by cloning my fork, switching to the newDocs branch, and running the following. View target/html/user-guide.html in a web browser.

    sphinx-build -b html src/sphinx/ target/html/ &> sphinx.log

• To begin with I could make a PR with just the two stylesheets I modified. A small thing, easy to deal with.

[krasserm]

To begin with I could make a PR with just the two stylesheets I modified. A small thing, easy to deal with.

Please do that, thanks!

[krasserm]

@mslinn please sign the CLA when making your first PR.

[mslinn]

I made Akka serializers and deserializers for the Scala versions of VectorTime and Versioned, and wrote tests. This code probably belongs in the main project, since everyone probably would benefit from a simple way to serialize/deserialize these objects. If the code was ported to the main project, the implicit classes would not be required.

I also updated the User Guide code in my fork to take advantage of these serializers / deserializers, plus wrote a few more to handle Int and Long.

I did not make Java serializers or deserializers for VectorTime or Versioned. It is not hard to do, but I get the sense it is not important so I did not bother. However, application.conf is already set up, just need to uncomment the entries if and when they are implemented.

[krasserm]

eventuate-core already contains a protobuf serializers for VectorTime and Versioned (protobuf definition). In general, all Eventuate objects that are persisted to an event log or sent over the network have a corresponding protobuf serializer (see also custom serialization).

I'm not sure if it is helpful to have custom serializers for all the messages/events used in the examples (as it bloats them significantly). As a user I'd find a single/separate example, how to write and configure a custom serializer more helpful, especially if these are protobuf or Avro serializers, as they are needed for supporting event schema evolution.

[mslinn]

Thanks for pointing me to the serialization documentation. I agree that the User Guide code bloats when serialization is added (seems to be at least 2x larger).

I did the following on my newSerializeDocs branch:

• Added a comment to the User Guide that states "The above serialization warning is addressed in the Custom Serialization document."
• Moved some (not all) of the serialization code that I wrote yesterday to a new SBT project in examples/serialization, and deleted the rest of the serialization code from the User Guide.

When ready, I'll make a new PR for the changes to the Serialization document, where we can address any issues that may exist with the code I wrote, and discuss best practices. For example, I used org.apache.commons.lang3.SerializationUtils for serializing objects that can be cast to Serializable, and java.nio.ByteBuffer for serializing Int and Long. If this is inadvisable, a few sentences about why this is so should be added, and guidance provided.

BTW, I could not find any reference to Apache Avro in the Eventuate code base. Presumably Avro is merely one serialization option, in addition to Google Protocol Buffers / ScalaPB, Apache Thrift / Scrooge, Pickling, Kryo / Chill, SerializationUtils, and java.nio.ByteBuffer.

BTW, In a performance benchmark, ScalaPB provided the overall fastest results, which is nice, because it is merely a Scala wrapper around Google Protocol Buffers.

[krasserm]

Sounds good.

where we can address any issues that may exist with the code I wrote, and discuss best practices

Let's only add an example how to write a custom protobuf serializer and remove the serializers that use SerializationUtils (as they just do Java serialization) and low-level ByteBuffer. That's all we need to have a best practices example.

I could not find any reference to Avro in the Eventuate code base

Eventuate uses protobuf only. Applications can use whatever they want.

[mslinn]

I updated my newSerializeDocs branch from master and pushed to my eventuate fork. There are lots of changes. I don't know how the restructuring could be done in smaller pieces without doubling the work.

1. Code examples are all standalone SBT projects that compile, and are located under examples.
2. All of the various documents have been edited to point to the new location.
3. Some of the code examples were quite long, so I broke them into smaller chunks and updated the markup.
4. The build passes
5. I wrote comments with FIXME and README so you can find them.

[krasserm]

@mslinn instead of pointing to changes in other repos, please make one or more PRs and let's discuss things there. Thanks for your understanding