Unify tutorials as scientific notebooks

[ctrueden]

We have tutorials in several places throughout the SciJava GitHub organizations and on the ImageJ wiki. Our goal is to create a coherent, ordered set of tutorials, through which new people can proceed in order to understand how to use the various APIs of the SciJava software stack.

Existing tutorials to unify

The following places have tutorials which we want to unify:

• imagej/imagej-tutorials (this repository). These are currently structured as individual Maven projects, written in Java.
• imglib/imglib2-tutorials. These are structured as a single Maven project, written in Java. They have a corresponding wiki page here. We probably want the contents of the wiki page to become the contents of the Beaker notebook. I.e.: we do not want to need a wiki page corresponding to each Beaker notebook.
• scifio/scifio-tutorials. Structured as one Maven project for the core tutorials, and another one for the OME-based ones. We can likely eliminate that distinction—just migrate the OME-XML tutorial into the sequence somewhere.
• imagej/imagej-scripting. These tutorials are written for the Script Editor. They use script parameters. We will need to carefully consider whether and how to convert them to Beaker notebooks. Perhaps we should have a separate set of tutorials for the Script Editor, centered here. It is for a potentially different audience.

How to unify?

We have initially settled on migrating everything to Beaker notebooks, with code written as Groovy nodes. We hope this will be more approachable for less experienced programmers, because we can then ignore many typing issues—especially ImgLib2's difficult recursive generics (e.g., T extends RealType<T>).

However, there are tradeoffs with this approach. Other possible unifications strategies we have considered, and which we may need to address separately somehow in some cases, include:

• Make examples available inside ImageJ's Script Editor. This would have two big advantages: 1) tutorials shipped with ImageJ itself; 2) tutorials can use script parameters for convenience and succinctness.
• Proper Maven projects with pom.xml and standard directory structure. The advantage of doing this is that some programmers really want to develop a Maven project, and want to know how to use ImageJ from that project. So the existing Maven examples actually illustrate that need very well.

Switching to Groovy will stop covering the above use cases. So we may need to continue to provide examples in the above two paradigms as well somehow.

The nice thing about the Groovy approach is that it works "standalone"—you create your ImageJ application context, and use it to do things, and that logic can be embedded in external software which is otherwise unrelated to ImageJ. Many programmers want to know how to do this. However, most scientific script writers do not need this, and instead want to write scripts that run from inside ImageJ.

In a perfect world, we would somehow autogenerate all three of these paradigms (Beaker, Script Editor, Maven project) from some common source. However, I currently believe doing that is infeasible. You can make a Maven project which builds Groovy code. However, that is of limited use for the people who want a Java-based Maven project. In general, converting code between languages has many pitfalls and edge cases.

So for now: we put everything into Beaker. Our goals are:

1. One Beaker notebook per tutorial.
2. Impose an order on the whole set of tutorials—i.e., number everything. Currently, some of the above are already ordered, and others have no stated order. There are many reasons that an ordered set of tutorials is more approachable.
3. Use the advantages of the notebook where possible to effectively teach. In other words: these are not just big "bags of code"—we have text nodes as well. Use the notebooks as notebooks. E.g., they support Markdown syntax.
4. Use the advantages of Groovy. It has many nice "syntactic sugar" etc. above and beyond what Java provides. We want the tutorials to focus on teaching the SciJava and ImageJ concepts, without lots of distracting boilerplate. While we are at it, they can have little "gems" which show off how convenient scripting languages are.
5. Put all information in a single place, linked from the wiki centrally, so that everyone finds all these tutorials, instead of needing to hunt around in different places.

Open questions

• Right now, the ImageJ tutorials do not emphasize that ImageJ is built on SciJava Common. E.g., there is a tutorial called intro-to-imagej-api which actually demonstrates core SciJava services—i.e., it does nothing imagej-specific whatsoever. Is this distinction important for our target audience(s)?
• How to divide up the effort of migrating these tutorials between @panovr and @etarena?
• How to cater to all target audiences as simply as possible? I.e.: which paradigm(s) are worth targeting?
  • Programmers need to see: A) how to make a Maven project; B) how to make a Gradle project; C) how to call from Java instead of Groovy; D) how to use the generics where applicable; E) how to create a new SciJava application context and propagate it.
  • Scripters need to see: A) how to write scripts which run from inside ImageJ/SciJava; B) how to write scripts which run from inside scientific notebooks like Beaker; C) how to write scripts which run standalone, on a cluster, etc.; D) how to use existing services and plugins from the current context, available as input parameters.
• Is it worth targeting Jupyter notebooks as well? GitHub renders them, which is awesome, but they do not support Groovy out of the box (it might be possible via the JSR-223 Jupyter plugin, but that plugin is immature). We could use the Scala Jupyter kernel (see these tutorials by @kmader, which do so for a couple of ImageJ tutorials), but language translating between Groovy and Scala is very likely untenable, or at least extremely ugly. (Scala's type system is very sophisticated and complex—any automated translation from Groovy would probably use lots of raw types, which is highly un-Scala-esque.)
• What content should live on the wiki? People look there for information. We can certainly link to the Beaker notebooks from the wiki, of course. But maybe we could be slicker: e.g., embed the Beaker notebook in the wiki page somehow...?
• What tutorials are still missing from this whole bag of stuff? Surely there are a few/many which we never wrote, but need. That is really a separate issue from this one, but is worth thinking about at the same time this work is being done.

Where to start

The work is being done on the dev-manual branch of this repository.

To complete this work, that branch needs a Beaker notebook for each of the above linked tutorials, in a coherent order. So one approach is:

1. Convert all tutorials to Beaker. Copy/paste, fix syntax errors, get them working.
2. Decide on the order. Number everything. I.e.: make a table of contents.
3. Clean up the notebooks to use more features of Groovy, and features of Beaker.
4. Identify and isolate common repeated code, and split out that code into common helper methods of appropriate components. For example, all notebooks will need to display images which show the results of operations. Beaker can display Java BufferedImage objects inline seamlessly. But we need a helper method which takes an ImgLib2 data structure + plane position as input and produces a BufferedImage as return value, so that Beaker will show it. The first attempt at this code exists on the branch now, inside several notebooks. It needs to be factored out into an external method.

[ctrueden]

Progress: 886d264