YAMTL batch solution for VIATRA CPS Benchmark

[arturboronat]

Dear VIATRA CPS benchmark maintainers,

My name is Artur Boronat from the University of Leicester and I have been working with the VIATRA CPS benchmark during the past recent months. Specifically, I would like to contribute a new (batch) solution based on YAMTL (https://yamtl.github.io/), our new MT engine.

In order to familiarize myself with the benchmark framework, I extracted relevant parts from it and used them in independent projects, which are publicly available in this GitHub repository. The main differences are that dependencies to the VIATRA CPS benchmark framework have been reimplemented:

• the different solutions have explicit runners that can be executed locally;
• models used in the local experiments were generated with the VIATRA CPS generator but are fixed;
• a different benchmark harness is used for enabling local experimentation;
• a test driver has been developed.

These changes also mean that the actual results obtained, although valid, may differ from the ones that the live benchmark will obtain. I would be very grateful if someone could integrate my solution into the live benchmark and let me know how to inspect the results.

The solution that I would like to contribute is this one, where the readme file explains where to find the relevant classes, which should facilitate the integration of the solution into the live benchmark. Note that this solution only implements the batch component of the benchmark and, thus, it only makes sense to compare it with the other batch solutions.

I have checked that the contributed solution passes the VIATRA CPS sanity checks from org.eclipse.viatra.examples.cps.xform.m2m.tests. However I had to make a few changes as explained here. Let me know if those changes sound acceptable.

Please let me know if you need any help in case you'd encounter problems during the integration process. I will be glad to discuss the results.

Thank you very much in advance.

Kind regards,
Artur

[abelhegedus]

Dear Artur,

thank you very much for reaching out and especially for choosing the VIATRA CPS benchmark for your tool.
We will look into the integration of your solution and the various points that you raised regarding the sanity checks.
Since you added YAMTL as a jar dependency and all other requirements (EMF, Xtend) should be already available in our target platform, I hope the integration will not be too cumbersome.

I can't tell exactly when we will be able to work on this, but if you are familiar with Eclipse IDE, Oomph, and Maven/Tycho, you should be able to get a development environment set up for the complete benchmark as described here and try to create an initial Pull Request. If you run into any issues, please let us know.

Cheers

[arturboronat]

Dear Ábel,

Thank you for your prompt reply.

I'm afraid I did not get too far with the Oomph setup when I tried to install it. That's why I opted for dissecting the framework in order to reuse relevant components.

On the one hand, the framework is quite comprehensive, which also means that it is reasonably complex to extend by someone who has not participated in its development.

On the other hand, I don't know how a solution for the benchmark can be tested and run locally - without using the Jenkins server.

Presumably, the integration can be done fairly quickly by an expert in the framework. This seems to be the most effective way of implementing the integration. I will be happy to help if there is any problem.

Thanks again!

[arturboronat]

I had another go at installing the benchmark using the Oomph installer.

In the advanced mode, regarding variables in the Contributor's guide, I could not find where to set them up. Some guidance would be helpful - I looked at Oomph's user guide but did not find where to set them up.

In the simple mode, going ahead with the installation without setting those variables the installer shows this output log.

Thanks.

[abelhegedus]

Dear Artur,

I'm sorry for not replying for so long, but I have not had any time to work on the CPS benchmark. However, I have managed to find time to work out a few problems related to setting up the development environment.

1. I have done some fixes, you should be able to get the Eclipse Installer to set up the IDE following the steps in https://github.com/viatra/viatra-cps-benchmark/wiki/Contributor's-Guide
2. I have also done some fixes on the VIATRA CPS demo, but that is only available as a Gerrit change, should be merged shortly: https://git.eclipse.org/r/c/127065/1
3. I have implemented your suggestions for the test suite, but since that is your code if we are completely fair, I would like to have you signing off the following Gerrit change: https://git.eclipse.org/r/c/127066/1 for that, you should have an Eclipse.org account and a signed ECA.

The next step will be to pull your code into the CPS benchmark repository on a new branch, subclass each test class in cps m2m.tests to make them run with a transformation not part of the VIATRA CPS example and wire it into the rcpapplication product. Alternatively, the m2m.tests code should be refactored a bit in a way that the whole test suite can be run with a provided transformation wrapper and reversing the dependency between m2m.tests and the various transformation implementations (their test harness code to be precise).

I will see when I will have time to work on the next steps, but you are still welcome to go ahead if you got the IDE set up with the Eclipse Installer.

[abelhegedus]

I have integrated the YAMTL implementation on branch issues/23. The functional tests are not yet executed and the support in the benchmark application is hard-coded, but it ran without errors on my machine. I'm currently in the process of getting the branch built on our CI server.

Note that it would be nice if YAMTL would be available as OSGi bundle from a P2 update site with proper dependencies instead of having a Maven jar that is an OSGi bundle wrapped in another OSGi bundle in order to export the packages correctly and integrate into a Maven/Tycho build.

[abelhegedus]

The first successful benchmark run with the integrated YAMTL solution is published here: https://build.incquerylabs.com/jenkins/job/viatra-cps-benchmark/job/issues%252F23/18/artifact/benchmark/cpsBenchmarkReport.html (separate diagrams in https://build.incquerylabs.com/jenkins/job/viatra-cps-benchmark/job/issues%252F23/18/artifact/benchmark/diagrams/ )

[arturboronat]

Dear Ábel,

Thank you very much for your support.

At the moment, I am working on an incremental version of the transformation, which should be ready soon (towards the end of this week). It would be great if we could use this momentum to integrate it as well.

Regarding the update site, I made a deliberate decision to use plain Java dependencies to develop YAMTL. I'm afraid I am not planning to configure an update site as a default option. If needed, a basic wrapper update site can be developed but that would add one more layer to the current onion.

Regarding the results above, the report refers to BATCH_VIATRA_QUERY_RETE-TEMPLATE, is that correct? Can the benchmark be generated for larger sizes and contextualized with other results?

Thanks again.

[abelhegedus]

At the moment, I am working on an incremental version of the transformation, which should be ready soon (towards the end of this week). It would be great if we could use this momentum to integrate it as well.

Sure, you can pull the branch and push commits on your fork or just let me know when to update the yamtl.jar and the transformation implementations.

Regarding the update site, I made a deliberate decision to use plain Java dependencies to develop YAMTL. I'm afraid I am not planning to configure an update site as a default option. If needed, a basic wrapper update site can be developed but that would add one more layer to the current onion.

Sure, I understand, as long as the standalone yamtl.jar is available, it can be updated in the yamtl.bundle plugin.

Regarding the results above, the report refers to BATCH_VIATRA_QUERY_RETE-TEMPLATE, is that correct? Can the benchmark be generated for larger sizes and contextualized with other results?

Sorry, I linked the wrong build, this is the one with YAMTL: https://build.incquerylabs.com/jenkins/job/viatra-cps-benchmark/job/issues%252F23/18/artifact/benchmark/cpsBenchmarkReport.html

The configurations are stored in the scripts/config folder, the benchmark is controlled by the data.json file. This configuration uses our "statistics based" model generation case and runs the most relevant tools. You can take a look at the other configurations (especially m2m) and propose another configuration that you would like us to run.

[arturboronat]

Dear Ábel,

Thank you very much for the updates - the results are consistent with those obtained in YAMTL's adaptation of the CPS benchmark for batch transformations.

On the other hand, apologies for not having pushed the results yet. My intention is to do this by extending the framework directly if the configuration works (according to Contributor's guide at https://github.com/viatra/viatra-cps-benchmark/wiki/Contributor's-Guide). This time, Eclipse was installed with Oomph but the following undocumented steps were found:

Sirius

Installation of Sirius 6.0 is recommended. I tried the installation with and without it, and this step seems optional.

Workspace configuration

The workspace is not configured automatically and the git repository included in the Git perspective does not contain code about the CPS benchmark.

After cloning https://github.com/viatra/viatra-cps-benchmark, when importing the Maven project with pom.xml at git/viatra-cps-benchmark as shown in
this figure, the message in this figure is displayed.
The problems were not resolved after installing Tycho Project Configurators.

Any guidance on configuring the workspace so as to be able to extend it directly would be appreciated. After running mvn verify from command line on the root project folder, Maven produced this error log, with the following message:

[ERROR] Cannot resolve project dependencies:
[ERROR]   Software being installed: com.incquerylabs.examples.cps.performance.tests 2.0.0.qualifier
[ERROR]   Missing requirement: com.incquerylabs.examples.cps.performance.tests 2.0.0.qualifier requires 'bundle org.eclipse.viatra.examples.cps.queries 0.1.0' but it could not be found

The problems that are reported by Eclipse after importing viatra-cps-benchmark with Maven-Tycho are here.

Thanks again!

[abelhegedus]

The workspace is not configured automatically and the git repository included in the Git perspective does not contain code about the CPS benchmark.

Did you make sure to check both VIATRA CPS Example and CPS Benchmark on the projects page?

After cloning https://github.com/viatra/viatra-cps-benchmark, when importing the Maven project with pom.xml at git/viatra-cps-benchmark

Importing as a Maven project was never tested and is not expected to work. The Oomph setup is meant to import all projects and their dependencies are provided by the Target platform.

[arturboronat]

Did you make sure to check both VIATRA CPS Example and CPS Benchmark on the projects page?

I followed your instructions: https://github.com/viatra/viatra-cps-benchmark/wiki/Contributor's-Guide

The Oomph setup is meant to import all projects and their dependencies are provided by the Target platform.

Then something did not work in the installation process. After inspecting the progress error log, the error seems to be due to

Cloning Git repo git@github.com:FTSRG/mondo-sam.git to /Applications/eclipse/viatra-cps-benchmark-master/git/mondo-sam
java.lang.Exception: org.eclipse.jgit.api.errors.TransportException: git@github.com:FTSRG/mondo-sam.git: Auth fail

I have to leave the office for the rest of the day and will get back to this next week.

[abelhegedus]

git@github.com:FTSRG/mondo-sam.git: Auth fail

You may have used a read-write clone url instead of the read only.

[arturboronat]

Thanks and, on the contrary, after setting MONDO-SAM and CPS Benchmark to HTTPS read-write, the installation process worked.

Just some final issues:

After checking out the branch issues/23 on viatra-cps-benchmark, the plug-in m2m.batch.cps2dep.yamtl.tests;bundle-version="2.0.0" cannot be found. This error appears (on the Java package explorer view) in CPS Benchmark/com.incquerylabs.examples.cps.rcpapplication.headless/META-INF/MANIFEST.MF (line 15).

After checking out the same branch on MONDO-SAM, the plug-in appears (in the Java package explorer view) under MONDO-SAM/viatra-cps-benchmark.git/bundles/. However, the pom in project Other Projects/eu.mondo.sam.core contains an error at line 30:

Plugin execution not covered by lifecycle configuration: org.apache.felix:maven-bundle-plugin:3.0.0:manifest (execution: bundle-manifest, phase: process-classes)

Finally, which class contains the main method/test to run the benchmark locally?

[arturboronat]

Dear Ábel,

In the meantime, I have pushed a first version of the incremental YAMTL variant to YAMTL's repository. It consists of two new projects that should allow us to compare the performance of YAMTL both in normal and incremental execution modes with the incremental variant of the CPS2DEP transformation. Both projects are passing the benchmark sanity tests and instructions to execute them can be found in their corresponding readme file. Below there are pointers to the projects and to relevant code:

• Incremental variant (run in batch mode) INCR_BATCH_MODE_YAMTL-TEMPLATE
  • Transformation
  • Example to run the transformation
• Incremental variant (run in incremental mode) INCR_INCR_MODE_YAMTL-TEMPLATE: this project may require configuring AspectJ in the Maven POM in order to track dependencies (see how it has been done in build.gradle and in aop.xml)
  • Transformation: same as above
  • Aspect
  • Example to run the transformation

The solution that was integrated into the benchmark could be renamed to BATCH_ONLY_YAMTL-TEMPLATE to avoid confusion, as this transformation cannot be fully run in incremental mode.

Let me know if there is any problem.

Thank you very much in advance!