Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[stevens] Build from Megamodels #13

Open
grammarware opened this issue Jul 13, 2018 · 7 comments
Open

[stevens] Build from Megamodels #13

grammarware opened this issue Jul 13, 2018 · 7 comments
Assignees
@manuelleduc @hernanponcedeleon @AEReview
Labels
accepted Artefact accepted

Comments

@grammarware
Copy link
Collaborator

Submitted by @PerditaStevens to https://github.com/modelsconf2018/artifact-evaluation/tree/master/stevens

Paper: http://homepages.inf.ed.ac.uk/perdita/MegamodelBuild/modelspaper.pdf
@grammarware grammarware added the submitted Received for review label Jul 13, 2018
@grammarware grammarware added under review Artefact being reviewed and removed submitted Received for review labels Jul 13, 2018
@manuelleduc
Copy link
Collaborator

Dear @PerditaStevens ,

Importing the megamodelbuild project in eclipse was straightforward thanks to maven, and I also easily executed the unit tests to validate that everything was working well.

However, I tried to execute the bxexample.sh script out of curiosity and obtained the following error.

./bxexample.sh 
[INFO] Scanning for projects...
Downloading from pluto-build: https://raw.githubusercontent.com/pluto-build/pluto-build.github.io/master/mvnrepository/build/pluto/maven-config-java/1.1.0-SNAPSHOT/maven-metadata.xml
Downloaded from pluto-build: https://raw.githubusercontent.com/pluto-build/pluto-build.github.io/master/mvnrepository/build/pluto/maven-config-java/1.1.0-SNAPSHOT/maven-metadata.xml (605 B at 525 B/s)
Downloading from pluto-build: https://raw.githubusercontent.com/pluto-build/pluto-build.github.io/master/mvnrepository/build/pluto/maven-config-common/1.1.0-SNAPSHOT/maven-metadata.xml
Downloaded from pluto-build: https://raw.githubusercontent.com/pluto-build/pluto-build.github.io/master/mvnrepository/build/pluto/maven-config-common/1.1.0-SNAPSHOT/maven-metadata.xml (607 B at 1.1 kB/s)
Downloading from metaborg-nexus-snapshots: http://artifacts.metaborg.org/content/repositories/snapshots/build/pluto/maven-config-java/1.1.0-SNAPSHOT/maven-metadata.xml
Downloading from sugar-lang: http://sugar-lang.github.io/mvnrepository/build/pluto/maven-config-java/1.1.0-SNAPSHOT/maven-metadata.xml
Downloading from sugar-lang: http://sugar-lang.github.io/mvnrepository/build/pluto/maven-config-common/1.1.0-SNAPSHOT/maven-metadata.xml
Downloading from metaborg-nexus-snapshots: http://artifacts.metaborg.org/content/repositories/snapshots/build/pluto/maven-config-common/1.1.0-SNAPSHOT/maven-metadata.xml
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building megamodelbuild 0.1
[INFO] ------------------------------------------------------------------------
Downloading from pluto-build: https://raw.githubusercontent.com/pluto-build/pluto-build.github.io/master/mvnrepository/org/sugarj/common/1.7.0-SNAPSHOT/maven-metadata.xml
Downloading from sugar-lang: http://sugar-lang.github.io/mvnrepository/org/sugarj/common/1.7.0-SNAPSHOT/maven-metadata.xml
Downloading from metaborg-nexus-snapshots: http://artifacts.metaborg.org/content/repositories/snapshots/org/sugarj/common/1.7.0-SNAPSHOT/maven-metadata.xml
Downloaded from sugar-lang: http://sugar-lang.github.io/mvnrepository/org/sugarj/common/1.7.0-SNAPSHOT/maven-metadata.xml (766 B at 2.2 kB/s)
Downloading from pluto-build: https://raw.githubusercontent.com/pluto-build/pluto-build.github.io/master/mvnrepository/org/sugarj/maven-config-java/1.1.0-SNAPSHOT/maven-metadata.xml
Downloading from sugar-lang: http://sugar-lang.github.io/mvnrepository/org/sugarj/maven-config-java/1.1.0-SNAPSHOT/maven-metadata.xml
Downloading from metaborg-nexus-snapshots: http://artifacts.metaborg.org/content/repositories/snapshots/org/sugarj/maven-config-java/1.1.0-SNAPSHOT/maven-metadata.xml
Downloaded from sugar-lang: http://sugar-lang.github.io/mvnrepository/org/sugarj/maven-config-java/1.1.0-SNAPSHOT/maven-metadata.xml (604 B at 1.6 kB/s)
Downloading from pluto-build: https://raw.githubusercontent.com/pluto-build/pluto-build.github.io/master/mvnrepository/org/sugarj/maven-config-common/1.1.0-SNAPSHOT/maven-metadata.xml
Downloading from sugar-lang: http://sugar-lang.github.io/mvnrepository/org/sugarj/maven-config-common/1.1.0-SNAPSHOT/maven-metadata.xml
Downloading from metaborg-nexus-snapshots: http://artifacts.metaborg.org/content/repositories/snapshots/org/sugarj/maven-config-common/1.1.0-SNAPSHOT/maven-metadata.xml
Downloaded from sugar-lang: http://sugar-lang.github.io/mvnrepository/org/sugarj/maven-config-common/1.1.0-SNAPSHOT/maven-metadata.xml (606 B at 1.8 kB/s)
Downloading from pluto-build: https://raw.githubusercontent.com/pluto-build/pluto-build.github.io/master/mvnrepository/build/pluto/pluto/1.9.0-SNAPSHOT/maven-metadata.xml
Downloading from sugar-lang: http://sugar-lang.github.io/mvnrepository/build/pluto/pluto/1.9.0-SNAPSHOT/maven-metadata.xml
Downloading from metaborg-nexus-snapshots: http://artifacts.metaborg.org/content/repositories/snapshots/build/pluto/pluto/1.9.0-SNAPSHOT/maven-metadata.xml
Downloaded from pluto-build: https://raw.githubusercontent.com/pluto-build/pluto-build.github.io/master/mvnrepository/build/pluto/pluto/1.9.0-SNAPSHOT/maven-metadata.xml (978 B at 1.7 kB/s)
Downloading from pluto-build: https://raw.githubusercontent.com/pluto-build/pluto-build.github.io/master/mvnrepository/com/cedarsoftware/java-util-pluto-fixes/1.19.4-SNAPSHOT/maven-metadata.xml
Downloading from metaborg-nexus-snapshots: http://artifacts.metaborg.org/content/repositories/snapshots/com/cedarsoftware/java-util-pluto-fixes/1.19.4-SNAPSHOT/maven-metadata.xml
Downloading from sugar-lang: http://sugar-lang.github.io/mvnrepository/com/cedarsoftware/java-util-pluto-fixes/1.19.4-SNAPSHOT/maven-metadata.xml
Downloaded from pluto-build: https://raw.githubusercontent.com/pluto-build/pluto-build.github.io/master/mvnrepository/com/cedarsoftware/java-util-pluto-fixes/1.19.4-SNAPSHOT/maven-metadata.xml (1.2 kB at 2.2 kB/s)
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ megamodelbuild ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 0 resource
[INFO] 
[INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @ megamodelbuild ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ megamodelbuild ---
[WARNING] 
java.lang.ClassNotFoundException: build.pluto.cli.Main
    at java.net.URLClassLoader.findClass (URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass (ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass (ClassLoader.java:357)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:281)
    at java.lang.Thread.run (Thread.java:745)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.050 s
[INFO] Finished at: 2018-07-17T15:27:38+02:00
[INFO] Final Memory: 14M/193M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:java (default-cli) on project megamodelbuild: An exception occured while executing the Java class. build.pluto.cli.Main -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

May I ask if the script is still relevant to the project?

Sincerely,

Manuel Leduc

@PerditaStevens
Copy link
Collaborator

PerditaStevens commented Jul 17, 2018 via email

@AEReview
Copy link
Collaborator

** Summary
The paper presents an approach for sound, optimal and flexible build for megamodels. The work extends the work of Erdweg et talk. on Pluto.
The artifact per se, is made of the implementation of two proof of concept (toy) examples proposed by the author.

As-is, the examples are early stage and are illustrations that are backing the theoretical elements in the paper. They do not aim at implementing and validating the author’s approach on a concrete case study, or elaborate on an example of a reasonable size and complexity.

with regard to artifact evaluation criteria:
** Consistent with the paper
The proposed artifact elaborated on some elements in the paper and represent a proof of concept whose tests are executed correctly. As-is they represent a good complement to the paper.

** As complete as possible
N/A.

** Well-documented
I am missing a small introduction and guidelines towards the toy examples (what to expect apart from the logs, or how to interpret the latter).
Though rather intuitive, I would not consider the current artifact as sufficiently documented for an outsider.

** Easy to (re)use
Though the artifact is exercisable, there is little reusability of the current artifact to expect without large, more complex, and more documented examples.

@manuelleduc
Copy link
Collaborator

Summary
The submitted artifact is part of a paper presenting an approach to improve the compilation chain of model-driven projects by reasoning on mega-models.
It takes the form of a Java project that demonstrates the use of the Pluto build system in the context of the paper by implementing the two proposed motivational examples.

Documented
The documentation is minimalist, the only executable artifacts are the junit tests of the project.
It could be helpful for the user to have some guidelines of what we can infer from the unit test executions.

Consistent
The artifacts are relevant and models accurately the two examples depicted in the paper.

Complete
The artifacts covers the entirety of the use cases covered in the paper.

Exercisable
The project can build imported, build and tested without problems.

Availability
A snapshot of the proposed project have been submitted to ReMoDD.

Minor comments

  • The documentation can be improved easily by integrating an inventory of the provided artifacts;
  • I advise to remove the unused bxexample.sh script;
  • Even if the code is documented, I feel like it could be more introductory by adding some overview of the relevant samples of code to look at and help the reader find the interesting parts, without having to read the whole code base, in the documentation.

@hernanponcedeleon
Copy link

Summary

This artifact accompanies a paper to improve build systems using megamodels and a notion of consistency (consistency can be used for checking which parts of the system need to be re-built).

The artefact consist of Java classes with some documentation and a couple of unit tests.

Assessment

Coverability: Met the expectation

The java classes cover the implementation of the two examples given in the paper.

Packaging: Met expectations

The both the Java classes and the unit test can easily be built using maven.

Reproducibility and consistency: Fell below expectations

I found this artifact very difficult to evaluate in terms of reproducibility or consistency w.r.t the paper.
For the java code, the author claims that the code is expected to be read rather than executed.
However despite the fact that there are classes for each of the examples, I found not way to relate the code (for example the restoreConsistency method of each class) to the paper.
Something similar happens to the unit tests. I found no way (and there is not documentation) to relate this to the paper besides the fact that there is a class with unit tests for each example.

Documentation: Fell below expectations

There is no documentation on what can be done with the artifact besides "reading the code and executing the unit tests". The reviewer has to guess what needs to be done.
Reading other reviewers comments, it seems that several of us were confused and e.g. tried to execute the bxexample.sh script. A nice documentation would have avoided this kind of confusion.
For reading the code, I would have expected some comments on how to related the code, to the paper.
For the unit tests, I would have expected some explanation of what each test is actually verifying or validating.
Also, the documentation is distributed between different README files or comments in different classes, making it very difficult to follow.
Also, it took me a while to find out that the unit test could be executed by running "mvn clean test".

@PerditaStevens
Copy link
Collaborator

PerditaStevens commented Jul 18, 2018 via email

@grammarware
Copy link
Collaborator Author

Dear @PerditaStevens,

Based on all the comments and the reviews provided by the members of the Artifact Evaluation Committee of MoDELS 2018, we have reached the conclusion that this artifact conforms to the expectations and is hereby approved. Please use the badge instructions page to add the badge of approval to your article, and add the link to the ReMoDD entry with URI http://remodd.org/node/582 to the camera ready version of the paper.

Thank you very much for putting extra effort into the preparation and finalising of the artifact. If any of the comments above are still not addressed, please try to accommodate them before the conference. In particular, documentation could be made more reader-friendly and help potential reusers to navigate through the artifact internals.

@grammarware grammarware added accepted Artefact accepted and removed under review Artefact being reviewed labels Jul 20, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@manuelleduc manuelleduc

@hernanponcedeleon hernanponcedeleon

@AEReview AEReview

Labels
accepted Artefact accepted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@manuelleduc @grammarware @PerditaStevens @hernanponcedeleon @AEReview