Improvement ideas

• Benchmark results should be pushed to a Git repository from CI job
• Add plot for Generation time (since it uses VIATRA Queries and had a huge regression after DRed merge)
• Plots should use common scaling (10^0, 10^2, ...) to make results comparable between runs ( https://github.com/FTSRG/mondo-sam/blob/0.1-maintenance/reporting/shiny/plots/PlotContainer.R#L217-L226 )
• Use Java code to process raw results (e.g. with https://github.com/eclipse/january )
• Maybe use some Java based tool for generating diagrams
  • http://www.jfree.org/jfreechart/ seems to be the only option
  • see also: https://stackoverflow.com/questions/265777/what-is-the-best-open-source-java-charting-library-other-than-jfreechart
  • it may be that R (ggplot) or Python ( https://matplotlib.org/ ) is still better choice
• Make industrial query benchmark output benchmark results in MONDO-SAM format (no need to use the benchmark engine itself)
• Consider adding a separate data preprocessing before plot generation (to make it easier to browse numeric results as well for quick comparisons)
• Model statistics should be stored in results (currently it is only partially visible in the logs)
• Write Jenkins plugin for displaying/generating Trend diagrams (similarly to build time in https://build.incquerylabs.com/jenkins/job/viatra-cps-benchmark/buildTimeTrend )

Benchmark results repository structure

Location e.g. github.com/viatra/viatra-cps-benchmark-results

Files are in italic, folders in regular case.

• ReadMe.md: Link to config index files and general information on repository
• m2m
• m2m-reduced
• <config-name>
  • config-name.md: List of results (linking to details) with basic info (date, execution environment, VIATRA repo URL, VIATRA revision)
  • 201704251200-build67
  • 201705051600-build91
  • <date>-build<build-number> (or other unique identification)
    • result files without diagrams (or maybe only PDF, if vector graphic), including logs, separate JSON files, combined JSON and CSV
  • trend-20170505-dred-performance-optimization
  • trend-<date>-<id>
    • Store derived results from trend reports (comparison of different runs)

It would be nice to use Github Pages for browsing the results, but that can wait.