Specifiy setup for MSL Regression Testing

[modelica-trac-importer]

Reported by leo.gall on 13 Jan 2014 11:07 UTC
As decided at the 81st Modelica Design Meeting, I worked on the details of MSL regression tests.
You can find a draft proposal attached to this ticket.

SetupForMSLRegressionTesting_2014-01-13.pdf

This is an early draft. Any comments to the planned file structure, naming, and the overall test procedure are highly welcome!

@tool vendors: Are you able to submit the proposed file structure?
@Library officers / MAP-LIB: Any information missing? Should the full result file (Dymola: .mat) be stored or not?
@dietmar: Any improvements needed in order to make generation of the overview table easier?

After deciding on file structure and simulation settings, I'm going to create a first set of reference result files.

---

Migrated-From: https://trac.modelica.org/Modelica/ticket/1392

[modelica-trac-importer]

Comment by dietmarw on 13 Jan 2014 14:03 UTC
Before people complain, the correct link in footnote 7 should read:
"gen_fmi_web.py from https://svn.fmi-standard.org/fmi/branches/FMISite/dev/"

Also for section "3.4 Result file sizes" I would say that versioning the reference result data is enough. If people like to check the exact tool result they can simply rerun the simulation.

[modelica-trac-importer]

Comment by jmattsson on 16 Jan 2014 09:16 UTC
It seems a bit unclear which files under TestResults are to be submitted by tool vendors and which are to be generated by MA.

[modelica-trac-importer]

Comment by leo.gall on 16 Jan 2014 11:16 UTC
Replying to [comment:2 jmattsson]:

It seems a bit unclear which files under TestResults are to be submitted by tool vendors and which are to be generated by MA.

You are right, tool vendors only need to submit a subset.

A minimum set of files from tool vendors would be:

• ModelName.csv
• creation.txt
• simulate_*.log

It would be nice to have (additionally):

• ModelName.mat (a full result file, in tool specific format. Does not need to be .mat)
• translate_*.log (translation diagnostics, if not included in simulation log)

I don't have a strong opinion on file names and structure of log files. So, feel free to propose improvements, if it would facilitate your work of generating results.

[modelica-trac-importer]

Comment by sjoelund.se on 21 Jan 2014 11:32 UTC
Note:
2.2 and 2.3 do not line up if Option 1 (Flat) structure is used. It's my preferred choice.
And it's a little odd to have ModelName.csv, but simulate_*.log rather than result.csv.

[modelica-trac-importer]

Comment by sjoelund.se on 21 Jan 2014 11:35 UTC
Making comparisonSignals.txt mandatory would be a nice option since then the tool vendors only needs to download these files to generate their results (the reference results are not needed except if you want to check your results).

[modelica-trac-importer]

Comment by leo.gall on 22 Jan 2014 16:50 UTC
Replying to [comment:4 sjoelund.se]:

Note:
2.2 and 2.3 do not line up if Option 1 (Flat) structure is used. It's my preferred choice.
And it's a little odd to have ModelName.csv, but simulate_*.log rather than result.csv.

Section 2.3 assumes the hierarchical structure which has been preferred in section 2.2.
As already discussed, I don't have a strong opinion on the folder structure.

I'm aware that it is odd to have ModelName.csv instead of result.csv.
In the beginning I used the name result.csv but switched to ModelName.csv.
Reasons:

1. The compare tool uses the CSV file name for generating report file names. So, result.csv leads to stupid report names.
2. When searching for comparison problems (i.e. opening files manually and comparing them in e.g. in Dymola, Matlab, Python), it's more convenient to work with a useful file name instead of a generic name like result.csv.

If more people prefer the flat folder structure, we could switch to that.

[modelica-trac-importer]

Comment by leo.gall on 22 Jan 2014 16:59 UTC
Replying to [comment:5 sjoelund.se]:

Making comparisonSignals.txt mandatory would be a nice option since then the tool vendors only needs to download these files to generate their results (the reference results are not needed except if you want to check your results).

I agree, comparisonSignals.txt could be mandatory for reference files.
During automatic generation, this file is created anyway.
For maintaining the reference files, it would be a little more work for the library developer, if he has to generate new references. The process, how references are going to be created in the future isn't defined, yet.
But this could be solved by a simple script extracting the comparisonSignals from the reference CSV.

For tool vendors, it is not necessary to submit comparisonSignals.txt together with each set of CSV files. That's just redundant.

[modelica-trac-importer]

Comment by jmattsson on 23 Jan 2014 08:30 UTC
Replying to [comment:6 leo.gall]:

Replying to [comment:4 sjoelund.se]:
I'm aware that it is odd to have ModelName.csv instead of result.csv.
In the beginning I used the name result.csv but switched to ModelName.csv.
Reasons:

1. The compare tool uses the CSV file name for generating report file names. So, result.csv leads to stupid report names.
2. When searching for comparison problems (i.e. opening files manually and comparing them in e.g. in Dymola, Matlab, Python), it's more convenient to work with a useful file name instead of a generic name like result.csv.

Seems reasonable.

If more people prefer the flat folder structure, we could switch to that.

I prefer the hierarchical structure.

[modelica-trac-importer]

Comment by leo.gall on 18 Jun 2014 10:50 UTC
Status update:

• Reference results for the Modelica and ModelicaTest are available on MA server with SSH login.

  Access info:

  https://svn.modelica.org/projects/RegressionTesting/AccessInfo/read-access/README.html

  They have been generated with Dymola 2014 FD01 and MSL tag v3.2.1+build.2.release.

• Info on the structure is available in the slides from design meeting in 82nd Modelica Design Meeting in Lund:

  https://svn.modelica.org/projects/ModelicaDesign/trunk/MeetingMinutesMaterial/min82_2014_Lund/Slides-and-Documents/MSLRegressionTesting.pdf

  I'm going to create a readme from that document.

• Open issues with compare tool are documented in tickets:

  https://trac.modelica.org/Modelica/query?status=accepted&status=assigned&status=new&status=reopened&component=--CSV-compare--&order=priority

[modelica-trac-importer]

Comment by sjoelund.se on 23 Jun 2014 10:16 UTC
rsync does not seem to work for the read access. This makes updating the results slow even with a 100 Mbit connection.

[modelica-trac-importer]

Comment by sjoelund.se on 24 Jun 2014 09:28 UTC
Some tests lack reference results (because they would be empty). Wouldn't it make sense to just keep "time" as the comparison signal (it is listed for examples with more signals)? Just need a file containing (for stopTime=1.0):

time
0.0
1.0

[modelica-trac-importer]

Comment by leo.gall on 25 Jun 2014 09:09 UTC
Replying to [comment:10 sjoelund.se]:

rsync does not seem to work for the read access. This makes updating the results slow even with a 100 Mbit connection.

Does write access help?
https://svn.modelica.org/projects/RegressionTesting/AccessInfo/write-access/README.html