Regression tests failed with invalid syntax error

[ramisetti]

All my regression tests have failed with invalid syntax in errorPlotting.py file.
I compiled openfast on mac os (version 10.15.2) and gfortran version 4.9.2 and ran the ctest following the instructions in Section 3.2.2.4.1. I appreciate your help in resolving this issue.

FYI: I have python 3.5.4 with numpy and matplot lib installed.

The below is the output produced by running the ctest -V -R AWT_YFix_WSt

1: Test timeout computed to be: 5400
1: Traceback (most recent call last):
1: File "/XXXX/openfast/reg_tests/executeOpenfastRegressionCase.py", line 35, in
1: from errorPlotting import exportCaseSummary
1: File "/Users/sramiset/Desktop/openfast/reg_tests/lib/errorPlotting.py", line 194
1: _path = os.path.join(plot_path, f'{plot}_script.txt')
^
1: SyntaxError: invalid syntax
1/1 Test #1: AWT_YFix_WSt .....................***Failed 0.35 sec

0% tests passed, 1 tests failed out of 1

Label Time Summary:
aerodyn14 = 0.35 secproc (1 test)
elastodyn = 0.35 secproc (1 test)
openfast = 0.35 secproc (1 test)
servodyn = 0.35 secproc (1 test)

Total Test time (real) = 0.36 sec
The following tests FAILED:
1 - AWT_YFix_WSt (Failed)
Errors while running CTest

[rafmudaf]

Could you try with a newer version of Python? It may be that the error plotting module was written for the latest version of Python.

[ramisetti]

Thanks for your reply. Indeed I did tried using a different version of python after posting my query here but forgot to update it here. I can confirm the problem is resolved when I use Python 3.7.5. However, I am not sure if this still works with other higher versions of Python. I think it could be a good idea to make it more clear about which versions of Python work, in Section 2.2.1.3. Dependencies for the test suite, to help OpenFast users in future.

[rafmudaf]

It's true, it would be good to have a matrix of tests for different operating systems and Python versions. While developing these tools, we used Python 3.7 so its of course difficult to know what doesn't work on lower versions of Python. Generally, language features will work in larger minor version numbers but not necessarily for smaller minor versions, so we can reasonably assume everything works fine in Python 3.8.

In any case, thanks for reporting back and I'll update the documentation accordingly.

[rafmudaf]

Updated in dev.

[Christophe-delFosse]

Dear all,

While trying to do the regression tests for OpenFAST on a HPC cluster, many tests failed. To check what the cause of the fails was, I checked the verbose output of one test run. I get the following error while running ctest -V -R AWT_YFix_WSt:

test 1
Start 1: AWT_YFix_WSt

1: Test command: /usr/bin/python "/XXXX/OpenFAST/reg_tests/executeOpenfastRegressionCase.py" "AWT_YFix_WSt" "/XXXX/OpenFAST/build/glue-codes/openfast/openfast" "/XXXX/OpenFAST/reg_tests/.." "/XXXX/OpenFAST/build/reg_tests/glue-codes/openfast" "0.00001" "Linux" "Intel"
1: Test timeout computed to be: 5400
1: File "/XXXX/OpenFAST/reg_tests/executeOpenfastRegressionCase.py", line 167
1: results = list(zip(testInfo["attribute_names"], [*performance]))
1: ^
1: SyntaxError: invalid syntax

It seems that the synthax '*performance' is invalid. How can I solve this issue?

Thank you in advance!

Sincerely,
Christophe

[rafmudaf]

Hi @Christophe-delFosse can you verify that you're using Python 3?

[Christophe-delFosse]

I am using Python 3.7.4 compiled with GCCcore (module on HPC cluster is called Python/3.7.4-GCCcore-8.3.0). This module was loaded before I ran the regression tests.

[rafmudaf]

Try running that test command (the stuff after "Test command:") on its own outside of CTest. You can copy the command directly from the CTest output.

[Christophe-delFosse]

Dear @rafmudaf

I tried running the test command outside of CTest, but I get the same syntax error from line 167 in the python script "executeOpenfastRegressionCase.py". The arrow points at the '*' symbol in '*performance'. Is this meant to be a multiplication?

/XXXX/OpenFAST/build $ /usr/bin/python "/XXXX/OpenFAST/reg_tests/executeOpenfastRegressionCase.py" "AWT_YFix_WSt" "/XXXX/OpenFAST/build/glue-codes/openfast/openfast" "/XXXX/OpenFAST/reg_tests/.." "/XXXX/OpenFAST/build/reg_tests/glue-codes/openfast" "0.00001" "Linux" "Intel"
File "/XXXX/OpenFAST/reg_tests/executeOpenfastRegressionCase.py", line 167
results = list(zip(testInfo["attribute_names"], [*performance]))
^
SyntaxError: invalid syntax

Sincerely,
Christophe

[rafmudaf]

If its not a Python version problem, then I'm not sure what else might be happening. Just verify again that you're using the Python you intend to use. Typically, the Python at /usr/bin/python is the system Python which is usually Python 2. There may be another one at /usr/bin/python3.

Here's the difference regarding the line in question with Python 2 and 3:

(base) >>mbp@~$ python
Python 3.8.5 (default, Sep  4 2020, 02:22:02) 
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> l = [1,2,3]
>>> m = [*l,4]
>>> m
[1, 2, 3, 4]
>>> 
(base) >>mbp@~$ /usr/bin/python

WARNING: Python 2.7 is not recommended. 
This version is included in macOS for compatibility with legacy software. 
Future versions of macOS will not include Python 2.7. 
Instead, it is recommended that you transition to using 'python3' from within Terminal.

Python 2.7.16 (default, Jun  5 2020, 22:59:21) 
[GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.20) (-macos10.15-objc- on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> l = [1,2,3]
>>> m = [*l,4]
  File "<stdin>", line 1
    m = [*l,4]
         ^
SyntaxError: invalid syntax
>>> 

[Christophe-delFosse]

Dear @rafmudaf

I loaded the python 3.6.5 module on the supercomputer. When I run the python command I get the following:
$ python
Python 3.6.5 (default, Mar 19 2020, 12:36:46)
[GCC Intel(R) C++ gcc 6.4 mode] on linux
Type "help", "copyright", "credits" or "license" for more information.

l=[1,2,3]
m=[*l,4]
m
[1, 2, 3, 4]

Running /usr/bin/python is indeed resulting in the use of Python version 2:
$ /usr/bin/python
Python 2.7.5 (default, Nov 16 2020, 22:23:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

The following thing I tried was to run the test command outside ctest by using python instead of /usr/bin/python, but I have the following error:
/XXXX/OpenFAST/build $ python "/XXXX/OpenFAST/reg_tests/executeOpenfastRegressionCase.py" "AWT_YFix_WSt" "/XXXX/OpenFAST/build/glue-codes/openfast/openfast" "/XXXX/OpenFAST/reg_tests/.." "/XXXX/OpenFAST/build/reg_tests/glue-codes/openfast" "0.00001" "Linux" "Intel"
Error: executable at /XXXX/OpenFAST/build/glue-codes/openfast/openfast does not have proper permissions.

When I try to just run OpenFAST without ctest, it works fine:
/XXXX/OpenFAST/build/glue-codes/openfast $ openfast -v

---

OpenFAST

Copyright (C) 2021 National Renewable Energy Laboratory
Copyright (C) 2021 Envision Energy USA LTD

This program is licensed under Apache License Version 2.0 and comes with ABSOLUTELY NO WARRANTY.
See the "LICENSE" file distributed with this software for details.

---

OpenFAST-v2.5.0
Compile Info:

• Compiler: Intel(R) Fortran Intel(R) 64 Compiler for applications running on Intel(R) 64,
  Version 19.0.5.281 Build 20190815
• Architecture: 64 bit
• Precision: double
• Date: Apr 06 2021
• Time: 14:24:46
  Execution Info:
• Date: 04/08/2021
• Time: 14:38:08+0200

OpenFAST terminated normally.

I would really like to check if all functionalities in OpenFAST are running properly. So I would really appreciate being able to run the ctest command properly and seing the tests pass. Could you help me further please?

Thank you in advance!

Sincerely,
Christophe

[rafmudaf]

Just a tip - its generally safer and more clear to use the entire path to your Python interpreter (or any program). So in your case, you can do which python do find the path and then use that in your command.

What are the permissions on the OpenFAST executable (ls -l build/glue-codes/openfast)? You could try opening them up a bit - chmod 755 build/glue-codes/openfast. Are you running CTest directly or doing this through a job scheduler?

[Christophe-delFosse]

How can I change CTest to choose the correct path to my Python interpreter instead of /usr/bin/python?

The permissions on the OpenFAST executable are as follows:
$ ls -l glue-codes/openfast
total 34092
drwxr-x--- 3 vsc34206 vsc34206 4096 Apr 6 14:24 CMakeFiles
-rw-r--r-- 1 vsc34206 vsc34206 172773 Apr 7 11:37 Cp_Ct_Cq.DTU10MW.txt
-rw-r----- 1 vsc34206 vsc34206 8834 Apr 6 14:24 Makefile
drwxr-x--- 3 vsc34206 vsc34206 4096 Apr 6 20:43 Testing
-rw-r----- 1 vsc34206 vsc34206 2045 Apr 6 14:24 cmake_install.cmake
-rwxr-x--- 1 vsc34206 vsc34206 34559688 Apr 6 14:49 openfast

I am running CTest directly on an interactive node. Later when everything runs fine, I will start using a job scheduler.

[rafmudaf]

You can set this CMake flag: PYTHON_EXECUTABLE:

cmake .. -DPYTHON_EXECUTABLE=<the full path to python 3>

I see that this is not documented in readthedocs, so that's my bad.

[Christophe-delFosse]

Thank you for all the tips. I am able to have more passed tests now. However, there are still 21 failed tests:

/XXXX/OpenFAST/build $ ctest
Test project /XXXX/OpenFAST/build
Start 1: AWT_YFix_WSt
1/51 Test #1: AWT_YFix_WSt ................................. Passed 5.94 sec
Start 2: AWT_WSt_StartUp_HighSpShutDown
2/51 Test #2: AWT_WSt_StartUp_HighSpShutDown ............... Passed 10.35 sec
Start 3: AWT_YFree_WSt
3/51 Test #3: AWT_YFree_WSt ................................ Passed 12.92 sec
Start 4: AWT_YFree_WTurb
4/51 Test #4: AWT_YFree_WTurb .............................. Passed 33.04 sec
Start 5: AWT_WSt_StartUpShutDown
5/51 Test #5: AWT_WSt_StartUpShutDown ...................... Passed 14.85 sec
Start 6: AOC_WSt
6/51 Test #6: AOC_WSt ...................................... Passed 6.64 sec
Start 7: AOC_YFree_WTurb
7/51 Test #7: AOC_YFree_WTurb .............................. Passed 47.02 sec
Start 8: AOC_YFix_WSt
8/51 Test #8: AOC_YFix_WSt ................................. Passed 14.12 sec
Start 9: UAE_Dnwind_YRamp_WSt
9/51 Test #9: UAE_Dnwind_YRamp_WSt .........................***Failed 25.75 sec
Start 10: UAE_Upwind_Rigid_WRamp_PwrCurve
10/51 Test #10: UAE_Upwind_Rigid_WRamp_PwrCurve .............. Passed 20.63 sec
Start 11: WP_VSP_WTurb_PitchFail
11/51 Test #11: WP_VSP_WTurb_PitchFail ....................... Passed 8.30 sec
Start 12: WP_VSP_ECD
12/51 Test #12: WP_VSP_ECD ................................... Passed 10.13 sec
Start 13: WP_VSP_WTurb
13/51 Test #13: WP_VSP_WTurb ................................. Passed 28.28 sec
Start 14: SWRT_YFree_VS_EDG01
14/51 Test #14: SWRT_YFree_VS_EDG01 .......................... Passed 73.62 sec
Start 15: SWRT_YFree_VS_EDC01
15/51 Test #15: SWRT_YFree_VS_EDC01 .......................... Passed 35.79 sec
Start 16: SWRT_YFree_VS_WTurb
16/51 Test #16: SWRT_YFree_VS_WTurb ..........................***Failed 124.63 sec
Start 17: 5MW_Land_DLL_WTurb
17/51 Test #17: 5MW_Land_DLL_WTurb ........................... Passed 49.51 sec
Start 18: 5MW_OC3Mnpl_DLL_WTurb_WavesIrr
18/51 Test #18: 5MW_OC3Mnpl_DLL_WTurb_WavesIrr ............... Passed 160.78 sec
Start 19: 5MW_OC3Trpd_DLL_WSt_WavesReg
19/51 Test #19: 5MW_OC3Trpd_DLL_WSt_WavesReg .................***Failed 437.82 sec
Start 20: 5MW_OC4Jckt_DLL_WTurb_WavesIrr_MGrowth
20/51 Test #20: 5MW_OC4Jckt_DLL_WTurb_WavesIrr_MGrowth .......***Failed 857.31 sec
Start 21: 5MW_ITIBarge_DLL_WTurb_WavesIrr
21/51 Test #21: 5MW_ITIBarge_DLL_WTurb_WavesIrr .............. Passed 17.75 sec
Start 22: 5MW_TLP_DLL_WTurb_WavesIrr_WavesMulti
22/51 Test #22: 5MW_TLP_DLL_WTurb_WavesIrr_WavesMulti ........ Passed 37.27 sec
Start 23: 5MW_OC3Spar_DLL_WTurb_WavesIrr
23/51 Test #23: 5MW_OC3Spar_DLL_WTurb_WavesIrr ............... Passed 33.41 sec
Start 24: 5MW_OC4Semi_WSt_WavesWN
24/51 Test #24: 5MW_OC4Semi_WSt_WavesWN ...................... Passed 83.61 sec
Start 25: 5MW_Land_BD_DLL_WTurb
25/51 Test #25: 5MW_Land_BD_DLL_WTurb ........................ Passed 113.07 sec
Start 26: 5MW_OC4Jckt_ExtPtfm
26/51 Test #26: 5MW_OC4Jckt_ExtPtfm .......................... Passed 9.68 sec
Start 27: HelicalWake_OLAF
27/51 Test #27: HelicalWake_OLAF .............................***Failed 0.83 sec
Start 28: EllipticalWing_OLAF
28/51 Test #28: EllipticalWing_OLAF ..........................***Failed 0.84 sec
Start 29: IEA_LB_RWT-AeroAcoustics
29/51 Test #29: IEA_LB_RWT-AeroAcoustics ..................... Passed 20.66 sec
Start 30: WP_Stationary_Linear
30/51 Test #30: WP_Stationary_Linear ......................... Passed 1.01 sec
Start 31: Ideal_Beam_Fixed_Free_Linear
31/51 Test #31: Ideal_Beam_Fixed_Free_Linear .................***Failed 1.11 sec
Start 32: Ideal_Beam_Free_Free_Linear
32/51 Test #32: Ideal_Beam_Free_Free_Linear ..................***Failed 1.11 sec
Start 33: 5MW_Land_BD_Linear
33/51 Test #33: 5MW_Land_BD_Linear ........................... Passed 16.58 sec
Start 34: 5MW_OC4Semi_Linear
34/51 Test #34: 5MW_OC4Semi_Linear ........................... Passed 124.41 sec
Start 35: ad_timeseries_shutdown
35/51 Test #35: ad_timeseries_shutdown .......................***Failed 0.68 sec
Start 36: bd_5MW_dynamic
36/51 Test #36: bd_5MW_dynamic ...............................***Failed 0.50 sec
Start 37: bd_5MW_dynamic_gravity_Az00
37/51 Test #37: bd_5MW_dynamic_gravity_Az00 ..................***Failed 0.53 sec
Start 38: bd_5MW_dynamic_gravity_Az90
38/51 Test #38: bd_5MW_dynamic_gravity_Az90 ..................***Failed 0.50 sec
Start 39: bd_curved_beam
39/51 Test #39: bd_curved_beam ...............................***Failed 0.52 sec
Start 40: bd_isotropic_rollup
40/51 Test #40: bd_isotropic_rollup ..........................***Failed 0.53 sec
Start 41: bd_static_cantilever_beam
41/51 Test #41: bd_static_cantilever_beam ....................***Failed 0.55 sec
Start 42: bd_static_twisted_with_k1
42/51 Test #42: bd_static_twisted_with_k1 ....................***Failed 0.53 sec
Start 43: hd_OC3tripod_offshore_fixedbottom_wavesirr
43/51 Test #43: hd_OC3tripod_offshore_fixedbottom_wavesirr ...***Failed 0.51 sec
Start 44: hd_5MW_ITIBarge_DLL_WTurb_WavesIrr
44/51 Test #44: hd_5MW_ITIBarge_DLL_WTurb_WavesIrr ...........***Failed 0.53 sec
Start 45: hd_5MW_OC3Spar_DLL_WTurb_WavesIrr
45/51 Test #45: hd_5MW_OC3Spar_DLL_WTurb_WavesIrr ............***Failed 0.52 sec
Start 46: hd_5MW_OC4Semi_WSt_WavesWN
46/51 Test #46: hd_5MW_OC4Semi_WSt_WavesWN ...................***Failed 0.51 sec
Start 47: hd_5MW_TLP_DLL_WTurb_WavesIrr_WavesMulti
47/51 Test #47: hd_5MW_TLP_DLL_WTurb_WavesIrr_WavesMulti .....***Failed 0.52 sec
Start 48: beamdyn_utest
48/51 Test #48: beamdyn_utest ................................ Passed 0.31 sec
Start 49: nwtc_library_utest
49/51 Test #49: nwtc_library_utest ........................... Passed 0.18 sec
Start 50: fvw_utest
50/51 Test #50: fvw_utest .................................... Passed 0.16 sec
Start 51: inflowwind_utest
51/51 Test #51: inflowwind_utest ............................. Passed 0.20 sec

59% tests passed, 21 tests failed out of 51

Label Time Summary:
aeroacoustics = 20.66 secproc (1 test)
aerodyn = 0.68 secproc (1 test)
aerodyn14 = 257.84 secproc (8 tests)
aerodyn15 = 2027.04 secproc (20 tests)
beamdyn = 135.52 secproc (11 tests)
bem = 0.68 secproc (1 test)
dynamic = 1.53 secproc (3 tests)
elastodyn = 2160.17 secproc (26 tests)
extptfm = 9.68 secproc (1 test)
hydrodyn = 1754.94 secproc (13 tests)
linear = 144.21 secproc (5 tests)
map = 88.42 secproc (3 tests)
moordyn = 83.61 secproc (1 test)
offshore = 1630.54 secproc (12 tests)
olaf = 1.68 secproc (2 tests)
openfast = 2438.78 secproc (34 tests)
servodyn = 2403.53 secproc (27 tests)
static = 2.12 secproc (4 tests)
subdyn = 1455.91 sec*proc (3 tests)

Total Test time (real) = 2446.91 sec

The following tests FAILED:
9 - UAE_Dnwind_YRamp_WSt (Failed)
16 - SWRT_YFree_VS_WTurb (Failed)
19 - 5MW_OC3Trpd_DLL_WSt_WavesReg (Failed)
20 - 5MW_OC4Jckt_DLL_WTurb_WavesIrr_MGrowth (Failed)
27 - HelicalWake_OLAF (Failed)
28 - EllipticalWing_OLAF (Failed)
31 - Ideal_Beam_Fixed_Free_Linear (Failed)
32 - Ideal_Beam_Free_Free_Linear (Failed)
35 - ad_timeseries_shutdown (Failed)
36 - bd_5MW_dynamic (Failed)
37 - bd_5MW_dynamic_gravity_Az00 (Failed)
38 - bd_5MW_dynamic_gravity_Az90 (Failed)
39 - bd_curved_beam (Failed)
40 - bd_isotropic_rollup (Failed)
41 - bd_static_cantilever_beam (Failed)
42 - bd_static_twisted_with_k1 (Failed)
43 - hd_OC3tripod_offshore_fixedbottom_wavesirr (Failed)
44 - hd_5MW_ITIBarge_DLL_WTurb_WavesIrr (Failed)
45 - hd_5MW_OC3Spar_DLL_WTurb_WavesIrr (Failed)
46 - hd_5MW_OC4Semi_WSt_WavesWN (Failed)
47 - hd_5MW_TLP_DLL_WTurb_WavesIrr_WavesMulti (Failed)
Errors while running CTest

Which of these failed tests are important and should normally pass? I assume that the data to which the test results is compared has slightly different results, but I want to be sure that my installation of OpenFAST behaves as it should.

[rafmudaf]

Did you compile the driver codes for the regression tests? If not, you'll need to do make beamdyn_driver hydrogen_driver aerodyn_driver. You could also just do make install to compile everything.

[Christophe-delFosse]

Are these driver codes for the regression tests also compiled when running the following:

cmake .. -DBUILD_TESTING=ON -DPYTHON_EXECUTABLE='path-to-python'
make install

Or is there some other code I must download and install?

[rafmudaf]

Yes, they're compiled in the make install command. So if that's the case, then it could be the same permissions issue you saw with OpenFAST. Pick on of the bd_ tests and run it with the verbose flag to get some more information.

[Christophe-delFosse]

You were right! I changed the permissions again with chmod 755 for the three drivers. Now I have only the following failed tests remaining:
The following tests FAILED:
9 - UAE_Dnwind_YRamp_WSt (Failed)
16 - SWRT_YFree_VS_WTurb (Failed)
19 - 5MW_OC3Trpd_DLL_WSt_WavesReg (Failed)
20 - 5MW_OC4Jckt_DLL_WTurb_WavesIrr_MGrowth (Failed)
27 - HelicalWake_OLAF (Failed)
28 - EllipticalWing_OLAF (Failed)
31 - Ideal_Beam_Fixed_Free_Linear (Failed)
32 - Ideal_Beam_Free_Free_Linear (Failed)
46 - hd_5MW_OC4Semi_WSt_WavesWN (Failed)

Are these tests important?

[rafmudaf]

You can ignore 9 and 16.

For the others, you can start to drill down into each one individually. You already know what to do... run each with the verbose flag and see if there's some syntax error or permissions problems or something else that is preventing the simulation from running. If it runs, then check the log file that will be listed in the Test command: section of the verbose output to see if there's any indication of an error.

If the simulation terminates normally, then you can inspect the outputs by generating error plots. Check back in before you get to that point and I'll list those steps.

[Christophe-delFosse]

I checked test 19 - 5MW_OC3Trpd_DLL_WSt_WavesReg (Failed) and the log file shows that the simulation terminates normally. Could you list the steps to generate the error plots please? Thank you in advance.

[Christophe-delFosse]

Tests 19, 20, 31, 32 and 46 give normal simulation terminations. For tests 27 and 28 I get the following message in the log file: No OpenMP support.

[rafmudaf]

For 27 and 28, is that the last line of the log or does it continue to output the time sequence? More importantly, does it end with OpenFAST terminated normally.?

[Christophe-delFosse]

For 27 and 28 it is the last line of the log. I don't have the 'OpenFAST terminated normally'.

[rafmudaf]

@andrew-platt @ebranlard Do you have any thought on this? It looks like its on the main branch (v2.5.0).

[Christophe-delFosse]

If the simulation terminates normally, then you can inspect the outputs by generating error plots. Check back in before you get to that point and I'll list those steps.

Would it be possible to list those extra steps to generate error plots please? This way I can check whether the error is very large or not. Thanks a lot for the help so far!

[rafmudaf]

First, you'll need to install the Bokeh python package at version 1.4 (pip install Bokeh==1.4). Do you have access to pip on your system for the Python you're using?

Then, add the plotting flag to the CMake configuration: cake .. -DCTEST_PLOT_ERRORS=ON. You can just run this command directly, no need to recompile. See the CMake portion of the OpenFAST docs for more information.

Finally, run the failing tests individually and look for the html file in those case directories.

[Christophe-delFosse]

I ran the cmake .. -DCTEST_PLOT_ERRORS=ON command and I have installed the Bokeh==1.4 package through miniconda and activated an environment that includes this package. When I run the individual test case 31: Ideal_Beam_Fixed_Free_Linear, I don't get a html file in the case directory /XXXX/OpenFAST/build/reg_tests/glue-codes/openfast/Ideal_Beam_Fixed_Free_Linear. What am I doing wrong?

[rafmudaf]

The linearization tests are slightly different and will not generate a plot. If you want to understand the differences there, you'll have to dive deeper into the comparison of the .lin files.