Execution hault in FAST.Farm tests with MinGW on Windows #784
Comments
|
Hi @erikfurevik Nice job debugging the tests and explaining your process. MinGW on Windows is not a well tested or well supported configuration for OpenFAST. It should work fine, but may require some minor tweaks to the CMake configuration, as you've already seen. Also, just a heads up that the Intel compilers are now free for all systems. Any test with an exit code of 0 is more than likely not a problem. The results comparison failed, but the test cases executed to completion. The The non-zero exit code for the FAST Farm tests are concerning. It looks like there's a seg fault given that the log file stops printing. Are you able to compile FAST Farm in debug mode? You can do this with CMake: |
rafmudaf
changed the title
|
Thank you for the suggestions. Do you mean that using Intel compilers would be better? Regarding FAST.Farm I deleted the build folder and recompiled with the debug option as suggested. Here is both console output and LastTest.log. The test-specific log files are still empty. |
Yes, with OpenFAST the Intel compilers will be better. The build system is configured to support Intel specifically and you will get better run time performance. As for your crash in FAST.Farm, there's nothing obvious in the CTest output. Could you run one of these cases with your FAST.Farm debug binary directly? You can use the command given in the CTest output but remove the pipe to the log file: I'm hoping we get an idea of where it stops running. |
|
I understand, I will probably try Intel compilers if this can't be solved, though I would prefer this open-source configuration. Here is what I get when running the command: And the other test for completeness: |
|
@bjonkman do you have any insight on this? |
|
@rafmudaf, the first versions of FAST8 didn't run properly with MinGW, either. I built the executable with debugging symbols ( |
|
@rafmudaf , @bjonkman Following up on this issue, I tried debugging with gdb to find out where the exception was occuring and then played with the type definitions to identify the culprit. It seems like a stack corruption exception thrown by __chkstk_ms() at launch of FWrap_Init. Digging deeper, this appears to stem from the input of type FWrap_MiscVarType, more precisely to its element of type FAST_TurbineType (which brings us away from FAST.farm to the turbine-level OpenFAST), and further down to MAP_Data. Wondering how the MAP dll that is used together with the distributed binaries is handled with MinGW? Is/Should it be compiled again? One thing that makes me doubt about this conclusion though: compiling with single precision shifts the exception further down as "running FASTWrapper" is written to stdout. I did not try debugging there. EDIT: it does not seem to be due to MAP_Data specifically, the latter only significantly increases the stack size and triggers __chkstk_ms(). By using the compiler flag -fno-automatic to avoid using stack (only for relevant files/fortran modules), the code runs a bit further but gets stuck sooner or later. Running out of ideas, I think IVF is the only option until further notice. |
|
This has been an issue for me too. I typically compile OpenFAST tools on windows using GNU Fortran (Rev5, Built by MSYS2 project) 10.2.0 and cmake with "MinGW Makefiles", but FAST.Farm has always failed at initialization (crashing somewhere between WakeDynamics and FastWrapper). I had never found the source of the error, even when compiling in debug. It's one of these errors where depending where you put a print statement, the code will crash at different locations. I have not tried Intel Inspector. @ValentinChb if you find the error that would be of great help to me! |
|
@ValentinChb , can you run your executable in Intel's Inspector program? It is extremely difficult to find memory problems like this without a tool like Inspector. As @ebranlard said, the debugger will often fail in different places based on small differences in code, and can lead you down many different dead-end paths. To use Inspector: Create a new project, and set the application to be the executable you built with the debugging symbols. Set the application parameters to point to your input file (include the path if the input file isn't in the working directory listed just below it in Inspector) I would expect it to fail and point you to the offending line in the source code. If you can post the results here, that would be very helpful. |
|
@ebranlard same setup, same issue. And @bjonkman I concur on the difficulty of finding the error using the debugger alone, the exception being basically triggered after an accumulation of memory handling errors, you'll always only cacth "the last drop". Thanks (also @rafmudaf) for updating me on Intel's and Microsoft's move toward free licensing (although free licensing and open source are two different things and I'm still relunctant to use Visual Studio if I can avoid. Wondering what their business model is now ??). So I gave Intel Inspector a go. Had to check "analyze stack access", else no problem was reported. But I guess the results contain a lot of "false positive" and are hard to interprete. Here is a snapshot:
Any clue? I tried running Intel Inspector on compiled binaries (being aware I'd not get symbols), just to try. Hard to interprete but there is indeed a number of "false positive" (I presume) problems. It would be good to have an IVF debug version to compare with. EDIT: I also tried running Inspector on OpenFAST, which runs fine in appearance. Many problems found are, if not identical, very similar to what I got with FAST.farm. They may either be false positive or not critical then... Maybe things that came after FAST v8, which I understood was made compatible with MinGW Bonnie? |
|
Thanks, @ValentinChb . Inspector can give false positives, particularly on file I/O and on internal libraries, but it does seem like we may have several things to look at in FAST.Farm. Did the code actually complete? Or did it end due to one of these memory issues? I will try to run Inspector on my Intel version. What input files (model) are you using? One of the r-tests? |
|
@bjonkman The FAST.farm analysis did not complete, ended at the same place as before. I'm using the TSinflow r-test. |
|
I fixed a few issues where FAST.Farm memory wasn't deallocated here: https://github.com/bjonkman/openfast/tree/b/FAST.Farm-memory. I am not sure that will fix any of the issues you are seeing, unless there was an issue with the OutList array allocation in gfortran (it may not like the way the INTENT(OUT) arguments combined with ALLOCATABLE properties?). You could try it and see. Here are the results from Inspector from my Intel Fortran build (though it aborted because I didn't have the controller DLLs set properly. I'll update the results when that finishes.) All of these errors are in the following categories:
I don't see anything in this list that concerns me at the moment. But, it is different from the gfortran case, which we'll have to check out more closely. |
|
Possibly related: #895 |
|
I've tried with the latest dev and gfortran 10.3.0, the issue is still here. |
Desciption
Hi, I am trying to complete the r-tests on windows. First I will describe some problems I have encountered on the way and what I did to solve them (in case it is of use to you, or if it's the cause of current issues for me). Then, what my current issue is.
Procedure
git clone --single-branch --branch dev --recursive https://github.com/OpenFAST/openfast.gitcmake .. -G "MinGW Makefiles" -DBUILD_TESTING -DBUILD_FASTFARMmake install. So far no errors.ctest --output-on-failure. Nearly every test fails due to: "ModuleNotFoundError: No module named 'rtestlib'". I see that this library is in C:\openfast\reg_tests\lib, so I move all 5 files there to C:\openfast\reg_tests.ctest --output-on-failure. Now, nearly all fail due to:Error: file does not exist at C:/openfast/build/glue-codes/openfast/openfast. I see from C:\openfast\build\Testing\Temporary\LastTest.log that the commands being run are of the formApparently an .exe extension is missing from the executable file shown in bold, as adding this to the command completes with code 0. I believe the commands are decided by C:\openfast\build\reg_tests\CTestTestfile.cmake, so I enter this file and add .exe to all the executables (openfast, fast.farm, drivers).
The issue
I'm most interested in the two FAST.Farm tests. Both
"C:/openfast/build/reg_tests/glue-codes/fast-farm\TSinflow\TSinflow.log" and
"C:/openfast/build/reg_tests/glue-codes/fast-farm\LESinflow\LESinflow.log" are empty files.
Also, if I try to run to run a FAST.Farm test manually, I get no output. See below.
I'm very grateful if someone has a good idea about what's wrong.
The text was updated successfully, but these errors were encountered: