wind turbine simulations failing in Nalu-wind

[tchatte3]

Hi Nalu-wind team,

My wind turbine simulations are failing with this git-commit

 Nalu-Wind GIT Commit SHA: a3fd9cce80b4ed5f4887a4b7e0a6473dfb50f36d-DIRTY
Trilinos Version: 12.17-gcf3010a755 

The code fails with a segmentation fault. This only happens with actuator disc simulations, if I comment out the ADM part, the code works fine.

However, when I tried running the same code with an older version of nalu-wind it works fine. I tested my inputs on the following build

Nalu-Wind Version: v1.2.0
   Nalu-Wind GIT Commit SHA: d503c4e7fef16d0d7ec5ebfcb306d00e66f54e77
   Trilinos Version: 12.17-gb42c5a1586

and it works fine.

Attached log and input yaml files for reference.

abl_simulation.log
abl_simulation.yaml.txt

[sayerhs]

@tchatte3 The actuator disk/line model received a significant overhaul as part of our Q2 milestone work (see #550). I am wondering if this is causing some issues with your run. However, the older commit you provide is a bit too old. Did you arrive at that commit through git-bisect? If not, would you mind at least testing commit 87b46a1 and see if that works for your setup? That will allow us to pin point if #550 introduced a regression for your use case. Ideally, if you can provide the results of git-bisect that would be best.

[sayerhs]

Linking #537 to this issue.

[tchatte3]

@sayerhs , we did not find the offending commit by git-bisect. We updated the entire version two weeks back, and found this commit just accidentally. Sure, I will test the commit 87b46a1 and see if that works.

[psakievich]

@tchatte3 the issue seems to be that you've selected to output a field named actuator_sources in the yaml when the actual field is named actuator_source . I checked the git history and this name has not changed in the last 4 years. I'm not sure why an older commit would have worked. With the run that worked off the older commit did the actuator_sources field actually get populated with sensible values? @sayerhs do you recall if we changed parsing/IO field checking behavior recently-ish?

[sayerhs]

@psakievich Looking at the commit history, I don't see anything that would have changed parsing/IO field checking. Also the output section can list fields that don't exist in the code, it will just print out a warning message in the log file, and doesn't segfault.

nalu-wind/src/Realm.C

Lines 2019 to 2030 in 454b575

	for ( std::set<std::string>::iterator itorSet = outputInfo_->outputFieldNameSet_.begin();
	itorSet != outputInfo_->outputFieldNameSet_.end(); ++itorSet ) {
	std::string varName = *itorSet;
	stk::mesh::FieldBase *theField = stk::mesh::get_field_by_name(varName, *metaData_);
	if ( NULL == theField ) {
	NaluEnv::self().naluOutputP0() << " Sorry, no field by the name " << varName << std::endl;
	}
	else {
	// 'varName' is the name that will be written to the database
	// For now, just using the name of the stk field
	ioBroker_->add_field(resultsFileIndex_, *theField, varName);
	}

[sayerhs]

Looking at @tchatte3 's log file that is indeed the case

Realm::create_output_mesh(): Begin
 Sorry, no field by the name actuator_sources
Realm::create_output_mesh() End

So I don't think the error is caused because of that.

[tchatte3]

@sayerhs, @psakievich , this is my old yaml, so yes it still contains the old actuator_sources instead of the actuator_source, but it does pass with warning, and does not cause any issue. And changing it to actuator_source creates the same problem.

[lawrenceccheung]

Hi @tchatte3,
I happen to have a build that's based off of a3fd9cc with some minor changes (but not to the actuator disk part). I tried your actuator disk setup using a different FAST model lying around, and everything works fine.

I also tried your exact input file, again with a different FAST model and quickly changing some BC's and stuff, and it's at least able to reach iteration 1. (It fails later, but maybe because I don't have your exact setup.)

Would you mind sharing your FAST input files and possibly your domain/other setup files? I can try running/debugging more on my end.

Lawrence

[psakievich]

Sorry, I guess I didn't scroll down all the way when looking at the logfile. I thought it died on that line. Glad it doesn't 😁
Ah @lawrenceccheung that might be it! We made changes to the OpenFAST C++ API as part of the actuator refactor to break away from requiring std::vector. @tchatte3 could you try with a fresh build of OpenFAST and see if that works?

[lawrenceccheung]

Thanks @psakievich. For your reference, @tchatte3, I'm using OpenFAST version 1365da7 from the dev branch, downloaded on April 27.

Lawrence

[tchatte3]

@lawrenceccheung , @psakievich thanks for these feedbacks/inputs. I will go ahead and upgrade openFAST and see how it goes.

[tchatte3]

@lawrenceccheung @psakievich, @sayerhs I tested the ADM model with the following code-builds.

   Nalu-Wind GIT Commit SHA: a3fd9cce80b4ed5f4887a4b7e0a6473dfb50f36d-DIRTY
   Trilinos Version: 12.17-gcf3010a755

and the openFAST version used is 1365da71

With these builds the code fails at exactly the same location where it was failing before.

[lawrenceccheung]

Hi @tchatte3, if you can upload the FAST files and maybe other setup files, I'll try running it on the build I have here.

Lawrence

[psakievich]

This looks like it is an issue with openfast. @tchatte3 did your d503c4e and a3fd9cc builds share the same install of openfast? Also does this case run when you run openfast as a stand alone application with your input files? Perhaps uploading the files will be the quickest way to evaluate as @lawrenceccheung suggested.

[sayerhs]

@tchatte3 If it is not possible to share the OpenFAST input files as @lawrenceccheung suggests, can you do a sanity check and run the regression tests with your build? IF they fail, that would help @lawrenceccheung and @psakievich start investigating.

[tchatte3]

@sayerhs, @psakievich @lawrenceccheung
nrel5mw.fst.in.txt

I have uploaded the fst input file. Hope this will work. Aerodyn, Elastodyn input files are difficult to share as they contain proprietary information. Hope, the reference 5MW NREL turbine can serve as a benchmark openFAST module input files.

d503c4e and a3fd9cc commits were tested with the same commit of openFAST which @lawrenceccheung suggested.

@sayerhs I will test the regression tests with the current build as well.

[tchatte3]

@sayerhs , @psakievich , @lawrenceccheung

The regression test

nrel5MWactuatorDisk 

fails at exactly the same way as previous with my builds of openFAST and Nalu-wind.

My stacktrace

OSS: Using decomposition method 'RCB' for 384,000 elements on 20 processors.
Application 130034 is crashing. ATP analysis proceeding...

ATP Stack walkback for Rank 12 starting:
  _start@start.S:120
  __libc_start_main@0x2aaab2837f89
  main@0x551ad4
  sierra::nalu::Simulation::initialize()@0x63ceac
  sierra::nalu::Realms::initialize()@0x674a15
  sierra::nalu::Realm::initialize()@0x668739
  sierra::nalu::Realm::initialize_post_processing_algorithms()@0x655c9d
  sierra::nalu::ActuatorFAST::initialize()@0x989d0f
  fast::OpenFAST::init()@0x2b64c75
  FAST_OpFM_Init@0x2b68067
ATP Stack walkback for Rank 12 done
Process died with signal 11: 'Segmentation fault'
Forcing core dumps of ranks 12, 0

[sayerhs]

@tchatte3 Thanks, I was expecting this. So one more check... are you able to run the fast input deck with the standalone FAST Fortran executable and the C++ driver?

[lawrenceccheung]

Hi @tchatte3,

So I tested the nrel5mw.fst.in you sent with the baseline 5MW files, and that seems to be working.

However, there is (at least) one vital change that happened to the Aerodyn files between the Nalu-Wind commits d503c4e and a3fd9cc. One thing I would check is that the AeroDyn15.dat you have contains the AFTabMod parameter:

======  Airfoil Information =========================================================================
          1   AFTabMod           - Interpolation method for multiple airfoil tables {1=1D interpolation on AoA (first table only); 2=2D interpolation on AoA and Re; 3=2D interpolation on AoA and UserProp} (-)

And while we're at it, also check that the DBEMT_Mod and tau1_const parameters are there too -- see issue 562.

Basically @psakievich's refactor requires the updated OpenFAST, and the latest openFAST added the new parameter(s), so everything kind of needs to be updated.

One more thing that could help debug is to look at where OpenFAST stops. In the stdout file, you might see something at the end like

 OpenFAST-v2.3.0-12-g1365da71-dirty
 Compile Info:
  - Architecture: 64 bit
  - Precision: double
  - Date: Apr 27 2020
  - Time: 15:19:24
 Execution Info:
  - Date: 06/05/2020
  - Time: 17:51:22-0600

 OpenFAST input file heading:
     FAST Certification Test #01: NREL 5.0 MW Baseline Wind Turbine (Onshore)

 Running ElastoDyn.
 Running AeroDyn.
[skybridge-login10:198944] *** Process received signal ***
[skybridge-login10:198944] Signal: Segmentation fault (11)

Telling you that there was a problem with Aerodyn. Otherwise, it moves onto BEM and UnsteadyAero, etc.

Lawrence

[tchatte3]

@lawrenceccheung @sayerhs Thanks for the headsup. Yes, my stdout does show that their is a problem with the Aerodyn. it stops at Aerodyn, just like you saw. Sure, I will take a look at the Aerodyn files. Will update it, accordingly.

[tchatte3]

Thanks, @lawrenceccheung , by adding the line AFTabMod in Aerodyn v15 file I was able to run the regression test as well as my own case with 5MW NREL reference turbine example. This issue is resolved now. I am closing it.

[lawrenceccheung]

Hi @tchatte3,

One more thing I forgot to mention. In your input file, you have density set to two different values.

In the material properties, the reference density is set to 1.0

        - user_constants:
            reference_density: 1.
            reference_temperature: 300.0
            gravity: [0.0, 0.0, -9.81]

but the input to OpenFAST is 1.225.

      Turbine0:
        air_density: 1.225

Everything will run perfectly fine, but the output from OpenFAST will report higher forces/thrust and the turbine outputs will be off by a certain percentage.

This issue has been brought up before -- ideally the code should link density better between openfast and nalu-wind, and avoid inconsistencies like this. But it's always good to check =).

Lawrence

[tchatte3]

@lawrenceccheung, thanks for the input. Indeed, as a workaround it would be better to shift to a python wrapper to write the entire .yaml file, taking minimal inputs from the user (as setup.yaml has been doing), so this redundancy checks can be avoided.