Issue with Multiple turbines in ADM/ALM model

[tchatte3]

Hi Nalu-Team,

I tried extending the regression test case for NREL5MWActuatorline/NREL5MWActuatorDisk model for one turbine to multiple turbines, but it is giving me segmentation faults.

This is a very minimal modification to the regression_test example and it works perfectly fine for one turbine. The error occurs only when I am trying to it use multiple turbines. This was also tested for multiple processors, but the segmentation fault does not go away

Log files, and stderr files are attached for reference.

abl_simulation.log
abl_simulation.yaml.txt
user_script.stderr.txt
user_script.stdout.txt

Best,
Tanmoy

[psakievich]

@tchatte3 looking at where you're stdout is terminating it looks like this is failing in OpenFAST (potentially in servodyn). Does this case work for you with one turbine? It could be that your library for servodyn is a) not compiled or b) is not being used by the second turbine.

I also see that you're sha from the log file looks like it is from a build in December (d503c4e). Could you first try updating builds for OpenFAST and nalu-wind? When you compile OpenFAST try adding in the compiler flag -D DBG_OUTS. If your building via cmake then add -DCMAKE_CXX_FLAGS=-D DBG_OUTS and -DCMAKE_Fortran_FLAGS=-D DBG_OUTS. There is some new functionality in OpenFAST that was added for new features in nalu-wind via #491.

Also when you compile nalu-wind you can compile with either tests or examples turned on to ensure the servodyn library gets compiled for the nrel5MW case.

[tchatte3]

@psakievich , it is indeed failing in OpenFAST, I had some issues with Servodyn in the regression test, so my feature switch for Servodyn, Hydrodyn etc in .fst file has been kept to "0". This case worked for me, for one turbine, but not for two turbines.

I will try recompiling the Servodyn library, and see if it works.

[lawrenceccheung]

@tchatte3 Hi Tanmoy,

I think I'm able to replicate your problem on a very recent nalu-wind build with a very similar (but not exactly the same) setup. For me, the problem is independent of whether Servodyn is active or not.

Can you send us the exact grid/domain dimensions and the processor counts that you're using? Something like the alm_preprocess.yaml would be perfect.

If memory allows, can you also try running this case on exactly 2 processors? I'm looking into whether there's an issue with the processor/turbine assignments.

Thanks,

Lawrence

[tchatte3]

@lawrenceccheung Thanks. Here is the preprocess file. I am trying to run it on a very coarse grid with 20 processes.

# -*- mode: yaml -*-
#
# Nalu Preprocessing utility
#
# Generate a temperature profile with capping inversion for use with ablNeutralEdge case
#

nalu_abl_mesh:
  output_db: mesh_abl.exo

  spec_type: bounding_box

  fluid_part_name: fluid_part

  vertices:
  - [0.0, 0.0, 0.0]
  - [5000.0, 5000.0, 1000.0]

  mesh_dimensions: [100, 100, 50]
  xmin_boundary_name: west
  xmax_boundary_name: east
  ymin_boundary_name: south
  ymax_boundary_name: north
  zmin_boundary_name: lower
  zmax_boundary_name: upper

[tchatte3]

@lawrenceccheung Unfortunately, there are restriction in the resources; I am not permitted to go below some threshold MPI processes. I will see if I can get down the threshold to 2.

[lawrenceccheung]

@tchatte3 Thanks Tanmoy. It's looking like the issue is stemming from the domain decomposition and processor/turbine assignment. For instance, if you try separating the turbines more, like putting one at [500.0, 2500.0, 0.0] and the other at [4500, 2500.0, 0.0], you might have more success (it worked for me on that domain and on 20 cores).

Once I dig deeper I'll provide some more details and see if I can come up with a fix.

Lawrence

[tchatte3]

@lawrenceccheung Hi Lawrence, thanks for the insights. Yes, I just tested that for [4500, 2500, 0] on 20 procs and works for me too.

Sure, look forward to it.

[psakievich]

@lawrenceccheung @tchatte3 currently openfast only supports one turbine per rank. The current implementation assigns them based off the geometric location of the hubs so if the domain decomposition is such that multiple turbines are assigned to one rank this would cause a failure. I'm addressing this as part of my refactor of the entire actuator framework in #532.

[tchatte3]

@psakievich Thanks!

[gantech]

I believe the OpenFAST - C++ API should be able to support more than 1 turbine per rank. I have been able to do this in the past. I'll look into this error tomorrow.

[lawrenceccheung]

@psakievich @gantech I'm seeing something strange in the coarse search of allocateTurbinesToProcs() in ActuatorFAST.C.

nalu-wind/src/actuator/ActuatorFAST.C

Lines 374 to 387 in 50ef780

	stk::search::coarse_search(
	boundingHubSphereVec_, boundingProcBoxVec_, searchMethod_,
	NaluEnv::self().parallel_comm(), searchKeyPair_, false);
	int iTurb = 0;
	std::vector<
	std::pair<boundingSphere::second_type, boundingElementBox::second_type>>::
	const_iterator ii;
	for (ii = searchKeyPair_.begin(); ii != searchKeyPair_.end(); ++ii) {
	const unsigned box_proc = ii->second.proc();
	FAST.setTurbineProcNo(iTurb, box_proc);
	iTurb++;
	}

Normally I would expect to see that searchKeyPair_.size() = boundingHubSphereVec_.size() after it runs, and of course both should be the same as the number of turbines.

boundingHubSphereVec_.size() behaves fine, however, depending on how I run, I see that searchKeyPair_.size() can be larger or smaller than the number of turbines. That means that in the subsequent for loop, it'll either try to assign non-existent turbines to processor, or some turbines won't get assigned to a processor at all. It looks like it depends on how many cores are used, how the domain is decomposed, and where the turbines are placed.

This might all get solved later in Phil's refactoring, but it's something very odd that I happen to see.

Lawrence

[gantech]

First off.. I was able to reproduce this error on a simple modification of the regression test to a 2 turbine case. Though I don't have a fix for this problem, here's what I have.

OpenFAST C++ API compiles an executable called openfastcpp. You can run that executable like this mpirun -np 1 openfastcpp on this case. The basic C++ API then works with 3 turbines on 1 processor with the latest dev branch of OpenFAST.

openfast_3turbine_1proc_demo.zip

From what I remember the Actuator line implementation did have this capability to run multiple turbines on 1 proc before. I can't be sure of when that capability regressed. Unfortunately, going back in Nalu-wind means going back in Trilinos as well and I don't have the time to deal with installing older Trilinos now.

[gantech]

You could replace the function that allocates turbines to procs with a simple call to FAST.allocateTurbinesToProcsSimple() https://github.com/OpenFAST/openfast/blob/dev/glue-codes/openfast-cpp/src/OpenFAST.cpp#L830 and see if that's where the bug is.

[lawrenceccheung]

@gantech Thanks for the suggestion. I replaced the assignment routines in ActuatorFAST::allocateTurbinesToProcs() with FAST.allocateTurbinesToProcsSimple(), and things seem much more robust now. It runs the multiple turbine case that Tanmoy brought up, and seems to be function well regardless of where the turbines are placed.

However, it does not work when Nturbines > Ncores, so if there are 2 turbines, then it must be run on at least 2 cores. I'm not sure if that's a limitation of the openFAST verison I'm using (branch 6d9e2b9), or if there's something else we need to do.

Lawrence

[gantech]

Hey Lawrence. I'm glad this helped move things along. The example I shared is definitely using the same allocation function https://github.com/OpenFAST/openfast/blob/dev/glue-codes/openfast-cpp/src/FAST_Prog.cpp#L123 I hope you were able to run 3 turbines on 1 proc with that example satisfying the nTurbines > nCores criterion. As far as use of Nalu-wind + actuator and the same criterion, I'm not sure.

[psakievich]

@gantech @lawrenceccheung @tchatte3 I will add multiple turbines per rank into my test procedure after I complete the requirements for the Q2 milestone unless there is some provoking reason to be able to run nTurbines>ranks in the immediate future. This is a new requirement with respect to the milestone so I'm going to put it at the back of my queue. However, my current code will definitely eliminate the issues from using stk::search that @lawrenceccheung has identified.

[psakievich]

@tchatte3 you can try swapping ActDiskFAST for ActDiskFASTNGP. This will let you run to the nturb = nproc limit but not nturb>nproc which is still not supported in nalu-wind. The new implementation is also faster than the legacy ActDiskFAST implementation.

[psakievich]

@tchatte3 do you still have outstanding issues or would it be acceptable to close this issue?

[tchatte3]

@psakievich Yes, I would close this issue. Thanks for the note.