MPI fatal error when using more than 6 processors at Turbomachinery Preprocessing step

[ashvinmahajan]

### MPI partitioning issues for Turbomachinery for processor count more than 6 in mpi

I have noticed an issue with running my turbomachinery model in SU2 version 7.04, 7.1 and version 7.1.1 during the Turbomachinery preprocessing step at the "Initialize Turbo Vertex Structure" where it incorrectly calculates the number of spanwise section in zone 0 and then there is a MPI fatal error. This happens only when I use more than 6 processors. For less than or equal to 6 processors there is no error. I have this issue with 7.04, 7.1 and now 7.1.1.

This issue has a previous bug report in the MPI partitioning issues for Turbomachinery runs #990

I am sure the config file is correct. I have my solutions from 6 processors, but I really need to scale up the processor count so that the restitution time is within a reasonable time.
Below is an example of where it crashes for 7 processors:

WITH 7 processors:

---------------------- Turbomachinery Preprocessing ---------------------

Initialize Turbo Vertex Structure.
Number of span-wise sections in Zone 0: 139. [This is incorrect number of span-wise sections]
Max number of span-wise sections among all zones: 139.
Initialize solver containers for average and performance quantities.
Compute inflow and outflow average geometric quantities.
[1:node0567] unexpected disconnect completion event from [0:node0564]
[3:node0566] unexpected disconnect completion event from [0:node0564]

WITH 6 processors:

Initialize Turbo Vertex Structure.
Number of span-wise sections in Zone 0: 137. [This is the correct number of span-wise sections]
Max number of span-wise sections among all zones: 137.
Initialize solver containers for average and performance quantities.
Compute inflow and outflow average geometric quantities.
Transfer average geometric quantities to zone 0.
Inlet area for Row 1: 30.7045 cm^2.
Oulet area for Row 1: 16.9716 cm^2.
Recomputed number of blades for Row 1: 36.
Initialize inflow and outflow average solution quantities.
Inlet flow angle for Row 1: -0.00422866°.
Outlet flow angle for Row 1: -28.9224°.
------------------------------ Begin Solver -----------------------------
.......

System settings:
CLUSTER

• OS: CentOS release 6.5
• C++ compiler and version: GCC/7.3.0-2.30
• MPI implementation and version: compiler/intel/2018.0.1 intelmpi/2018.0.1/64 mkl/2018.0.1
• SU2 Version: [v7.1 and 7.1.1]
  Thanks
  Ashvin

[pcarruscag]

Checkout the small_fixes branch please, I hope the problem is "that" simple as I will not debug that function any further because it has the world record for nested loops. If it does not work you will have to hope for help from the turbo folks.

      for (iMarker = 0; iMarker < nMarker; iMarker++){
        for (iMarkerTP=1; iMarkerTP < config->GetnMarker_Turbomachinery()+1; iMarkerTP++){
          if (config->GetMarker_All_Turbomachinery(iMarker) == iMarkerTP){
            if (config->GetMarker_All_TurbomachineryFlag(iMarker) == marker_flag){
              for (iVertex = 0; iVertex < nVertex[iMarker]; iVertex++) {
                iPoint = vertex[iMarker][iVertex]->GetNode();
                for (jMarker = 0; jMarker < nMarker; jMarker++){
                  if (config->GetMarker_All_KindBC(jMarker) == PERIODIC_BOUNDARY) {
                    PeriodicBoundary = config->GetMarker_All_PerBound(jMarker);
                    jVertex = nodes->GetVertex(iPoint, jMarker);
                    if ((jVertex != -1) && (PeriodicBoundary == (val_iZone + 1))){
                      coord = nodes->GetCoord(iPoint);
                      switch (config->GetKind_TurboMachinery(val_iZone)){
                      case CENTRIFUGAL

[ashvinmahajan]

I installed the small_fixes build and ran the case, but the problem still persists with respect to the number of cores. I can only run the turbomachinery case with maximum 6 cores. I will wait for the turbomachinery folks to look at it. I hope they find the problem.
Thanks

[pcarruscag]

Do you get the right number of span sections?

[pcarruscag]

I lied and tried to fix it better, I tried it with one of the examples and it does not crash anymore...
Wait for the tests to pass, do git pull to get the changes and try it again please, maybe it will work, maybe it will blow up somewhere else.
You can use more cores without using "more MPI" if you enable OpenMP:
https://su2code.github.io/docs_v7/Build-SU2-Linux-MacOS/
There is a talk explaining how to use it here https://su2foundation.org/su2conference2020/ (hybrid parallel)

[ashvinmahajan]

I hard coded the number of span-wise sections for my case as suggested in #990. By hard coding the number of span-wise sections I am able to get it to work on cores more than 6. For now this works for me.

I added the change in the file CPhysicalGeometry.cpp of version V711

if(marker_flag == OUTFLOW){
// Added by Ashvin MArch 4th 2021
nSpanWiseSections[INFLOW - 1] = 137;
nSpanWiseSections[OUTFLOW - 1] = 137;
if(nSpanWiseSections[INFLOW -1] != nSpanWiseSections[OUTFLOW - 1]){
char buf[100];
SPRINTF(buf, "nSpan inflow %u, nSpan outflow %u", nSpanWiseSections[INFLOW], nSpanWiseSections[OUTFLOW]);
SU2_MPI::Error(string(" At the moment only turbomachinery with the same amount of span-wise section can be simulated\n") + buf, CURRENT_FUNCTION);
}
else{
config->SetnSpanWiseSections(nSpanWiseSections[OUTFLOW -1]);
}
}

}

}