how to improve the efficiency of HPC process?

[Sun-Liu-Yang]

I am trying to use MPI functionality for large-scale parallel computing on a supercomputer cluster. The number of solving units is in the tens of millions, and the number of degrees of freedom of nodes is in the billions.
When I was solving a problem with millions of degrees of freedom, I found that the solving efficiency was already very slow. The official suggested a single core degree of freedom of 20000, but currently my single core solving degree of freedom is only about 10000.
What aspects should I improve the calculation settings to significantly increase computational efficiency and meet the demand for solving billions of degrees of freedom?

[GiudGiud]

Hello

This is a complex problem and the simplest step to take is simply to profile a shorter run to see which step in your simulation is not scaling to tens of millions of cores. The limiting step is likely specific to your problem and to your solver setup.
https://mooseframework.inl.gov/application_development/profiling.html

[Sun-Liu-Yang]

how to reduce the memory occupition？
when I using the MPI funtion, the memory occupition of the CPU is very, about 30G/CPU on a supercomputer cluster. But the memory of the cluster is not enough and the effecifency is slow.
the elements number of my model is about 30,000,000, and the freedom is almost 100,000,000.
the list is my input file of the model.
thank you very much!

[Mesh]
  type = FileMesh
  file ='2024-11-22.inp'
[]

[GlobalParams]
  displacements = 'disp_x disp_y disp_z'
[]

[Variables]
  [disp_x]
  []

  [disp_y]
  []

  [disp_z]
  []
[]

[Physics]
  [SolidMechanics]
    [QuasiStatic]
      [all]
        strain = FINITE
        add_variables = false
        generate_output = 'stress_xx 
                                    stress_yy 
                            stress_zz 
                           min_principal_stress max_principal_stress
                           volumetric_strain vonmises_stress'
      []
    []
  []
[]


[Functions]

  [axial_pressure_1]
    type = PiecewiseLinear
    x = '0  1  2'
    y = '0  1e6  2e6'
  []
 

  [confining_pressure]
    type = PiecewiseLinear
    x = '0  1   1.5  2'
    y = '0  1e6 1e6 1e6'
  []
[]

[BCs]

  [Pressure]

    [load1]
      boundary = 'up_to_force'
      function = 'axial_pressure_1'        # Pa
    []

    [load2]
      boundary = 'side_to_confing'
      function = 'confining_pressure'   # Pa
    []

  []

  [anchor_x]
    type = DirichletBC
    variable = disp_x
    boundary = '0'
    value = 0.0
  []

  [anchor_y]
    type = DirichletBC
    variable = disp_y
    boundary = '0'
    value = 0.0
  []

  [anchor_z]
    type = DirichletBC
    variable = disp_z
    boundary = '0'
    value = 0.0
  []

[]

[Materials]

  [elasticity_tensor_soil]
    type = ComputeIsotropicElasticityTensor
    bulk_modulus = 50e6
    poissons_ratio = 0.25
  []

  [stress]
    type = ComputeFiniteStrainElasticStress
  []

  [density]
    type = GenericConstantMaterial
    prop_names = density
    prop_values = 1800 # kg/m^3
  []

[]




[Executioner]

  type = Transient
  solve_type = JFNK
  l_max_its = 100000
  nl_max_its = 100
  l_tol = 1E-6
  nl_abs_tol = 1E-4
  start_time = 0.0
  end_time = 0.02
  dt = 0.01
  dtmin = 0.001

[]

[Outputs]
  exodus = true
[]

[GiudGiud]

Reducing the number of outputs will reduce the number of fields so i would advise doing that

        generate_output = 'stress_xx 
                                    stress_yy 
                            stress_zz 
                           min_principal_stress max_principal_stress
                           volumetric_strain vonmises_stress'

Then are you getting good convergence with JFNK? That's fairly uncommon.
I would recommend you switch to PJFNK or Newton and use Hypre boomeramg which should scale well for solid mechanics

[Sun-Liu-Yang]

Hello there is a new problem I want to request.
My test model include over 30,000,000 elements and the total Dofs is over 100,000,000.
I used the mesh spliting function pre-spliting the mesh into 3072 partitons ,and resubmited the job using the mesh spilitting configuration .cpr file. But it take several hours to prepare the mesh and cannot continue.

Preparing Mesh state resisting several hours and cannot continue.

mesh splitting common
resubmit commond : mpiexec -n 3072 ./solid_mechanics-opt-intel_2024 -i triaxial/123.i --use-split --split-file 2024-11-22.cpr
mesh spilitting commond: mpiexec -n 3072 ./solid_mechanics-opt-intel_2024 -i triaxial/123.i --split-mesh 3072 --split-file 2024-11-22.cpr

calculating state ：

�[36m
*** Info ***
SolidMechanics Action: selecting 'total small strain' formulation. Use incremental = true to select 'incremental small strain' instead.�[39m
Currently Setting Up
Setting Mesh
Reading Mesh...... [�[33m 43.21 s�[39m] [�[33m 361 MB�[39m]
Finished Setting Mesh [�[33m 43.29 s�[39m] [�[33m 361 MB�[39m]
Setting Up Undisplaced Mesh
Preparing Mesh........................................................

[GiudGiud]

Hello

Using many parallel processes to split it actually a bad idea, because it's replicated during that phase.
Try this:

./solid_mechanics-opt-intel_2024 -i triaxial/123.i --split-mesh 3072 --split-file 2024-11-22.cpr

and if it does not work, use as few mpi processes as possible