HPC issue on running MOOSE: obtain different results when using multiple nodes compared with single node

[chunhuizhao478]

Check these boxes if you have followed the posting rules.

• Q&A General is the most appropriate section for my question
• I have consulted the posting Guidelines on the Discussions front page
• I have searched the Discussions forum and my question has not been asked before
• I have searched the MOOSE website and the documentation does not answer my question
• I have formatted my post following the posting guidelines (screenshots as a last resort, triple back quotes around pasted text)

Question

Hi all,

I encounter a serious issue that obtain different results when running on Expanse cluster. The first case I use single node, 40 cores per node, the second case I use 2 nodes, 40 cores per node (80 cores in total), it appears the results for single node works fine (produce zero velocity field), but for the 2 nodes case, there are weird velocity field observed in the domain, I have a custom stress material object, and I suspect it is due to mesh partition and node communications, and I wonder what I should do to debug such issue, thanks!

[GiudGiud]

hello

can you please attach the convergence logs for both solves, as well as the Executioner block in use

thx
Guillaume

[chunhuizhao478]

Thanks @GiudGiud I will update it later and let you know!

[chunhuizhao478]

multi_nodes.txt
single_node.txt

[Executioner]
    type = Transient
    solve_type = 'NEWTON'
    # solve_type = 'PJFNK'
    start_time = -1e-12
    end_time = 1e100
    num_steps = 200
    l_max_its = 200
    l_tol = 1e-7
    nl_rel_tol = 1e-6
    nl_max_its = 10
    nl_abs_tol = 1e-8
    petsc_options_iname = '-ksp_type -pc_type -ksp_initial_guess_nonzero'
    petsc_options_value = 'gmres     hypre True'
    # petsc_options_iname = '-pc_type -pc_factor_shift_type'
    # petsc_options_value = 'lu       NONZERO'
    # petsc_options_iname = '-ksp_type -pc_type -pc_hypre_type  -ksp_initial_guess_nonzero -ksp_pc_side -ksp_max_it -ksp_rtol -ksp_atol'
    # petsc_options_value = 'gmres        hypre      boomeramg                   True        right       1500        1e-7      1e-9    '
    # petsc_options_iname = '-ksp_type -pc_type'
    # petsc_options_value = 'gmres ilu' 
    # petsc_options_iname = '-ksp_type -pc_type'
    # petsc_options_value = 'preonly lu'    
    automatic_scaling = true
    # nl_forced_its = 3
    # line_search = 'bt'
    # dt = 1e-2
    [TimeStepper]
        type = FarmsIterationAdaptiveDT
        dt = 0.01
        cutback_factor_at_failure = 0.5
        optimal_iterations = 5
        growth_factor = 1.5
        max_time_step_bound = 1e10
    []
    [./TimeIntegrator]
        type = NewmarkBeta
        beta = 0.25
        gamma = 0.5
        inactive_tsteps = 1
    [../]
[]

Hi @GiudGiud, I have attached Muti-cores (2:40) and single-core (1:40) logs, and the execution block. Let me know if you have any suggestions, thanks!

[GiudGiud]

do you really need to be running to these insane times? 10^11s is 3000 years
what are you simulating?

This is somewhat loose.
I dont think that's what's biting you on the final time steps but it could be if the results in the first few time steps are wrong.
Are the results different for the first few time steps?

    nl_rel_tol = 1e-6

[chunhuizhao478]

Hi @GiudGiud, I think I figure it out, I have a damage region defined in the central domain, if I refine the damage region and adopt a more smooth transition from maximum damage to zero damage, the imbalance of strain can be reduced and eventually gone.