Memory "management" issue with intel

[guillaumevernieres]

The soca variational application takes an insane amount of memory on Hecules and Gaea (~8TB for the simple 3DVAR), both use intel 2021.9.0. The same application on Hera requires ~0.8TB of memory, the intel compiler version on Hera is 2021.5.0.

I have no idea if the compiler is the issue.

[guillaumevernieres]

I'm labeling this as soca, but I wonder if it's an issues for the fv3-jedi application as well. @RussTreadon-NOAA , @CoryMartin-NOAA or others, have you tried running the variational application on Hercules lately?

[travissluka]

fyi @fmahebert

[guillaumevernieres]

path to logs:

/work2/noaa/da/gvernier/runs/profiling-3dvar

[fmahebert]

Yes, we've seen the intel compiler produce executables that take up way more memory for a little while now. We haven't been able to pinpoint the cause yet. Thanks for opening an issue to track this and to share your measurements.

Related issue (presumably):

• https://github.com/JCSDA-internal/fv3-jedi/issues/1256

[guillaumevernieres]

Running with map profiling:
https://github.com/JCSDA-internal/fv3-jedi/issues/1256#issuecomment-2471504465

[jswhit]

@fmahebert has there been any progress on reducing the memory footprint using intel? This has been blocking us for months now - can't run even a low resolution coupled DA experiment.

[fmahebert]

@jswhit There's been no progress towards understanding this issue from the JCSDA core team, largely for lack of resources (not a lack of concern).

[dkokron]

If you set me up with a small (<= 32 nodes) reproducer on wcoss2, then I can take a look and work with Intel on a solution. I have a complete build of the global-workflow on dogwood.
dogwood:/lfs/h2/hpc/support/daniel.kokron/Projects/GlobalWorkflow/global-workflow

[jswhit]

would using gcc for GDASapp (instead of intel) be a potential workaround for this in the short term?

[shlyaeva]

@jswhit yes, certain compilers/platform combinations certainly work significantly better. E.g. intel + hera doesn't have this issue as Guillaume pointed out. Using gnu on orion/hercules is another option, I think Guillaume had a lot more success with that than with intel.
Bo did a great summary of the results with gnu vs intel on hera and orion in https://github.com/JCSDA-internal/fv3-jedi/issues/1256. It's profiling fv3-jedi + LETKF, but I think it's representative of runs with soca, as well as with Var too.

[jswhit]

@shlyaeva I'm mainly interested in running on gaea, but I think the issue there is the same as on hercules/orion (they all use a newer intel compiler than hera).