Linear algebra cleanup

[klevzoff]

Not sure how to best describe this PR. Essentially I'm splitting off a bunch of code cleanup/refactoring from my multiscale branch to get reviewed separately. I'm not sure how it came to include 127 files, but I don't think I touched any files unnecessarily.

Most notably:

1. Refactor direct solver interfaces (SuiteSparse and SuperLU_Dist) to reduce code duplication:
   • Each one is taken out of package-specific solver classes and made into a full-fledged solver class of its own, templated on LAI.
   • Differences between 3 LA packages in how matrix/vector entries are extracted and/or gathered/scattered (in case of SuiteSparse) are abstracted by new set of classes, HypreExport/PetscExport/TrilinosExport.
2. Add a new base class, LinearSolverBase, that extends the concept of PreconditionerBase and is a new common base for new direct solver classes and package-specific solver classes (HypreSolver, etc.). The latter now only implement iterative solves and have been split into setup/solve stages, which allows using them as preconditioners (in case someone wants a really accurate inverse approximation, for comparison purposes).
3. Simplify PreconditionerBase interface by removing the overload setup(Matrix const &, DofManager const &). Instead, added the ability to "attach" a DofManager to a matrix object (essentially, just store a pointer to it with the matrix). This makes a whole lot of things a whole lot simper.
4. Reimplement a few operations in HypreMatrix/HypreVector replacing raw loops with RAJA kernels and correct exec policy. This should make them usable with hypre-GPU. Not all operations are fully GPU-safe; some, like single-row-access stuff, don't make sense on the GPU and should potentially be deprecated/removed.
5. Split testLAOperations into separate CTest targets for vectors, matrices and solvers (now prefixed with testLinearAlgebra, for easier grouping). Implemented a proper unit test for vector classes that tests each function in vector's API (actually this is brought over from my unmerged Tpetra PR) - we should do this for matrices at some point but it's way more tedious.
6. Reorganized some file locations and/or names (e.g. GMRESsolver -> GmresSolver to be consistent with project conventions).
7. Cleaned up Hypre MGR stuff a bit by splitting into separate recipe classes. Not yet the final form I'd like to see, but hopefully a small step towards a more declarative style description of recipes.

The rest is just code cleanup and minor usability improvements here and there.

All tests pass, no rebaseline necessary rebaseline needed because some linear solver parameters changed type from strings to enums and some changed enum values (we should really be adding new options at the end, to avoid value changes, but I took this chance since I'll be rebaselining anyway).

Rebaseline PR: https://github.com/GEOSX/integratedTests/pull/130

[klevzoff]

@castelletto1 @francoishamon I refactored MGR recipes into these classes. The class constructor chooses all relevant MGR parameter values, and then the setup function applies them on the solver. Ideally I wanted the recipes to be purely static sets of parameter choices which then get applied by some generic common code, but some dynamism is necessary (in particular, to set up coarse indices in variable-size problems like multiphase; but also maybe some parameter choices could depend on runtime characteristics of the problem, like size?), so I left it like this. We can feed more data about the problem into the constructor as we need it.

Anyway, I really hope that I didn't mess up when transferring the parameter values. I'm running a CO2 tutorial and the two multiphase benchmarks in the repo that use MGR, but for other solvers I will have to just switch some of the integrated tests to MGR to verify. But even then I don't know what optimal performance looked like. Let me know if you see anything weird here or if you have other cases you'd like me to run.

[klevzoff]

On a side note, it baffles me that hypre does not provide named constants for various parameter choices, at least in the form of macro defs, and keeps using raw integer constants - even in its central, most established components like BoomerAMG. If you're lucky, you'll find a description in function comment, but I found those to be missing or out-of-date sometimes.

If hypre has a fundamental/philosophical reason to not provide constants, do you think it would be worthwhile defining them on the GEOSX side? E.g.

namespace geosx { 
namespace hypre {
enum RelaxType : HYPRE_Int
{
  jacobi = 0, // Jacobi
  seqGaussSeidel = 1, // Gauss-Seidel, sequential (very slow!)
  ...
  l1jacobi = 18 // \f$\ell_1\f$-scaled jacobi
};
} }

[castelletto1]

This looks great! I would suggest to have separate files for each MGR strategy class.

[castelletto1]

I think it'd much clearer if we introduced (not necessarily in this PR) constants in GEOSX as you're suggesting

[andrea-franceschini]

Nice job! Thanks

[castelletto1]

The linear algebra component keeps improving, thank you very much @klevzoff! Comments I left don't need to be all addressed within this PR. Some could definitely be deferred for a Doc Day. IMO time is mature to put together an rst document giving an overview of linear algebra in GEOSX (beyond linear solvers).

[castelletto1]

I prefer the capitalized version (BICGSTABSolver, CGSolver, GMRESSolver) since these are all acronyms. This follows the same naming logic adopted in physics solvers where FEM, FVM -- and not Fem, Fvm -- are appended. If we wanted to be very sophisticated we may use BiCGStabSolver and GMResSolver, which I don't like though.

[klevzoff]

I'd be on board if we found a way to have the acronym at the end of the name as in LaplaceFEM and SinglePhaseFVM. Having another capital letter follow the uppercase acronym feels a bit off to me, nor do I like the old scheme (CGsolver). Maybe SolverCG (not too pretty either, but feels tolerable)?

[castelletto1]

I don't know what actions were finally decided with the hypre team on this issue. @rrsettgast any recent updates?

[castelletto1]

Could we move this

https://github.com/GEOSX/GEOSX/blob/51f80196beb7fc726c911fa47e0c04de8d1b6d7b/src/coreComponents/linearAlgebra/interfaces/InterfaceTypes.hpp#L36-L40

somewhere in codingUtilities?

[castelletto1]

Currently the initialize method for the three LAI interfaces has standard argc and argv parameters:

https://github.com/GEOSX/GEOSX/blob/51f80196beb7fc726c911fa47e0c04de8d1b6d7b/src/coreComponents/linearAlgebra/interfaces/hypre/HypreInterface.hpp#L44

https://github.com/GEOSX/GEOSX/blob/51f80196beb7fc726c911fa47e0c04de8d1b6d7b/src/coreComponents/linearAlgebra/interfaces/petsc/PetscInterface.hpp#L46

https://github.com/GEOSX/GEOSX/blob/51f80196beb7fc726c911fa47e0c04de8d1b6d7b/src/coreComponents/linearAlgebra/interfaces/trilinos/TrilinosInterface.hpp#L44

However, they are never used in the implementation (GEOSX_UNUSED_PARAM). Until we actually need them, I suggest to simplify the interface to:

static void initialize( ); 

[klevzoff]

Could we move this

https://github.com/GEOSX/GEOSX/blob/51f80196beb7fc726c911fa47e0c04de8d1b6d7b/src/coreComponents/linearAlgebra/interfaces/InterfaceTypes.hpp#L36-L40

somewhere in codingUtilities?

Yeah. I think common/GeosxMacros.hpp will do for now.

[klevzoff]

Until we actually need them, I suggest to simplify the interface to:

static void initialize( ); 

Done (after push)