SubMesh for Nedelec and Raviart-Thomas Bases

[mlstowell]

This PR is studying the feasibility of adding support for Nedelec and Raviart-Thomas basis functions on SubMesh objects.

As a partial test of this new capability we add two new examples ex34p.cpp and ex35p.cpp. Example 34 solves a magnetostatics problem where the current density is computed on a domain derived SubMesh. Example 35 uses a port boundary condition to drive a wave equation in the full domain (this is a modified version of ex22p). The wave equation can use either H1, ND, or RT basis functions. The field used as the port boundary condition is computed as an eigenmode on a SubMesh object which is defined using a subset of the boundary attributes. Four new unit tests are also added.

This PR includes a significant redesign of the DofTransformation class although the resulting design should be backward compatible. The purpose of this redesign was to provide access to the underlying face transformations without tying them to a particular FiniteElementSpace since we need to transform degrees of freedom between a parent mesh and a sub-mesh which have distinct finite element spaces.

This PR also includes two new 2D meshes with non-trivial sets of boundary attributes which were used for testing. Although the test codes themselves do not appear as part of this PR I thought the meshes might be useful and/or interesting even without the tests (e.g. see llnl-p3.mesh).

PR	Author	Editor	Reviewers	Assignment	Approval	Merge
#3379	@mlstowell	@tzanio	@jamiebramwell + @psocratis + @jandrej	6/4/23	7/28/23	7/29/23

PR Checklist

• Code builds.
• Code passes make style.
• Update CHANGELOG:
  • Is this a new feature users need to be aware of? New or updated example or miniapp?
  • Does it make sense to create a new section in the CHANGELOG to group with other related features?
• Update INSTALL:
  • Had a new optional library been added? If so, what range of versions of this library are required? (Make sure the external library is compatible with our BSD license, e.g. it is not licensed under GPL!)
  • Have the version ranges for any required or optional libraries changed?
  • Does make or cmake have a new target?
  • Did the requirements or the installation process change? (rare)
• Update continuous integration server configurations if necessary (e.g. with new version requirements for each of MFEM's dependencies)
  • .github
  • .appveyor.yml
• Update .gitignore:
  • Check if make distclean; git status shows any files that were generated from the source by the project (not an IDE) but we don't want to track in the repository.
  • Add new patterns (just for the new files above) and re-run the above test.
• New examples:
  • All sample runs at the top of the example source file work.
  • Update examples/makefile:
    • Add the example code to the appropriate SEQ_EXAMPLES and PAR_EXAMPLES variables.
    • Add any files generated by it to the clean target.
    • Add the example binary and any files generated by it to the top-level .gitignore file.
  • Update examples/CMakeLists.txt:
    • Add the example code to the ALL_EXE_SRCS variable.
    • Make sure THIS_TEST_OPTIONS is set correctly for the new example.
  • List the new example in doc/CodeDocumentation.dox.
  • If new examples directory (e.g.examples/pumi), list it in doc/CodeDocumentation.conf.in
  • Companion pull request for documentation in mfem/web repo:
    • Update or add example-specific documentation, see e.g. the src/examples.md.
    • Add the description, labels and screenshots in src/examples.md and src/img.
    • In examples.md, list the example under the appropriate categories, add new categories if necessary.
    • Add a short description of the example in the "Extensive Examples" section of features.md.
• New miniapps:
  • All sample runs at the top of the miniapp source file work.
  • Update top-level makefile and makefile in corresponding miniapp directory.
  • Add the miniapp binary and any files generated by it to the top-level .gitignore file.
  • Update CMake build system:
    • Update the CMakeLists.txt file in the miniapps directory, if the new miniapp is in a new directory.
    • Add/update the CMakeLists.txt file in the new miniapp directory.
    • Consider adding a new test for the new miniapp.
  • List the new miniapp in doc/CodeDocumentation.dox
  • If new miniapps directory (e.g.miniapps/nurbs), add it to MINIAPP_SUBDIRS in the makefile.
  • If new miniapps directory (e.g.miniapps/nurbs), list it in doc/CodeDocumentation.conf.in
  • Companion pull request for documentation in mfem/web repo:
    • Update or add miniapp-specific documentation, see e.g. the src/meshing.md and src/electromagnetics.md files.
    • Add the description, labels and screenshots in src/examples.md and src/img.
    • The miniapps go at the end of the page, and are usually listed only under a specific "Application (PDE)" category.
    • Add a short description of the miniapp in the "Extensive Examples" section of features.md.
• New capability:
  • All new public, protected, and private classes, methods, data members, and functions have full Doxygen-style documentation in source comments. Documentation should include descriptions of member data, function arguments and return values, template parameters, and prerequisites for calling new functions.
  • Pointer arguments and return values must specify whether ownership is being transferred or lent with the call.
  • Any new functions should include descriptions of their intended use e.g. for internal use only, user-facing, etc., along with references to example code whenever possible/appropriate.
  • Consider adding new sample runs in existing examples to highlight the new capability.
  • Consider saving cool simulation pictures with the new capability in the Confluence gallery (LLNL only) or submitting them, via pull request, to the gallery section of the mfem/web repo.
  • If this is a major new feature, consider mentioning it in the short summary inside README (rare).
  • List major new classes in doc/CodeDocumentation.dox (rare).
• Update this checklist, if the new pull request affects it.
• Run make unittest to make sure all unit tests pass.
• Run the tests in tests/scripts.
• (LLNL only) After merging:
  • Update internal tests to include the new features.

[mlstowell]

The problem with invalid ParSubMesh boundaries in parallel seems to arise because the faces_info array does not contain information about elements from neighboring processors which abut local edges. I'm not yet sure why this info is not there nor how to obtain it.

[mlstowell]

The problem with high order Nedelec triangles is caused an old friend. Nedelec basis functions associated with the interior of element faces are oriented with the reference triangle or quadrilateral. However, neighboring elements sharing a common face, or in this case neighboring meshes sharing a common face, may choose different orientations for their reference elements.

This problem is resolved using data from the DofTransformation objects returned by GetElementVDofs() and similar methods. We will need to incorporate this DofTransformation data into the ParTransferMap.

One possibility is to build a SparseMatrix containing this data and add a MatVec to the transfer step. This is conceptually simple but in most cases this matrix will be the identity or differ from the identity only slightly. It may be more efficient to create a specialized operator which only touches the DoFs which need to be manipulated.

[tzanio]

Merged in next for testing...

[tzanio]

Re-merged in next for testing...

[tzanio]

Re-merged in next for testing...

[v-dobrev]

I see that all three new examples support the --device/-d option, so I think we should add them to the list of the "device examples", here:

mfem/examples/makefile

Lines 30 to 32 in f97a770

	SEQ_DEVICE_EXAMPLES = ex1 ex3 ex4 ex5 ex6 ex9 ex22 ex24 ex25 ex26
	PAR_DEVICE_EXAMPLES = ex1p ex2p ex3p ex4p ex5p ex6p ex7p ex9p ex13p ex22p \
	ex24p ex25p ex26p

and here:

mfem/examples/CMakeLists.txt

Lines 120 to 124 in f97a770

	set(DEVICE_EXAMPLES
	# serial examples with device support:
	ex1 ex3 ex4 ex5 ex6 ex9 ex22 ex24 ex25 ex26
	# parallel examples with device support:
	ex1p ex2p ex3p ex4p ex5p ex6p ex7p ex9p ex13p ex22p ex24p ex25p ex26p)

[tzanio]

@mlstowell, can you please merge master here and resolve conflicts?

Also, do you want to make the above change suggested by @v-dobrev?

[tzanio]

Re-merged in next for testing...