Add OpenMP for FFTW

[ax3l]

Same as Hi-PACE/hipace#541

For best results, use close-by pinning, esp. with MPI, and avoid oversubscription of cores, esp. if no hyperthreading is available:

export OMP_PROC_BIND=spread
export OMP_PLACES=threads
export OMP_NUM_THREADS=1 # 1,2,4,...

Close #2036

Add Comment to Avoid Accidental Box Over-Subscription

Traditionally, we use OpenMP for the AMReX MFIter-loop over multiple boxes on a rank (and tile/vectorize inside the box).
If we want to parallelize the FFT here with OpenMP, we should make sure that specific loop does not use the pragma omp then (thx @RemiLehe for the reminder):

• add comments here:

• WarpX/Source/FieldSolver/SpectralSolver/SpectralFieldData.cpp

  Line 148 in add16de

  for ( MFIter mfi(mf); mfi.isValid(); ++mfi ){

• WarpX/Source/FieldSolver/SpectralSolver/SpectralFieldData.cpp

  Line 250 in add16de

  for ( MFIter mfi(mf); mfi.isValid(); ++mfi ){

Test with

• Langmuir and scale it up a bit in cells if needed (update: not needed, 64^3 was sufficiently large):

  WarpX/Regression/WarpX-tests.ini

  Lines 369 to 386 in add16de

  	[Langmuir_multi_psatd]
  	buildDir = .
  	inputFile = Examples/Tests/Langmuir/inputs_3d_multi_rt
  	runtime_params = algo.maxwell_solver=psatd psatd.fftw_plan_measure=0 warpx.cfl = 0.5773502691896258
  	dim = 3
  	addToCompileString = USE_PSATD=TRUE
  	restartTest = 0
  	useMPI = 1
  	numprocs = 2
  	useOMP = 1
  	numthreads = 1
  	compileTest = 0
  	doVis = 0
  	compareParticles = 1
  	particleTypes = electrons positrons
  	analysisRoutine = Examples/Tests/Langmuir/analysis_langmuir_multi.py
  	analysisOutputImage = langmuir_multi_analysis.png
  	tolerance = 5.e-11

Name                                                       NCalls  Incl. Min  Incl. Avg  Incl. Max   Max %
----------------------------------------------------------------------------------------------------------
export OMP_NUM_THREADS=1
SpectralSolver::ForwardTransform                              360      2.138      2.138      2.138  14.85%
SpectralSolver::BackwardTransform                             240      1.704      1.704      1.704  11.83%

export OMP_NUM_THREADS=2
SpectralSolver::ForwardTransform                              360      1.079      1.079      1.079  13.81%
SpectralSolver::BackwardTransform                             240     0.8471     0.8471     0.8471  10.84%

export OMP_NUM_THREADS=3
SpectralSolver::ForwardTransform                              360     0.8523     0.8523     0.8523  13.71%
SpectralSolver::BackwardTransform                             240     0.6879     0.6879     0.6879  11.07%

export OMP_NUM_THREADS=4
SpectralSolver::ForwardTransform                              360     0.7416     0.7416     0.7416  13.98%
SpectralSolver::BackwardTransform                             240     0.5924     0.5924     0.5924  11.17%

# Hyperthreading below
export OMP_NUM_THREADS=8
SpectralSolver::ForwardTransform                              360     0.7251     0.7251     0.7251  13.46%
SpectralSolver::BackwardTransform                             240     0.6363     0.6363     0.6363  11.81%

[RemiLehe]

Awesome! Thanks a lot for this change!