openPMD: Handle Zero Particles Well (#2980)

* openPMD: Handle Zero Particles Well When a time step for output encounters zero particles in a species, then we still want to dump them as "empty" species in openPMD. That simplifies post-processing a lot and we have the mechanisms in openPMD for it :) * openPMD: Emtpy Particle Writes Write empty records for iterations (steps or lab steps for BTD, respectively) without particles in a species. * Re-order: ED-PIC & Constant Particle Records - set attributes once - set constant records once - clean up into appropriate functions * Enable BTD for ADIOS :) Works now as well 🎉 * Fix lingo in comments (Reva) Thank you!! :) Co-authored-by: Revathi Jambunathan <41089244+RevathiJambunathan@users.noreply.github.com> Co-authored-by: Revathi Jambunathan <41089244+RevathiJambunathan@users.noreply.github.com>
ECP-WarpX · Mar 22, 2022 · be559b9 · be559b9
1 parent af55efa
commit be559b9
Show file tree

Hide file tree

Showing 4 changed files with 262 additions and 215 deletions.
diff --git a/Docs/source/usage/parameters.rst b/Docs/source/usage/parameters.rst
@@ -1908,7 +1908,6 @@ In-situ capabilities can be used by turning on Sensei or Ascent (provided they a
     ``bp`` is the `ADIOS I/O library <https://csmd.ornl.gov/adios>`_, ``h5`` is the `HDF5 format <https://www.hdfgroup.org/solutions/hdf5/>`_, and ``json`` is a `simple text format <https://en.wikipedia.org/wiki/JSON>`_.
     ``json`` only works with serial/single-rank jobs.
     When WarpX is compiled with openPMD support, the first available backend in the order given above is taken.
-    Note that when using ``BackTransformed`` diagnostic type, the openpmd format supports only ``h5`` backend for both species and fields, while ``bp`` backend can be used for visualizing fields, but not particles. The code will abort if ``bp`` is selected for particle output.
 
 * ``<diag_name>.openpmd_encoding`` (optional, ``v`` (variable based), ``f`` (file based) or ``g`` (group based) ) only read if ``<diag_name>.format = openpmd``.
      openPMD `file output encoding <https://openpmd-api.readthedocs.io/en/0.14.0/usage/concepts.html#iteration-and-series>`__.

diff --git a/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp b/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp
@@ -100,26 +100,6 @@ FlushFormatOpenPMD::FlushFormatOpenPMD (const std::string& diag_name)
     engine_type, engine_parameters,
     warpx.getPMLdirections()
   );
-
-  // Temporarily adding Abort for adios filetype if species is selected for BTD output
-  bool species_output = true;
-  int write_species = 1;
-  std::vector< std::string > output_species_names;
-  bool species_specified = pp_diag_name.queryarr("species", output_species_names);
-  if (species_specified == true and output_species_names.size() > 0) {
-      species_output = true;
-  } else {
-      // By default species output is computed for all diagnostics, if write_species is not set to 0
-      species_output = true;
-  }
-  // Check user-defined option to turn off species output
-  pp_diag_name.query("write_species", write_species);
-  if (write_species == 0) species_output = false;
-  if (diag_type_str == "BackTransformed" and species_output == true) {
-      if (m_OpenPMDPlotWriter->OpenPMDFileType() == "bp") {
-          amrex::Abort(" Currently BackTransformed diagnostics type does not support species output for ADIOS backend. Please select h5 as openpmd backend");
-      }
-  }
 }
 
 void
@@ -155,7 +135,7 @@ FlushFormatOpenPMD::WriteToFile (
         varnames, mf, geom, output_levels, output_iteration, time, isBTD, full_BTD_snapshot);
 
     // particles: all (reside only on locally finest level)
-    m_OpenPMDPlotWriter->WriteOpenPMDParticles(particle_diags, isBTD, totalParticlesFlushedAlready);
+    m_OpenPMDPlotWriter->WriteOpenPMDParticles(particle_diags, isBTD, isLastBTDFlush, totalParticlesFlushedAlready);
 
     // signal that no further updates will be written to this iteration
     m_OpenPMDPlotWriter->CloseStep(isBTD, isLastBTDFlush);

diff --git a/Source/Diagnostics/WarpXOpenPMD.H b/Source/Diagnostics/WarpXOpenPMD.H
@@ -135,6 +135,7 @@ public:
   void WriteOpenPMDParticles (
               const amrex::Vector<ParticleDiag>& particle_diags,
               const bool isBTD = false,
+              const bool isLastBTDFlush = false,
               const amrex::Vector<int>& totalParticlesFlushedAlready = amrex::Vector<int>());
 
   /** Write out all openPMD fields for all active MR levels
@@ -209,18 +210,31 @@ private:
       std::string& comp_name
   ) const;
 
-  /** This function sets up the entries for storing the particle positions, global IDs, and constant records (charge, mass)
+  /** This function sets up the entries for storing the particle positions and global IDs
   *
   * @param[in] currSpecies Corresponding openPMD species
   * @param[in] np          Number of particles
-  * @param[in] charge      Charge of the particles (note: fix for ions)
-  * @param[in] mass        Mass of the particles
+  * @param[in] isBTD       Is this a back-transformed diagnostics output?
   */
   void SetupPos (
+        openPMD::ParticleSpecies& currSpecies,
+        const unsigned long long& np,
+        bool const isBTD = false);
+
+  /** This function sets constant particle records and ED-PIC attributes.
+   *
+   * Sets the entries for storing particle position offset, constant records (charge, mass) and ED-PIC attributes.
+   *
+   * @param[in] currSpecies Corresponding openPMD species
+   * @param[in] np          Number of particles
+   * @param[in] charge      Charge of the particles (note: fix for ions)
+   * @param[in] mass        Mass of the particles
+   */
+  void SetConstParticleRecordsEDPIC (
         openPMD::ParticleSpecies& currSpecies,
         const unsigned long long& np,
         amrex::ParticleReal const charge,
-        amrex::ParticleReal const mass, bool const isBTD = false);
+        amrex::ParticleReal const mass);
 
   /** This function sets up the entries for particle properties
    *
@@ -269,7 +283,9 @@ private:
    * @param[in] int_comp_names The int attribute names, from WarpX
    * @param[in] charge         Charge of the particles (note: fix for ions)
    * @param[in] mass           Mass of the particles
-   * @param[in] isBTD is this a backtransformed diagnostics write?
+   * @param[in] isBTD is this a backtransformed diagnostics (BTD) write?
+   * @param[in] isLastBTDFlush is this the last time we will flush this BTD station?
+   * @param[in] ParticleFlushOffset previously flushed number of particles in BTD
    */
   void DumpToFile (ParticleContainer* pc,
             const std::string& name,
@@ -279,7 +295,9 @@ private:
             const amrex::Vector<std::string>& real_comp_names,
             const amrex::Vector<std::string>&  int_comp_names,
             amrex::ParticleReal const charge,
-            amrex::ParticleReal const mass, const bool isBTD = false,
+            amrex::ParticleReal const mass,
+            const bool isBTD = false,
+            const bool isLastBTDFlush = false,
             int ParticleFlushOffset = 0);
 
   /** Get the openPMD-api filename for openPMD::Series
@@ -294,9 +312,6 @@ private:
 
   std::unique_ptr<openPMD::Series> m_Series;
 
-  /** To set particle meta-data for the openpmd dump. */
-  bool m_doParticleSetUp = true;
-
   /** This is the output directory
    *
    * This usually does not yet end in a `/`.