Feature/sina/hdf5 output support #1480

doutriaux1 · 2024-12-20T00:09:23Z

Summary

This PR is a feature
It does the following (modify list as needed):
- Adds the capacity to output sina files as hdf5 and opens the door for more formats in the future

cyrush · 2024-12-20T00:13:08Z

src/axom/sina/core/Document.cpp

-      message << "The '" << RELATIONSHIPS_KEY
-              << "' element of a document must be an array";
-      throw std::invalid_argument(message.str());
+        conduit::Node relationship_nodes = asNode[RELATIONSHIPS_KEY];


this should be a ref to avoid a copy

cyrush · 2024-12-20T00:19:56Z

src/axom/sina/core/Document.cpp

+        conduit::Node modifiedRelationshipsNode;
+
+        removeSlashes(relationshipNode, modifiedRelationshipsNode);
+        relationshipsNode.append() = modifiedRelationshipsNode;


you could pass ref to append result into removeSlashes to avoid one extra copy.

cyrush · 2024-12-20T01:22:34Z

src/spack

was this sub module add intended?

bgunnar5

Left some suggestions for you. Also wondering if the fortran changes are tested?

bgunnar5 · 2024-12-20T17:59:55Z

src/axom/sina/core/Document.cpp

-}  // namespace
+}
+
+void protocol_warn(std::string protocol, std::string const &name) {


Function name should be camel case protocolWarn instead (see here). Also, protocol can probably be a const reference here like name is.

bgunnar5 · 2024-12-20T19:12:55Z

src/axom/sina/core/Document.cpp

+  size_t pos = name.rfind('.');
+
+  if (pos != std::string::npos) {
+        std::string found = name.substr(pos+1);
+
+        if (("." + found) != protocol && protocol == ".json") {
+          std::cout << ".json extension not found, did you mean to save to this format?";
+        } else if (("." + found) != protocol && protocol == ".hdf5") {
+          std::cout << ".hdf5 extension not found, did you use one of its other supported types? (h5, hdf, ...)";
+        } else {
+          return;
+        }
+  } else {
+    std::cout << "No file extension found, did you mean to use one of " << protocol << "'s supported types?";
+  }
+}


Can reduce redundancy in if/else chain if we use a map in this function instead:

// Define a mapping of protocols to their warning messages std::unordered_map<std::string, std::string> protocolMessages = { {".json", ".json extension not found, did you mean to save to this format?"}, {".hdf5", ".hdf5 extension not found, did you use one of its other supported types? (h5, hdf, ...)"} }; // Check that the file name has a file extension size_t pos = name.rfind('.'); if (pos != std::string::npos) { std::string found = name.substr(pos); // Check if the found extension matches the expected protocol if (found != protocol) { auto messageIt = protocolMessages.find(protocol); if (messageIt != protocolMessages.end()) { std::cout << messageIt->second << std::endl; } } } else { std::cout << "No file extension found, did you mean to use one of " << protocol << "'s supported types?" << std::endl; }

This might make it easier to add support for new file extensions in the future as well.

bgunnar5 · 2024-12-20T19:30:03Z

src/axom/sina/core/Document.cpp

+            // Find and replace all occurrences of "__SLASH__"
+
+            while ((pos = restoredKey.find(slashSubstitute, pos)) != std::string::npos) {
+                restoredKey.replace(pos, toReplace.length(), replacement);


toReplace does not exist within the scope of this function. Should this be slashSubstitute instead?

bgunnar5 · 2024-12-20T20:09:44Z

src/axom/sina/core/Document.cpp

+        if (relationship_nodes.number_of_children() == 0)
+        {
+            relationship_nodes.set(conduit::DataType::list());
+        }


Should we be adding in this same number of children check for Records as well? If so, maybe a refactor of this entire method would help reduce redundant code:

void Document::createFromNode(conduit::Node const &asNode, RecordLoader const &recordLoader) { auto processChildNodes = [&](const char* key, std::function<void(conduit::Node&)> addFunc) { if (asNode.has_child(key)) { conduit::Node& childNodes = asNode[key]; if (childNodes.number_of_children() == 0) { childNodes.set(conduit::DataType::list()); } if (!childNodes.dtype().is_list()) { std::ostringstream message; message << "The '" << key << "' element of a document must be an array"; throw std::invalid_argument(message.str()); } auto childIter = childNodes.children(); while (childIter.has_next()) { auto child = childIter.next(); addFunc(child); } } }; processChildNodes(RECORDS_KEY, [&](conduit::Node& record) { add(recordLoader.load(record)); }); processChildNodes(RELATIONSHIPS_KEY, [&](conduit::Node& relationship) { add(Relationship{relationship}); }); }

(This is untested so I'm not sure if it'd actually compile).

bgunnar5 · 2024-12-20T21:09:13Z

src/axom/sina/core/Document.cpp

+  if (protocol == Protocol::JSON)
  {
-    std::string message {"Could not save to '"};
-    message += fileName;
-    message += "'";
-    throw std::ios::failure {message};
+      protocol_warn(".json", fileName);
+      auto asJson = document.toJson();
+      std::ofstream fout {tmpFileName};
+      fout.exceptions(std::ostream::failbit | std::ostream::badbit);
+      fout << asJson;
+      fout.close();
+  }
+  else if (protocol == Protocol::HDF5)
+  {
+      protocol_warn(".hdf5", fileName);
+      document.toHDF5(tmpFileName);
+  }
+  else
+  {
+      throw std::invalid_argument("Invalid format choice. Please enter 'json' or 'hdf5'.");
+  }


Might consider porting this to switch rather than if/else

bgunnar5 · 2024-12-21T00:34:44Z

src/axom/sina/core/Document.cpp

+    if (protocol == Protocol::JSON) {
+        std::ifstream file_in {path};
+        std::ostringstream file_contents;
+        file_contents << file_in.rdbuf();
+        file_in.close();
+        node.parse(file_contents.str(), "json");
+        return Document {node, recordLoader};
+    } else if (protocol == Protocol::HDF5) {
+        conduit::Node modifiedNode;
+        conduit::relay::io::load(path, "hdf5", node);
+        restoreSlashes(node, modifiedNode);
+        return Document {modifiedNode, recordLoader};
+    }


Similar to saveDocument we might consider using a switch statement here as well.

bgunnar5 · 2024-12-21T00:38:18Z

src/axom/sina/core/Document.hpp

+/**
+ * \brief Get the current file format version.
+ *
+ * \return A string representing the file format version.
+ */
+inline std::string getSinaFileFormatVersion()
+{
+  return std::to_string(SINA_FILE_FORMAT_VERSION_MAJOR) + "." +
+    std::to_string(SINA_FILE_FORMAT_VERSION_MINOR);
+}


Remove as this is a duplicate of the same function below (lines 245-254)

bgunnar5 · 2024-12-21T00:39:58Z

src/axom/sina/docs/sphinx/tutorial.rst

@@ -53,7 +53,7 @@ Adding Data

 Once we have a Record, we can add different types of data to it. Any Datum
 object that is added will end up in the "data" section of the record in
-the JSON file.
+the file.


"the output file" instead for clarity

bgunnar5 · 2024-12-21T00:41:47Z

src/axom/sina/examples/sina_tutorial.cpp

 }
 //! [end io write]

 //! [begin io read]
 void load()
 {
-  axom::sina::Document doc = axom::sina::loadDocument("my_output.json");
+  axom::sina::Document doc1 = axom::sina::loadDocument("my_output.json");
+  axom::sina::Document doc2 = axom::sina::loadDocument("my_output.json", axom::sina::Protocol::HDF5);


I think this should this be "my_output.hdf5" as the first argument to loadDocument

doutriaux1 and others added 6 commits December 18, 2024 16:19

created a clean new branch from Gabriel's work to push to github repo

3f09c5e

Comments Implemented

e0cd5bf

More Comments

d2f38ac

Fix?

6c32398

Fix??

31242a6

Final Fix

aa9073d

doutriaux1 requested review from white238, cyrush, kennyweiss, rhornung67, bgunnar5 and HaluskaR December 20, 2024 00:09

cyrush reviewed Dec 20, 2024

View reviewed changes

src/spack

Copy link

Member

cyrush Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was this sub module add intended?

bgunnar5 reviewed Dec 21, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/sina/hdf5 output support #1480

Feature/sina/hdf5 output support #1480

doutriaux1 commented Dec 20, 2024

cyrush Dec 20, 2024

cyrush Dec 20, 2024

cyrush Dec 20, 2024

bgunnar5 left a comment

bgunnar5 Dec 20, 2024

bgunnar5 Dec 20, 2024

bgunnar5 Dec 20, 2024

bgunnar5 Dec 20, 2024

bgunnar5 Dec 20, 2024

bgunnar5 Dec 21, 2024

bgunnar5 Dec 21, 2024

bgunnar5 Dec 21, 2024

bgunnar5 Dec 21, 2024

Feature/sina/hdf5 output support #1480

Are you sure you want to change the base?

Feature/sina/hdf5 output support #1480

Conversation

doutriaux1 commented Dec 20, 2024

Summary

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bgunnar5 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment