Let F3D send it's output image in a stream rather than into a file on disk

[mrdeveloperdude]

See related issue for streaming input.

F3D currently accepts a png filename as a command-line argument for where to store it's output.

F3D will then write the output image into that file after rendering.

Writing the image to disk can incur significant IO penalty. Further, in many cases we won't need the image on disk at all, we simply want the image in memory for processing or redirection over a network or into an external storage system.

This feature request is for having a sanctified and documented way of streaming the image data out of F3D without having to save it to disk first to avoid said IO penalty.

There are ways to possibly do this (untested by issue author) by the use of tmpfs on linux platforms, and by passing /dev/stdout as the image filename.

There is also the possibility of using libf3d instead of the stand alone executable.

Here are some relevant comments from discord:

stream input is on the to-do list but requires changes in underlying libs so the ETA isn't entirely up to F3D (the changes are slowly but surely making their way into VTK as far as I know)
stream output should be doable on F3D's side

tracks that can be investigated without waiting for any new features:
using a ram disk, I'd be interested in seeing a benchmark of piping from stdout to stdin vs. writing and reading in a tmpfs directory (in theory piping should be better because it allows actual streaming, however in the case at hand we can't really render without buffering the whole input model and output image anyway so the streaming aspect won't matter and we just want fast/cheap IO)
using libf3d instead of the F3D application, for example through the python bindings, would let you read rendered images from memory and also allows to do multiple renders of the same model without reloading it

[mwestphal]

This is already supported by the libf3d, but there is no mechanism to expose that in the F3D application yet.

The first question would be to identify how to do that in a cross platform way.

[Meakk]

I confirm --output /dev/stdout works as intended.

[snoyer]

As @mwestphal pointed out, @Meakk's newly merged image::saveBuffer can be used for this feature.

Here's a quick and dirty patch that could be a start for someone feeling like doing a proper PR:

diff --git a/application/F3DStarter.cxx b/application/F3DStarter.cxx
index 43a326c6..876b7f61 100644
--- a/application/F3DStarter.cxx
+++ b/application/F3DStarter.cxx
@@ -14,6 +14,7 @@
 
 #include <cassert>
 #include <filesystem>
+#include <iostream>
 #include <set>
 
 namespace fs = std::filesystem;
@@ -163,8 +164,16 @@ int F3DStarter::Start(int argc, char** argv)
   this->Internals->Parser.GetOptions(
     this->Internals->AppOptions, this->Internals->DynamicOptions, files);
 
-  // Set verbosity level early from command line
-  F3DInternals::SetVerboseLevel(this->Internals->AppOptions.VerboseLevel);
+  const bool renderToStdout = this->Internals->AppOptions.Output == "-";
+  if (renderToStdout)
+  {
+    f3d::log::setVerboseLevel(f3d::log::VerboseLevel::ERROR);
+  }
+  else
+  {
+    // Set verbosity level early from command line
+    F3DInternals::SetVerboseLevel(this->Internals->AppOptions.VerboseLevel);
+  }
 
   // Load plugins from the app options
   this->Internals->Parser.LoadPlugins(this->Internals->AppOptions);
@@ -181,7 +190,10 @@ int F3DStarter::Start(int argc, char** argv)
       this->Internals->AppOptions, this->Internals->DynamicOptions, files);
 
     // Set verbosity level again if it was defined in the configuration file global block
-    F3DInternals::SetVerboseLevel(this->Internals->AppOptions.VerboseLevel);
+    if (!renderToStdout)
+    {
+      F3DInternals::SetVerboseLevel(this->Internals->AppOptions.VerboseLevel);
+    }
   }
 
 #if __APPLE__
@@ -407,7 +419,15 @@ int F3DStarter::Start(int argc, char** argv)
       }
 
       f3d::image img = window.renderToImage(this->Internals->AppOptions.NoBackground);
-      img.save(this->Internals->AppOptions.Output);
+      if (renderToStdout)
+      {
+        const auto buffer = img.saveBuffer();
+        std::copy(buffer.begin(), buffer.end(), std::ostreambuf_iterator(std::cout));
+      }
+      else
+      {
+        img.save(this->Internals->AppOptions.Output);
+      }
     }
     // Start interaction
     else

[Meakk]

Is this - standard? I don't remember I've seen that in the past.
@snoyer PR? 😛

[snoyer]

Is this - standard? I don't remember I've seen that in the past

I don't know about standard but having - as a special case seems pretty accepted. Off the top of my head cat and curl use it for stdin/stdout; git and cd have their own meanings for - too but I wouldn't be able to tell you what they are.

@snoyer PR? 😛

Could slap a "good first issue" tag on it and let someone else have a go

[mwestphal]

I don't know about standard but having - as a special case seems pretty accepted.

Do you have a good source for this ?

[snoyer]

I don't know about standard but having - as a special case seems pretty accepted.

Do you have a good source for this ?

Here's what I could dig up to back up the anecdotal evidence, no real authoritative standard as far as I know:

• cat: "With no FILE, or when FILE is -, read standard input. "
• curl: "If the file name is a plain '-', it is instead written to stdout", "Use the file name "-" (a single dash) to use stdin instead of a given file"
• python's argparse: "FileType objects understand the pseudo-argument '-' and automatically convert this into sys.stdin for readable FileType objects and sys.stdout for writable FileType objects"
• git checkout: "You may also specify - which is synonymous to @{-1}"
• cd (not standard?): OLDPWD A pathname of the previous working directory, used cd -

maybe that's as close as it gets: "For utilities that use operands to represent files to be opened for either reading or writing, the '-' operand should be used to mean only standard input (or standard output when it is clear from context that an output file is being specified) or a file named -."

[mwestphal]

Thanks for digging that up. It is very linuxy though. We need to make sure this will have no impact and work as intended with macOS and Windows. Can - be a file on Windows ?

[snoyer]

I don't know about Windows but - can be a file on linux, if you wanted to write to it instead of stdout you'd need to put a more explicit path to point at it (eg. ./-, ~/-, /foo/bar/-, ...)

If this - idea is too linuxy for F3D it would only be a matter of deciding an alternative syntax (eg --output=stdout or whatever, with the same gotcha that that could be a file too) or a dedicated argument altogether (something like --pipe-output maybe?)

[mwestphal]

I'm not against the idea tbh, even if it is linuxy. I'm just trying to check for unforseen consequences.

[mwestphal]

@mrdeveloperdude the last nightly release contains that feature, use - as an output file.