Rs fmt specification - Journaling Algorithm for error, warning, fprin…

…tf (#1702)

* Journaling Algorithm for error, warning, fprintf

Removes dependence on ShadingContext and OIIO for logging error, warning, printf, and fprintf calls
for use on CPU and device, and tracks logging calls per thread index and shade index.
All calls to shader execution track thread index; ShaderGlobals contains shade_index (for processing
fmt specification calls) and thread_index for allocating memory per thread.

* Add new customization points in RendererServices for reporting errors, warnings, printf, and fprintf.
These customization points accept a fmtlib style format specifier (vs. printf style) whose argument types are represented as EncodedTypes (encodedtypes.h) and can be subsequently decoded via a utility function.

Removes dependence on ShadingContext and OSL::ErrorHandler for logging error, warning, printf, and fprintf calls
for use on CPU and device during Shader execution.  ShadingContext and OSL::ErrorHandler remain utilized during shader compilation/JIT.

Journal Buffer approach:  each thread gets its own chain of pages within the shared journal buffer for recording errors, warnings, etc.  By maintaining a separate chain per thread we can record errors (etc.) in a non-blocking fashion. Shade index is recorded for use by error reporter to sort messages but it is not used at present.
Using a Journal buffer is up to a Renderer.  The Renderer use the new customization points (rs_errorfmt, rs_filefmt, rs_printfmt, rs_warningfmt or their virtual function equivalents) to interact with a journal::Writer and journal buffer owned by its RenderState.  The Renderer would of have to allocate and initialized the journal buffer with the requested page size and number of threads before executing a shader. After a shader is done executing, the contents of the journal buffer can be transfered back to the host and processed with a journal::Reader, which reports them through a journal::Reporter's virtual interface. Intent is for Renderers's to use the Journal buffer and provide their own overriden version of the journal::Reporter.  Testshade provides example usage.  For legacy purposes, the default virtual RendererServices will route errors to ShadingSystem's OSL::ErrorHandler, but intent is for Renderer's to switch over to using a Journal buffer or their own custom logging.

NOTE:  OSL library functions as well use Renderer Service free functions can directly call OSL::filefmt, OSL::printfmt, OSL::errorfmt, OSL::warningfmt which wrappers to the underlying free functions (who may record the messages into a journal buffer).

Added a new hashcode datatype in typespec.cpp; represented as "h" in function arguments in builtindecl.
To print closures added a new OSL function osl_closure_to_ustringhash to return a ustringhash_pod value directly.

Shader execute functions now require a zero-based thread_index to identify the calling thread; used for journalling.
RendererServices::get_texture_info parameter ShadingContext * changed to ShaderGlobals * (aka ExecutionContext).

ShaderGlobals now tracks both shade_index and thread_index.
As we look to hide/remove ShaderGlobals in the future, we still need a context object to replace the role the ShaderGlobals struct is currently serving (just minus the actual shader global variables).  As a first step towards this goal, we introduce the concept of an ExecutionContext and OpaqueExecutionContext.  These just alias to ShaderGlobals currently, but start to establish the idea of hiding its contents.  Instead of direct memory access to the ExecutionContext/ShaderGlobals new getter functions have been added that accept an OpaqueExecutionContext and provide the requested data.  When inlined there should be no performance penalty, and reliance on the ShaderGlobals implementation detail will be reduced.  IE:
    inline const Vec3 & get_P(OpaqueExecContextPtr oec);
    template<typename RenderStateT> RenderStateT* get_rs(OpaqueExecContextPtr oec);
    inline int get_raytype(OpaqueExecContextPtr oec);
Updated free function renderer services and opcolor to use the ExecContext, but left the rest alone to minimize size of this PR.

We introduce opfmt.cpp that provides shim function for llvm_gen to call, where the shim adapts datatypes from llvm to c++ forwarding calls to the free function renderer service error, warning, print, fileprint functions.

Split llvm_gen_printf into llvm_gen_printf_legacy, for string format calls or when using optix, and the new llvm_gen_print_fmt which will convert the printf style specifier to the fmtlib format that the Renderer Service customization points are expecting.

fprintf by default no longer writes out to a file, is routed to journal::Reporter::report_file_print which could be overriden (example in TestShade to maintain existing behavior).  However for security reasons, we do not recommend allowing OSL shader's direct file system but instead just an annotated log entry.

To maintain existing behavior of disallowing output of repeated errors/warnings over a history window, we have provided 2 classes that could be used during reporting: TrackRecentlyReported and Report2ErrorHandler.  TrackRecentlyReported can turn on/off limitting errors and warnings and control the history capacity.  Report2ErrorHandler simply reports errors to an existing OSL::ErrorHandler to allow for easy integration.  However we expect Renderers to not use Report2ErrorHandler and provide their own to directly interact with their logging systems.


---------

Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>

src/cmake/testing.cmake

@@ -143,6 +143,17 @@ macro ( TESTSUITE )
             add_one_testsuite ("${_testname}.opt" "${_testsrcdir}"
                                ENV TESTSHADE_OPT=2 )
         endif ()
+        # Run the same test again with aggressive -O2 runtime
+        # optimization, triggered by setting TESTSHADE_OPT env variable.
+        # Skip OptiX-only tests and those with a NOOPTIMIZE marker file.
+        if (NOT _testname MATCHES "optix"
+            AND NOT EXISTS "${_testsrcdir}/NOSCALAR"
+            AND NOT EXISTS "${_testsrcdir}/BATCHED_REGRESSION"
+            AND NOT EXISTS "${_testsrcdir}/NOOPTIMIZE"
+            AND NOT EXISTS "${_testsrcdir}/NORSBITCODE")
+            add_one_testsuite ("${_testname}.opt.rs_bitcode" "${_testsrcdir}"
+                               ENV TESTSHADE_OPT=2 TESTSHADE_RS_BITCODE=1)
+        endif ()
         # When building for OptiX support, also run it in OptiX mode
         # if there is an OPTIX marker file in the directory.
         # If an environment variable $TESTSUITE_OPTIX is nonzero, then
@@ -346,6 +357,9 @@ macro (osl_add_all_tests)
                 struct-nested struct-nested-assign struct-nested-deep
                 ternary
                 testshade-expr
+		test-fmt-arrays test-fmt-fileprint
+		test-fmt-cxpf  test-fmt-noise test-fmt-matrixcolor 
+                test-fmt-stpf test-fmt-errorwarning test-fmt-errorwarning-repeats
                 texture-alpha texture-alpha-derivs
                 texture-blur texture-connected-options
                 texture-derivs texture-environment texture-errormsg

src/include/OSL/encodedtypes.h

@@ -0,0 +1,217 @@
+// Copyright Contributors to the Open Shading Language project.
+// SPDX-License-Identifier: BSD-3-Clause
+// https://github.com/AcademySoftwareFoundation/OpenShadingLanguage
+
+#pragma once
+
+#include <OSL/oslconfig.h>
+
+OSL_NAMESPACE_ENTER
+
+
+// Substitute for variable argument list, which is traditionally used with a printf, with arrays
+// of EncodedTypes to identify types contained in a blind payload of values.
+enum class EncodedType : uint8_t {
+    // OSL Shaders could encode these types
+    kUstringHash = 0,
+    kInt32,
+    kFloat,
+
+    // OSL library functions or renderer services encode additional native types
+    kInt64,
+    kDouble,
+    kUInt32,
+    kUInt64,
+    kPointer,
+    kTypeDesc,
+    kCount
+};
+
+// Decode will use each EncodedType in the array to interpret the contents of the arg_values
+// parameter alongwith a fmtlib specifier identified by the ustring that the format_hash represents.
+// Contents of decoded_str are written over (not appended).
+// Returns # of bytes read from arg_values
+int
+decode_message(uint64_t format_hash, int32_t arg_count,
+               const EncodedType* arg_types, const uint8_t* arg_values,
+               std::string& decoded_str);
+
+namespace pvt {
+
+constexpr inline uint32_t
+size_of_encoded_type(EncodedType et)
+{
+    constexpr uint32_t SizeByEncodedType[] = {
+        sizeof(OSL::ustringhash), sizeof(int32_t), sizeof(float),
+        sizeof(int64_t),          sizeof(double),  sizeof(uint32_t),
+        sizeof(uint64_t),         sizeof(void*),   sizeof(OSL::TypeDesc),
+    };
+    static_assert(sizeof(SizeByEncodedType) / sizeof(SizeByEncodedType[0])
+                      == size_t(EncodedType::kCount),
+                  "Keep array contents lined up with enum");
+    return SizeByEncodedType[static_cast<int>(et)];
+}
+
+template<typename T>
+struct TypeEncoder;  //Undefined on purpose; all types must have a specialization
+
+template<> struct TypeEncoder<OSL::ustringhash> {
+    using DataType                     = ustringhash_pod;
+    static constexpr EncodedType value = EncodedType::kUstringHash;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const OSL::ustringhash& val) { return val.hash(); }
+};
+
+template<> struct TypeEncoder<OSL::ustring> {
+    using DataType                     = ustringhash_pod;
+    static constexpr EncodedType value = EncodedType::kUstringHash;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const OSL::ustring& val) { return val.hash(); }
+};
+
+template<> struct TypeEncoder<const char*> {
+    using DataType                     = ustringhash_pod;
+    static constexpr EncodedType value = EncodedType::kUstringHash;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const char* val) { return ustring(val).hash(); }
+};
+
+template<> struct TypeEncoder<std::string> {
+    using DataType                     = ustringhash_pod;
+    static constexpr EncodedType value = EncodedType::kUstringHash;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const std::string& val)
+    {
+        return ustring(val).hash();
+    }
+};
+
+template<> struct TypeEncoder<int32_t> {
+    using DataType                     = int32_t;
+    static constexpr EncodedType value = EncodedType::kInt32;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const int32_t val) { return val; }
+};
+
+template<> struct TypeEncoder<float> {
+    using DataType                     = float;
+    static constexpr EncodedType value = EncodedType::kFloat;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const float val) { return val; }
+};
+
+
+template<> struct TypeEncoder<int64_t> {
+    using DataType                     = int64_t;
+    static constexpr EncodedType value = EncodedType::kInt64;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const int64_t val) { return val; }
+};
+
+// On macOS 10.+ ptrdiff_t is a long, but on linux this
+// specialization would conflict with int64_t
+#if defined(__APPLE__) && defined(__MACH__)
+template<> struct TypeEncoder<ptrdiff_t> {
+    using DataType                     = int64_t;
+    static constexpr EncodedType value = EncodedType::kInt64;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const ptrdiff_t val) { return val; }
+};
+#endif
+
+template<> struct TypeEncoder<double> {
+    using DataType                     = double;
+    static constexpr EncodedType value = EncodedType::kDouble;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const double val) { return val; }
+};
+
+template<> struct TypeEncoder<uint64_t> {
+    using DataType                     = uint64_t;
+    static constexpr EncodedType value = EncodedType::kUInt64;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const uint64_t val) { return val; }
+};
+
+template<> struct TypeEncoder<uint32_t> {
+    using DataType                     = uint32_t;
+    static constexpr EncodedType value = EncodedType::kUInt32;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const uint32_t val) { return val; }
+};
+
+template<typename T> struct TypeEncoder<T*> {
+    // fmtlib only supports const void *, all other pointers must be
+    // converted with a cast or helper like fmt::ptr(p)
+    static_assert(
+        (std::is_same<void*, T*>::value || std::is_same<const void*, T*>::value),
+        "formatting of non-void pointers is disallowed, wrap with fmt::ptr");
+
+    using DataType                     = const T*;
+    static constexpr EncodedType value = EncodedType::kPointer;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const T* val) { return val; }
+};
+
+template<> struct TypeEncoder<OSL::TypeDesc> {
+    // To avoid warnings about non-pod types when we PackArgs
+    // we will encode into builtin type
+    using DataType = uint64_t;
+    static_assert(sizeof(OSL::TypeDesc) == sizeof(DataType), "unexpected");
+    static constexpr EncodedType value = EncodedType::kTypeDesc;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const OSL::TypeDesc& val)
+    {
+        return OSL::bitcast<DataType>(val);
+    }
+};
+
+// Promote thinner types
+template<> struct TypeEncoder<int16_t> {
+    using DataType                     = int32_t;
+    static constexpr EncodedType value = EncodedType::kInt32;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const int16_t val) { return DataType(val); }
+};
+
+template<> struct TypeEncoder<int8_t> {
+    using DataType                     = int32_t;
+    static constexpr EncodedType value = EncodedType::kInt32;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const int8_t val) { return DataType(val); }
+};
+
+template<> struct TypeEncoder<uint16_t> {
+    using DataType                     = uint32_t;
+    static constexpr EncodedType value = EncodedType::kUInt32;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const uint16_t val) { return DataType(val); }
+};
+
+template<> struct TypeEncoder<uint8_t> {
+    using DataType                     = uint32_t;
+    static constexpr EncodedType value = EncodedType::kUInt32;
+    static_assert(size_of_encoded_type(value) == sizeof(DataType),
+                  "unexpected");
+    static DataType Encode(const uint8_t val) { return DataType(val); }
+};
+}  // namespace pvt
+
+
+OSL_NAMESPACE_EXIT

src/include/OSL/fmt_util.h

@@ -0,0 +1,134 @@
+// Copyright Contributors to the Open Shading Language project.
+// SPDX-License-Identifier: BSD-3-Clause
+// https://github.com/AcademySoftwareFoundation/OpenShadingLanguage
+
+#pragma once
+
+#include <OSL/rs_free_function.h>
+
+// OSL library functions as well as Renderer Service free functions
+// can utilize OSL::filefmt, OSL::printfmt, OSL::errorfmt, and
+// OSL::warningfmt wrappers to accept variable arguments to be
+// encoded into a EncodeType array and PackedArgument buffer
+// that underlying render service free functions require.
+// NOTE:  All strings arguments to the underlying render service free
+// functions must be ustringhash.  To make this easier, these wrappers
+// automatically convert ustring and const char * to ustringhash.
+
+OSL_NAMESPACE_ENTER
+
+namespace pvt {
+// PackedArgs is similar to tuple but packs its data back to back
+// in memory layout, which is what we need to build up payload
+// to the fmt reporting system
+template<int IndexT, typename TypeT> struct PackedArg {
+    explicit PackedArg(const TypeT& a_value) : m_value(a_value) {}
+    TypeT m_value;
+} __attribute__((packed, aligned(1)));
+
+template<typename IntSequenceT, typename... TypeListT> struct PackedArgsBase;
+// Specialize to extract a parameter pack of the IntegerSquence
+// so it can be expanded alongside the TypeListT parameter pack
+template<int... IntegerListT, typename... TypeListT>
+struct PackedArgsBase<std::integer_sequence<int, IntegerListT...>, TypeListT...>
+    : public PackedArg<IntegerListT, TypeListT>... {
+    explicit PackedArgsBase(const TypeListT&... a_values)
+        // multiple inheritance of individual components
+        // uniquely identified by the <Integer,Type> combo
+        : PackedArg<IntegerListT, TypeListT>(a_values)...
+    {
+    }
+} __attribute__((packed, aligned(1)));
+
+template<typename... TypeListT> struct PackedArgs {
+    typedef std::make_integer_sequence<int, sizeof...(TypeListT)>
+        IndexSequenceType;
+    PackedArgsBase<IndexSequenceType, TypeListT...> m_components;
+
+    explicit PackedArgs(const TypeListT&... a_values)
+        : m_components(a_values...)
+    {
+    }
+};
+}  // namespace pvt
+
+
+
+template<typename FilenameT, typename SpecifierT, typename... ArgListT>
+void
+filefmt(OpaqueExecContextPtr oec, const FilenameT& filename_hash,
+        const SpecifierT& fmt_specification, ArgListT... args)
+{
+    constexpr int32_t count = sizeof...(args);
+    constexpr OSL::EncodedType argTypes[]
+        = { pvt::TypeEncoder<ArgListT>::value... };
+    pvt::PackedArgs<typename pvt::TypeEncoder<ArgListT>::DataType...> argValues {
+        pvt::TypeEncoder<ArgListT>::Encode(args)...
+    };
+
+    rs_filefmt(oec, OSL::ustringhash { filename_hash },
+               OSL::ustringhash { fmt_specification }, count,
+               (count == 0) ? nullptr : argTypes,
+               static_cast<uint32_t>((count == 0) ? 0 : sizeof(argValues)),
+               (count == 0) ? nullptr : reinterpret_cast<uint8_t*>(&argValues));
+}
+
+template<typename SpecifierT, typename... ArgListT>
+void
+printfmt(OpaqueExecContextPtr oec, const SpecifierT& fmt_specification,
+         ArgListT... args)
+{
+    constexpr int32_t count = sizeof...(args);
+    constexpr OSL::EncodedType argTypes[]
+        = { pvt::TypeEncoder<ArgListT>::value... };
+    pvt::PackedArgs<typename pvt::TypeEncoder<ArgListT>::DataType...> argValues {
+        pvt::TypeEncoder<ArgListT>::Encode(args)...
+    };
+
+    rs_printfmt(oec, OSL::ustringhash { fmt_specification }, count,
+                (count == 0) ? nullptr : argTypes,
+                static_cast<uint32_t>((count == 0) ? 0 : sizeof(argValues)),
+                (count == 0) ? nullptr
+                             : reinterpret_cast<uint8_t*>(&argValues));
+}
+
+template<typename SpecifierT, typename... ArgListT>
+void
+errorfmt(OpaqueExecContextPtr oec, const SpecifierT& fmt_specification,
+         ArgListT... args)
+{
+    constexpr int32_t count = sizeof...(args);
+    constexpr OSL::EncodedType argTypes[]
+        = { pvt::TypeEncoder<ArgListT>::value... };
+    pvt::PackedArgs<typename pvt::TypeEncoder<ArgListT>::DataType...> argValues {
+        pvt::TypeEncoder<ArgListT>::Encode(args)...
+    };
+
+    rs_errorfmt(oec, OSL::ustringhash { fmt_specification }, count,
+                (count == 0) ? nullptr : argTypes,
+                static_cast<uint32_t>((count == 0) ? 0 : sizeof(argValues)),
+                (count == 0) ? nullptr
+                             : reinterpret_cast<uint8_t*>(&argValues));
+}
+
+template<typename SpecifierT, typename... ArgListT>
+void
+warningfmt(OpaqueExecContextPtr oec, const SpecifierT& fmt_specification,
+           ArgListT... args)
+{
+    constexpr int32_t count = sizeof...(args);
+    constexpr OSL::EncodedType argTypes[]
+        = { pvt::TypeEncoder<ArgListT>::value... };
+    pvt::PackedArgs<typename pvt::TypeEncoder<ArgListT>::DataType...> argValues {
+        pvt::TypeEncoder<ArgListT>::Encode(args)...
+    };
+
+    rs_warningfmt(oec, OSL::ustringhash { fmt_specification }, count,
+                  (count == 0) ? nullptr : argTypes,
+                  static_cast<uint32_t>((count == 0) ? 0 : sizeof(argValues)),
+                  (count == 0) ? nullptr
+                               : reinterpret_cast<uint8_t*>(&argValues));
+}
+
+
+OSL_NAMESPACE_EXIT