glTF Boombox test model

[emackey]

This draft PR serves as a place for discussion and testing of the work-in-progress glTF node for MaterialX.

Included here are:

• libraries/bxdf/gltf_pbr.mtlx - @proog128's first prototype of the glTF node for MaterialX.
• resources/Materials/Examples/glTFPBR/gltf_pbr_default.mtlx - Sample usage of the glTF node.

This branch also has a simple test model.

1. Build the "MaterialXView" project, and run it.
2. Click "Load Mesh" on resources/Geometry/boombox.obj. (This needs updating to load glTF directly!)
3. Click "Load Material" on resources/Materials/Examples/glTFPBR/gltf_pbr_boombox.mtlx.

The material in step 3 uses the prototype glTF node to display the boombox sample's material. It should look like this:

Known issues:

• Alpha blending not tested yet
• Transmission not working
• @proog128 had a number of concerns about the PBR implementation details (Tobias can you copy those here?)
• Would be fantastic to get refraction and other advanced stuff working in MaterialXView.
• Need a converter to read a glTF material from glTF JSON and automatically construct the corresponding .mtlx definition using the new glTF node of MaterialX.
• Add more issues below...

[CLAassistant]

All committers have signed the CLA.

[proog128]

A couple of issues I ran into when creating the glTF node:

• In glTF, the emission layer is below the clearcoat layer. As a result, there is a directional tint applied to the emission layer. How can this be achieved with MaterialX? MDL recently introduced a directional_factor() function for EDFs, does something similar exist in MaterialX?
• Is it possible to switch between thin and volumetric surfaces based on an input parameter? In glTF, if thickness == 0, the surface should be treated as thin-walled, probably corresponding to the thin_surface MaterialX node. If thickness > 0, the surface should be volumetric, and we should switch to the surface MaterialX node. Is this possible?
• thin_surface does not work in MaterialXView with the GLSL generator. Error message: "Could not find a matching implementation for node 'thin_surface' matching target 'genglsl'". Am I doing something wrong?
• I couldn’t get transmission to work in MaterialXView. Probably a stupid mistake somewhere, I didn’t spend too much time on it.
• I am using generalized_schlick_bsdf almost everywhere, because it is the most flexible node I could find, and as such fits nicely the KHR_materials_specular extension. generalized_schlick_bsdf is parametrized with f0 and 90 color. I compute f0 and f90 from IOR, specular and specular color. However, I wasn’t able to take the IOR of the outside volume into account when computing f0. I had to hard-code it to 1, breaking use-cases with nested dielectrics. Not sure what the best solution is, maybe using dielectric_bsdf instead of generalized_schlick_bsdf for glass would work, but it may lose some flexibility with specular color. Needs more investigation (and maybe a change/addition of parameter in one of the MaterialX BSDFs).
• Again specular color: in KHR_materials_specular we scale the base layer with (1.0 – max(fr(f0.r, f0.g, f0.b)), where f0 is computed from specular, specular color and IOR. Taking the maximum avoids dealing with complementary colors, which could result in unintuitive behavior. I wasn’t able to find a good solution in MaterialX for this.
• Currently I use the following structure to blend between plastic (specular reflection + diffuse reflection) and glass (specular reflection + specular transmission):
  layer(generalized_schlick_bsdf(scatter_reflect), mix(diffuse_bsdf(), generalized_schlick_bsdf(scatter_transmit), transmission_weight))
  Alternatively, I could have used
  mix(generalized_schlick_bsdf(scatter_reflect_transmit), layer(generalized_schlick_bsdf(scatter_reflect), diffuse_bsdf()), transmission_weight)
  Is there a preferred way of doing that? For example, in iray/MDL the first approach gives more darkening at grazing angles for increasing roughness, which is just the result of how fresnel_layer is implemented in MDL (see libbsdf). The second approach is, however, less flexible with regards to specular and specular color.
• Is there a reference ray tracer that natively supports MaterialX to verify the material shader? I didn’t look into the verification topic yet, but especially regarding volumes, refraction and attenuation I was wondering how to check this with MaterialXView. There is an MDL generator which could be used with iray, but that’s not really a renderer with “native” MaterialX support. So how well does this pipeline work? Is it good enough to serve as a reference? Does it really support all of MaterialX and is correct/”lossless” in all corner-cases? Or are there cases in which Iray/MDL works slightly different than expected, and we at the end have to deal not only with glTF/MaterialX conversion issues, but also with issues resulting from a non-lossless MaterialX/MDL conversion?

[fire]

https://gpuopen.com/learn/amd-usd-hydra-blender/ appears to support MaterialX but there are some hitches.

[niklasharrysson]

@proog128 These are great questions. And justified questions since they touch on areas in MaterialX that are still in the design phase or lacking on the implementation side. There a lot of things covered here but I will try to provide some answers.

1) Emission under coat
In an earlier version of MaterialX this was supported through an explicit Fresnel node that could add a directional factor to any node, including EDF's for emission. This was later removed because we found it to be too unrestricted to be able to add directional dependence in this form, for example MDL had no such support at the time. As you point out MDL recently added the directional_factor() for EDF's, so now the situation has improved.

Our current plan for this feature is to handle this through the vertical layering operator, adding a layer(BSDF, EDF) version that can define a BSDF layered over an EDF. The same method we use for "BSDF over BSDF" layering could then be used for "BSDF over EDF" layering, where the EDF is attenuated by the directional dependent albedo from the BSDF on top. Adding this new layer node version is in the proposal/design phase.

2) Thin-walled materials
The thin_surface node was added in an early version to cover the use-case of thin-walled materials. However we never got around to implement it, and that's why you see an error "Could not find a matching implementation...". We have later come to realize the thin-walled property is better handled on material terminal level then on surface shader level. So there is a proposal to change this, removing the thin_surface node and introducing properties for both thin-walled and double-sidedness on the surfacematerial node.

3) Transmission in MaterialXView
We have a simple implementation of alpha blending transmission in MaterialXView. Make sure "Render Transparency" is enabled under Advanced Settings. If you still don't get any results please report an issue on MaterialX Github.

4) Nested dielectrics
This came up as part of the work to standardize OSL closures for MaterialX's PBS library. The design we settled on is to introduce a medium_vdf describing the interior of a dielectric medium, and let this be associated with the dielectric interface (BSDF) using a layer operator: layer(BSDF, VDF). This medium_vdf holds the current IOR traveled through as well as a priority for nested dielectrics.

The signatures for these closures are part of OSL now, you can find medium_vdf here:
https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1371/files#diff-eccfda9f53d75768b0bc580320d4fd9417f3d0d1d644cf3fac000f394f036d72R657

At this point we have settled for the signatures but no implementation is started on yet as far as I know. The plan is to implement all of the closures in OSL "testrender", to serve as a reference implementation of MaterialX PBS nodes. Of interest might also be the full discussions that lead up to these OSL closures, which was done in a PR on MaterialX side: AcademySoftwareFoundation#614

Note that some of the closures are slightly different from the MaterialX node counterparts. When/if the OSL implementation turns out to work well we will incorporate these design changes on MaterialX side.

5) Scale base layer with (1.0 – max(fr(f0.r, f0.g, f0.b))
In MaterialX the standard method we use for scaling the base is directional albedo. For dielectric_bsdf the tint color is not involved in that calculation so we should get no complementary colors (https://github.com/materialx/MaterialX/blob/main/libraries/pbrlib/genglsl/mx_dielectric_bsdf.glsl#L31)

For generalized_schlick_bsdf, where F0/F90 affects the albedo we take the average of the resulting color components.
(https://github.com/materialx/MaterialX/blob/main/libraries/pbrlib/genglsl/mx_generalized_schlick_bsdf.glsl#L31)
So I don't think we should have a problem with complementary colors here either, but please let us know if you find issues.

6) Blending between plastic and glass
I think the most natural setup is your second example. I would say the recommended way to describe glass is dielectric_bsdf(scatter_reflect_transmit). A renderers can then handled the full spherical distribution as a single BSDF. But as mentioned you loose the separation of controls. So if you need the controls layer(dielectric_bsdf(scatter_reflect), dielectric_bsdf(scatter_transmit)) has to be used instead.

An alternative we are considering, which also is part of the new OSL closure definitions, is to have tint controls over the two lobes instead of the scatter_mode enum with just on or off:
https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1371/files#diff-eccfda9f53d75768b0bc580320d4fd9417f3d0d1d644cf3fac000f394f036d72R524

7) Reference ray tracer
As mentioned above the goal is for OSL "testrender" to become this reference, with open source implementations of all the closures. Unfortunately none of the teams involved have had the bandwidth to start working on the implementation yet.

[proog128]

@niklasharrysson Thanks for the answers and sorry for the delay!

1) Emission under coat
2) Thin-walled materials
7) Reference ray tracer

Looking forward to these features!

3) Transmission in MaterialXView

Unfortunately, "Render Transparency" doesn't help. @bernardkwok wants to look into this issue in MaterialXView.

4) Nested dielectrics

Thanks for the links, very interesting discussions! Is this medium_vdf part of a surfaceshader or does it require a volumeshader? I am struggling with adding volume effects/VDFs to the material. I tried it in pull request #2, but it doesn't work. I am not sure if it fails due to missing features in MaterialXView or because I am doing something wrong. I would appreciate it if you have a hint.

5) Scale base layer with (1.0 – max(fr(f0.r, f0.g, f0.b))

For generalized_schlick_bsdf, where F0/F90 affects the albedo we take the average of the resulting color components.

Oh, I see, that makes sense. The average avoids complementary colors. Does it also avoid energy gain in each color channel? If for example f0 = f90 = [1, 1, 0], then the average is 0.66. In

result = D * F * G * comp * occlusion * weight / (4.0 * NdotV)
       + base * (1.0 - avgDirAlbedo * weight)

the final color is computed by summing (per channel) LayerBSDF * fresnel + BaseBSDF * (1.0 - average(albedo(LayerBSDF * fresnel))). For red and green channels, we end up with LayerBSDF * 1 + BaseBSDF * (1.0 - 0.66 * stuff), which is greater than 1. Can you confirm this observation? Am I missing something?

6) Blending between plastic and glass

Separate tint for reflection and transmission is a step into the right direction (for glTF), but not enough when considering specular color. Specular tint in glTF is angle-dependent. For a single-channel weight specular and a RGB f0 color specular_color, glTF computes

fresnel = schlick_fresnel(f0 = toF0(ior = 1.5) * specular_color)
dielectric_brdf = specular_brdf * specular * fresnel + diffuse_brdf * (1 - sepcular * maxValue(fresnel))

whereas the dielectric_bsdf computes

fresnel = dielectric_fresnel(ior = 1.5)
dielectric_bsdf = specular_brdf * fresnel * reflection_tint + diffuse_brdf * (1 - fresnel) * transmission_tint

if I understood it correctly. Any chances that MaterialX includes a function that is compatible to glTF? Otherwise we could go with layer(dielectric_bsdf(scatter_reflect), dielectric_bsdf(scatter_transmit)) as you mentioned. Maybe we could switch between scatter_reflect_transmit and separate scatter_reflect/scatter_transmit based on whether a sepcular color is set. This would complicate the node graph, but may have better rendering quality if the default specular color (=[1,1,1]) is used.

[niklasharrysson]

3) Transmission in MaterialXView

Great, I will sync up with Bernard on that.

4) Nested dielectrics

Yes this is because of missing support in MaterialXView. Volumetrics in general is missing implementations in our GLSL codegen. The medium_vdf closure I mentioned is what we came up with to describe nested dielectrics in OSL, and when this has been implemented and proved to work out well we will add a corresponding MaterialX VDF node. So this is still in the early stages and not ready for use yet. (we know it works well in for example Arnold but we need more work to confirm it will work well in general).

But to answer your question, when the medium_vdf node is available our thinking is that it will mainly be used as part of a surfaceshader, as a component in a layer stack. Typically you will associate it with a transmissive BSDF, to describe it's interior medium, by layering it as the base on a BSDF-over-VDF version of the layer operator.

layer(dielectric_bsdf(scatter_reflect_transmitt), medium_vdf())

5) Scale base layer with (1.0 – max(fr(f0.r, f0.g, f0.b))

Yes your observation is correct. Since we use the average in this case there might be a problem with energy conservation in some cases. We have not seen that as a problem in practice yet, with a saturated/non-white tint for a dielectric it's a non-physical setup anyway. But using max of the components instead of average would be a better strategy to avoid this. Thanks for pointing that out.

6) Blending between plastic and glass

Actually the transmission_tint would not be applied in this case of layering a dielectric over a diffuse, these tints are internal to the dielectric BSDF where transmission_tint would control the tint of the transmission lobe.

But in terms of the fresnel effect you are right that with dielectric_bsdf this is monochromatic. I think the generalized_schlick_bsdf would be a better candidate to achieve what you are looking for, since it's using Schlick fresnel similar to glTF (with a difference being the use of average instead of maxValue, but we should probably change that).

Also want to clarify the statement I made about using dielectric_bsdf for glass, this was just a recommendation as dielectric_bsdf is the node that is most simple to use for physical glass. But for more artistic controls generalized_schlick_bsdf is a better candidate. We just have to make sure it's working with transmission as well on the implementation side. We will look into that.

[proog128]

Thanks for the answers and for bringing the topics into the MaterialX repo.

4) Nested dielectrics and transmission color

One thing I didn't understand yet is how the IOR ratio of nested volumes impacts generalized_schlick_bsdf. Does the new medium_vdf and its IOR impact f0 and f90 of generalized_schlick_bsdf? We compute f0 from outside and inside IOR, specular and specular color parameters:

f0 = specular * min(specular_color * ((inside_ior - outside_ior) / (inside_ior + outside_ior))^2)
f90 = specular

(As specular is part of f0 and f90, it could be passed as weight to generalized_schlick_bsdf, if I am not mistaken).

Is it possible to access the outside_ior in the graph?

Another question: does the MDL backend support volumes already? Would it be possible to test volumes in #2 with the MDL code generator and Iray or the MDL-SDK DXR renderer? What is the workflow for generating MDL code? So far I used MaterialXView and pressed the M key to export MDL. But this isn't going to work if the MaterialX file with volumes cannot be loaded properly.

[niklasharrysson]

Hi Tobias, I'll try to provide some more answers here:

Could specular be passed as weight to generalized_schlick_bsdf?

Yes you should be able to pass your specular as weight since this is a weight for the overall BSDF contribution. So setting f0 to this result calculated from IOR, setting f90 to white, and setting weight from specular.

Is it possible to access the outside_ior in the graph?

There is no control of the "outside" IOR in the material description. The current IOR has to be tracked by the renderer when a new interface is entered. The initial outside IOR could for example be set as a global property for all outside media, or set and tracked from the camera origin, or assumed as IOR=1.0 for vacuum. But it's a renderer problem to track IOR, since the outside changes when entering a volume, and only the renderer has knowledge about how you arrived at a new interface.

MDL and volumetric support?

About volumetrics in general I've taken a closer look at where we stand now, and unfortunately I would have to say we are still too early in the implementation stages for this to be of practical use yet.

For MDL we do have codegen support for the volume shader and the VDF nodes. But there is no good support to combine volumes with a surface shader in order to achieve what you want in #2

There is no volumeshader input on the surfacematerial node, so that is probably why you get the "No renderable elements found..." error message. Instead, as mentioned above, adding a volume interior to a surface shader is designed to be done using a BSDF-over-VDF layer operator, but this is not implemented in any of the language codegens yet.

The quickest path forward would probably be to implemented this layer operator for the MDL codegen, and then one could start experimenting with volumes inside surface shaders in practice.

[niklasharrysson]

The quickest path forward would probably be to implemented this layer operator for the MDL codegen, and then one could start experimenting with volumes inside surface shaders in practice.

I should add that the BSDF-over-VDF layer operator is something we are working on as part of a larger task to improve our BSDF implementations. So hopefully we can have that working in MDL "soon". I'll ping again when we have something to test.

[proog128]

MDL and volumetric support?

OK, thanks for the clarification. I will be happy to test it once it's available!

Is it possible to access the outside_ior in the graph?

How does the generalized_schlick_bsdf react to the tracked IOR? So far I was assuming that only the refraction effect, e.g, the direction into which the ray is transmitted, is affected by the tracked IOR. So the refracted direction is based on the tracked IOR (i.e., the outside_ior or the incident_ior) and the IOR from the medium_vdf (i.e., the inside_ior or the transmitted_ior). The Fresnel-based blending weight, on the other hand, depends solely on the colors given by f0 and f90 parameters. Now the problems is that computing the correct value for f0 needs access to the IORs from both sides of the surface (or the ratio). That's btw. also an unsolved problem for writing and MDL shader for the glTF BSDF.

The dielectric_bsdf doesn't have this problem, because it gets the (transmitted) IOR as input and can compute the Fresnel effect internally based on the two IORs.

The background here is that the glTF BSDF by default (i.e., specular =1, specular color = 1, (transmitted_)ior = 1.5) should behave "physically-correct". On top of that, and only if required, specular and specular color can be used to optimize f0 for certain cases which are not handled by such a simplistic model (and thus require artistic tweaking), for example fabrics. But the effect of specular and specular color on f0 is always relative to what was computed from the IOR. Specular and specular color do not override the f0-from-IOR, it's a multiplication.

[niklasharrysson]

MDL and volumetric support?

Just a quick update that we have support for the BSDF-over-VDF layer operator now, so it should be possible to start experimenting with this in MDL, using an MDL renderer that supports volumetric rendering (e.g. Iray).

This layer operator is implemented for GLSL and OSL as well, but for those targets we don't have the VDF itself yet. For GLSL Lucasfilm is working on this, and some other improvements to transmission, so hopefully we will have better transmission, with absorption, in MaterialXView as well soon.

IOR and generalized_schlick_bsdf
I've asked if someone from the Lucasfilm team could give more details on how IOR is handled for generalized_schlick_bsdf.

[jstone-lucasfilm]

It sounds like this question is specifically about how to handle exterior IOR with generalized_schlick_bsdf, since this node has no IOR input for the renderer to dynamically modify based on the exterior medium. This is a great question, and we may need to add an exterior_ior input to that node (and other similar nodes in the future) in order to correctly handle nested dielectrics. I would be open to that change if it seems logical to others, and we could discuss this proposal at the next MaterialX TSC.

[proog128]

@niklasharrysson That sounds great, I will try it out in the next days, thank you very much.

@jstone-lucasfilm Yes, the question is about handling of exterior IOR and nested dielectrics in generalized_schlick_bsdf.

We need to compute f0 and f90 based on color inputs (called "specular color" and "specular" in glTF), interior IOR (called "ior" in glTF) and exterior IOR. Exterior IOR must be provided by the renderer.

Do you mean an explicit external_ior input on generalized_schlick_bsdf, next to weight, color0, color90? Or an implicit exterior_ior, internally set by the renderer (based on the current nested dielectrics stack) and passed to the function that evaluates generalized_schlick_bsdf?

[jstone-lucasfilm]

@proog128 That's a good question to discuss, and this could either be an explicit exterior_ior input on BSDF nodes, or a MaterialX code generator setting that is automatically passed to BSDF nodes as needed. We're planning to add this topic to our MaterialX TSC meeting tomorrow (10am Pacific Time), and you're very welcome to join us there if you have time.

Here's the ASWF Calendar with a Zoom link for the MaterialX TSC meeting:

https://www.aswf.io/meeting-calendar/

And here are instructions for joining the MaterialX channel of the ASWF Slack, where our meeting agendas are organized, and proposals for new features can be brought up:

https://www.materialx.org/DiscForum.html

[proog128]

Thanks for the invitation, looking forward to discussing this in the meeting.

[proog128]

Transmission and volumes using the BSDF-over-layer operator works, thanks @niklasharrysson. I use the following structure:

layer(
    top = generalized_schlick_bsdf(
        color0 = min(toF0(ior) * specularColor, 1) * specular,
        color90 = specular,
        roughness = roughness,
        scatter_mode = 'R'),
    base = layer(
        top = dielectric_bsdf(
            tint = baseColor,
            ior = ior,
            roughness = roughness,
            scatter_mode = 'T'),
        base = anisotropic_vdf(...)))

A couple of images rendered in the dxr example from the MDL SDK (using MDL codegen):

Attenuation:

Transmission Color:

Rough Reflection and Transmission:

Transmission and specular color:

Thin walled (glTF volume thickness = 0) is still to be done. But I think this is a task for next year :-)

[niklasharrysson]

@proog128 That's great news! Thanks for sharing the results.

Sounds good to defer work on thin-walled materials to next year. We are revisiting the description of thin-walled and double sided materials in MaterialX. So work on that might be better to delay until the new description is in place.

[kwokcb]

Hi @emackey, @jstone-lucasfilm, @proog128
As Tobias has put up the PR to merge the PBR definition and I have the glTF geometry part loading, I'd like to merge in the boombox example over to ASWF MaterialX main if that's agreeable. Reading glTF directly to MaterialX I have started but
would like that as a separate commit as I'm still thinking how best to add this in.

I am running tests in a local fork first (kwokcb#14) and if that' passes will put up an ASWF PR.
Of course all pending the PBR definition PR getting checked in first.

[emackey]

@kwokcb Sounds great! I can open a new PR for my BoomBox sample material, if you like. I guess I need to figure out how to sign the CLA?

[kwokcb]

I've finished testing and added some small updates so that the default and boombox files get included as part of unit testing and CI. Results are in the PR.

[kwokcb]

The ASWF PR is now up here: AcademySoftwareFoundation#870.

[emackey]

Closing this draft. Further discussion in: AcademySoftwareFoundation#870