DX11 Backend

[sectopod]

Fixes #89, Fixes #849, Fixes #878, Fixes #882, Fixes #897, Fixes #902, Fixes #906

Breaking changes

• TriangleFan is removed
• R8_G8_B8 and R3_G3_B2 surface types are removed, and with them the old Srgb8 shortcut
• scaled channel formats (integers that are seen as floats) are removed
• explicit mipmap generation is removed. You can tell gfx to generate mipmaps on texture data initialization instead.
• Factory::update_*() methods are removed in favor of CommandBuffer-based updates and providing the initial data for resource creation, which now supports multiple slices and mips.
• device capabilities are reduced by cutting out GL-specific stuff.
• there is a new Bind flag set that you'll need to pass for some of the object construction methods
• Usage enum has undergone a rewrite
• there is a new ShaderSet type that may need for the PSO (or shader program) construction
• new structures for SRV/RTV/DSV creation: ResourceDesc, RenderDesc, and DepthStencilDesc
• CommandBuffer interface has minor changes in the scissor and indexed draw semantics, and a major change in the clearing semantics.
• SubmitInfo is gone. Encoder has a single flush method instead of as_buffer and reset.
• Phantom is renamed to Typed
• gfx_constant_struct requires the shader-visible names now, the same as gfx_vertex_struct
• shader version selection has been moved to gfx_app crate. Both core and render crates are now completely API-unaware.

Notable non-breaking changes

• there are new gfx_device_dx11 and gfx_window_dxgi crates. We also set up windows CI, which has GLFW stuff excluded.
• most, if not all, examples are rewritten to use the new gfx_app crate. It is now the only place that knows about all the windowing and API backends in this repository.
• Border mode for samplers
• new convenience type your_pso::Bundle and the constructor your_pso::bundle() are generated for you to contain the slice, PSO itself, and PSO run-time data objects.

Examples update

• most examples simply call launch_default, which selects the default backends for the current platform
• no longer using cast_slice - instead calling create_texture_const_u8
• passing in an array of input slices for texture creation - this saves "skybox" example half a page of code
• parameters are converted to be using the constant buffers for DX11 compatibility. Some are still being mirrored in globals to be compatible with GL2.
• shader versions for different models/backends are now properly specified and selected
• "Target0" name for the final PSO outputs at the moment. DX11 doesn't provide us the name, but it has the type to match on, and GL doesn't care. "Target0" is what we translate built-in DX11 semantic into.

Why DX11?

It's the most established API for high-performance games and graphics application to date. It has the best all-around driver support, variety of powerful tools, and is not cluttered by the legacy (looking at you, OpenGL).

Implementation details

The backend supports both native deferred contexts and emulated (in the same way we do with OpenGL). Once you obtained a device instance with gfx_window_dxgi::init, you can transform it into gfx_device_dx11::Deferred with a simple into() call. The application launcher uses native DX11 contexts by default.

Future work

• Integrate winit when it's ready (Integrate winit #907)
• Implement state caching for the DX11 command buffers

[homu]

☔ The latest upstream changes (presumably #864) made this pull request unmergeable. Please resolve the merge conflicts.

[kvark]

Check out wio for better COM handling: https://github.com/retep998/wio-rs/blob/master/src/com.rs

[sectopod]

Pretty much blocked now on the absence of D3DReflect interface... (cc @retep998) The DX backend may still have flaws in it, but there is only so much I can do without really progressing with shader reflection to get some tests going.

[retep998]

Sorry about that, I'll go finish it up right now. 🐰

[retep998]

I've pushed some stuff to winapi just now. Could you use a cargo local override to test the latest git version of winapi and friends to ensure it works and has everything you need before I cargo publish it?

[sectopod]

@retep998 thank you!

[homu]

☔ The latest upstream changes (presumably #874) made this pull request unmergeable. Please resolve the merge conflicts.

[sectopod]

I've finished the first stage - implementing the basis we use for our examples, including shader reflection with d3dcompile. Now proceeding to the second stage - running the examples, debugging the errors, and fixing them.

@retep998 First problem right away:

note: ld: cannot find -ld3dcompiler
ld: cannot find -ldxguid

I'm using these dependencies for the gfx_device_dx11 project:

d3d11-sys = "0.1"
d3dcompiler-sys = "0.0.1"
dxguid-sys = "0.0.1"
winapi = "0.2"

[retep998]

Ah, you're using the -gnu targets which come with a bundled MinGW that only has a few of the system import libraries, unlike -msvc which would work totally fine.
This should fix it: retep998/winapi-rs@e94ee58

[sectopod]

Thanks @retep998 , this works! I hope it doesn't conflict with -msvc.

[retep998]

I've set up the build scripts such that they only use the bundled import libraries for windows-gnu, so there's no worries of it causing problems for other targets.

[sectopod]

@retep998 Now I'm trying to figure out why D3DReflect fails. Here is how I call it:

let mut reflection = ptr::null_mut();
d3dcompiler::D3DReflect(code.as_ptr() as *const winapi::VOID,
            code.len() as winapi::SIZE_T, &dxguid::IID_ID3D11ShaderReflection, &mut reflection);

I'm getting HRESULT = 0x80004002, which translates to E_NOINTERFACE:

No such interface supported

Maybe the dxguid value for IID_ID3D11ShaderReflection is incorrect? My card doesn't support DX11 feature set, but it still runs D3D11, so I expect the code to be supported.

[sectopod]

There seem to be D3D11Reflect in the docs, that would zero out the chance for improper IID, but it doesn't look like the current bindings provide it.

[sectopod]

@retep998 found something in PixarAnimationStudios/OpenSubdiv#775, makes me a sad panda:

If a client app links against another version of the DX SDK than OSD was linked with (ex. June 2010 and Windows Kit ), then D3DReflect can return a E_NOINTERFACE error, with a null pointer.

[retep998]

Oh, there wasn't any D3D11Reflect in the Windows 10 SDK but there is in the Windows 8.1 SDK. This is a bit confusing. The docs also describe a D3dcompiler_47.lib but I don't see any library with that name in any SDK.

[retep998]

Maybe the dxguid value for IID_ID3D11ShaderReflection is incorrect?

I just compared the IID provided by 8, 8.1, and 10 of the Windows SDK headers to the IID in dxguid-sys and they all look identical.
8: DEFINE_GUID(IID_ID3D11ShaderReflection, 0x8d536ca1, 0x0cca, 0x4956, 0xa8, 0x37, 0x78, 0x69, 0x63, 0x75, 0x55, 0x84);
8.1: DEFINE_GUID(IID_ID3D11ShaderReflection,0x8d536ca1, 0x0cca, 0x4956, 0xa8, 0x37, 0x78, 0x69, 0x63, 0x75, 0x55, 0x84);
10: DEFINE_GUID(IID_ID3D11ShaderReflection, 0x8d536ca1, 0x0cca, 0x4956, 0xa8, 0x37, 0x78, 0x69, 0x63, 0x75, 0x55, 0x84);
Rust: DEFINE_GUID!{IID_ID3D11ShaderReflection, 0x8d536ca1, 0x0cca, 0x4956, 0xa8, 0x37, 0x78, 0x69, 0x63, 0x75, 0x55, 0x84}

Also D3D11Reflect is literally just this:

FORCEINLINE HRESULT
D3D11Reflect(_In_reads_bytes_(SrcDataSize) LPCVOID pSrcData,
             _In_ SIZE_T SrcDataSize,
             _Out_ ID3D11ShaderReflection** ppReflector)
{
    return D3DReflect(pSrcData, SrcDataSize,
                      IID_ID3D11ShaderReflection, (void**)ppReflector);
}

[retep998]

I have DX12 capable hardware so I'd be willing to test it to see if its just a feature level issue. What is a minimal example that I can build and run?

[retep998]

Running the triangle example I get this error:

C:\msys64\home\Peter\gfx [master ↑ +0 ~2 -0 !]> cargo run --example triangle
     Running `target\debug\examples\triangle.exe`
thread '<main>' panicked at 'called `Result::unwrap()` on an `Err` value: DriverType', ../src/libcore\result.rs:746
stack backtrace:
   0:     0x7ff697595877 - rt::lang_start::h2cbcbe89183bad7eW7y
   1:     0x7ff6975950b6 - rt::lang_start::h2cbcbe89183bad7eW7y
   2:     0x7ff697589eff - sys_common::unwind::begin_unwind_inner::h45b2e3feab309e9aLct
   3:     0x7ff69758a61d - sys_common::unwind::begin_unwind_fmt::h3ab52760803c5fb5Rbt
   4:     0x7ff6975915c2 - rust_begin_unwind
   5:     0x7ff69759aed8 - panicking::panic_fmt::h8430b6988b78563f6YL
   6:     0x7ff69749515d - unwrap_failed<gfx_window_dxgi::InitError>
                        at C:\msys64\home\Peter\src\libcore\macros.rs:29
   7:     0x7ff697491adc - unwrap
                        at C:\msys64\home\Peter\src\libcore\result.rs:687
   8:     0x7ff697491386 - launch
                        at C:\msys64\home\Peter\gfx\src\lib.rs:92
   9:     0x7ff6974912f3 - main
                        at C:\msys64\home\Peter\gfx\examples\triangle\main.rs:71
  10:     0x7ff697594a78 - rt::lang_start::h2cbcbe89183bad7eW7y
  11:     0x7ff697591528 - sys_common::thread_local::StaticKey::lazy_init::h7c119cb8af25fd07S2s
  12:     0x7ff6975948fd - rt::lang_start::h2cbcbe89183bad7eW7y
  13:     0x7ff6974a11c9 - main
  14:     0x7ff69759eb77 - __scrt_common_main_seh
                        at f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:264
  15:     0x7fff5ff28101 - BaseThreadInitThunk
Process didn't exit successfully: `target\debug\examples\triangle.exe` (exit code: 101)

[retep998]

After I installed the optional windows feature "Graphical Tools" I got further!

     Running `target\debug\examples\triangle.exe`
Creating swap chain of size 784x561
Success with driver D3D_DRIVER_TYPE(1)
thread '<main>' panicked at 'called `Result::unwrap()` on an `Err` value: DescriptorInit(VertexImport(0, None))', ../src/libcore\result.rs:746
stack backtrace:
   0:     0x7ff7fa9a53d7 - rt::lang_start::h2cbcbe89183bad7eW7y
   1:     0x7ff7fa9a4c16 - rt::lang_start::h2cbcbe89183bad7eW7y
   2:     0x7ff7fa999a5f - sys_common::unwind::begin_unwind_inner::h45b2e3feab309e9aLct
   3:     0x7ff7fa99a17d - sys_common::unwind::begin_unwind_fmt::h3ab52760803c5fb5Rbt
   4:     0x7ff7fa9a10b2 - rust_begin_unwind
   5:     0x7ff7fa9aaa38 - panicking::panic_fmt::h8430b6988b78563f6YL
   6:     0x7ff7fa8a902b - unwrap_failed<gfx::factory::PipelineStateError>
                        at C:\msys64\home\Peter\src\libcore\macros.rs:29
   7:     0x7ff7fa8a8e00 - unwrap
                        at C:\msys64\home\Peter\src\libcore\result.rs:687
   8:     0x7ff7fa8a8991 - new<gfx_device_dx11::factory::Factory>
                        at C:\msys64\home\Peter\gfx\examples\triangle\main.rs:39
   9:     0x7ff7fa8a1537 - launch
                        at C:\msys64\home\Peter\gfx\src\lib.rs:99
  10:     0x7ff7fa8a1263 - main
                        at C:\msys64\home\Peter\gfx\examples\triangle\main.rs:71
  11:     0x7ff7fa9a45d8 - rt::lang_start::h2cbcbe89183bad7eW7y
  12:     0x7ff7fa9a1018 - sys_common::thread_local::StaticKey::lazy_init::h7c119cb8af25fd07S2s
  13:     0x7ff7fa9a445d - rt::lang_start::h2cbcbe89183bad7eW7y
  14:     0x7ff7fa8b1b59 - main
  15:     0x7ff7fa9ae6e7 - __scrt_common_main_seh
                        at f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:264
  16:     0x7fff5ff28101 - BaseThreadInitThunk
Process didn't exit successfully: `target\debug\examples\triangle.exe` (exit code: 101)

[sectopod]

Awesome, so we should be good by using ID3D10ShaderReflection, for now at least. Too bad it's not exposed by winapi-rs :(

[sectopod]

@retep998 Now I'm puzzled.
Added D3D10Shader module myself (retep998/winapi-rs#260), and here is the code now:

    let hr = unsafe {
        d3dcompiler::D3DReflect(code.as_ptr() as *const winapi::VOID,
            code.len() as winapi::SIZE_T, &dxguid::IID_ID3D10ShaderReflection, &mut reflection)
    };

Still getting E_NOINTERFACE. Something is missing here.

[sectopod]

Maybe it's something wrong with my binaries? After all, this error may be interpret as "your binary doesn't have this interface". I'm compiling the shader with this batch:

set FXC="%DXSDK_DIR%\Utilities\bin\x64\fxc.exe"
mkdir data
%FXC% /T vs_4_0 /E Vertex /Fo data/vertex.fx triangle.hlsl
%FXC% /T ps_4_0 /E Pixel /Fo data/pixel.fx triangle.hlsl

EDIT: ah, no, the binaries are fine since @retep998 was able to reflect them

[retep998]

@sectopod Maybe you don't have something installed that is necessary to allow debugging and reflection? I'm on Windows 10 and I had to install the "Graphical Tools" optional system feature which I found out from http://blogs.msdn.com/b/chuckw/archive/2012/11/30/direct3d-sdk-debug-layer-tricks.aspx although not having it caused me to fail at creating the swap chain, not creating the reflection interface. It could possibly also be due to you using the -gnu toolchain (I used x86_64-pc-windows-msvc) which I really hope it isn't but I wouldn't put that beyond the ability of the MinGW linker to screw things up like that. I have a bunch of sdk layer DLLs in my system folder, do you have them too? http://i.imgur.com/zjmlAHY.png

[sectopod]

It's certainly not the "Graphics Tools", since these are installed and only needed for debug runtime.

DLL-wise, I don't have that many:

As for MSVC toolchain, I'm trying this now. Getting something weird upon running the example:

../cargo.exe: error while loading shared libraries: ?: cannot open shared object file: No such file or directory

On a side note, my cargo doesn't like [target.'cfg(windows)'.dependencies] at all, even though it's clearly documented in the manifest format. I need to somehow depend on dx11/dxgi stuff on windows and not depend on glfw in this case. Specifying all the target triples explicitly seems to be rather verbose.

[sectopod]

I think I get it now, was misreading the documentation. Specifying this flag doesn't generate mipmaps automatically. It just enables the capability of calling GenerateMipmap on an SRV later on. Thus, I'm going to enable the flag by default for any surface that may support it. Also, I'm removing the mipmap generation flag completely, since DX11 appears to be incapable of doing that at the resource creation level.

[retep998]

d3d11-sys = "0.2.0"
d3dcompiler-sys = "0.2.0"
dxguid-sys = "0.2.0"

[sectopod]

@retep998 thank you! Will patch the appveyor now and see if it passes through.

[sectopod]

@retep998 don't you have to change the dependencies to "0.2" for these crates and publish the new winapi-0.2.7? It doesn't look like your new sub-crates are used: https://ci.appveyor.com/project/kvark/gfx/build/1.0.19

[retep998]

winapi does not depend on the -sys crates except as a dev dependency for testing purposes. It is entirely up to gfx to specify the correct version of winapi crates to build against. Your src/backend/dx11/Cargo.toml still depends on the older versions of those crates, so you will need to update that file.

[sectopod]

@retep998 I see, thank you! Just verified it builds locally without cargo overrides.
For some reason, my build was failing with image-0.7, so I set 0.6 there, and it works (cc @bvssvni).
Will see what AppVeyor does now.

[sectopod]

Ok, we are green on windows msvc stable and linux stable/beta. Nightly is broken by #904 rustc regression. GNU build is failing on miniz-sys, which is just a dependency issue, and needs to be addressed on the appveyor script (or platform config) level rather than the code side.

I believe we are ready to , at last! @cmr can take the responsibility of 🔨 hitting the big green button.

[retep998]

I updated d3d11-sys too, you know.

[sectopod]

Sorry, I missed it. We'll update this at any point later, since it's not necessary and doesn't break the API.

[emberian]

Fails on nightly and the -gnu target (wrong setup on appveyor?), merging anyway!

[emberian]

🎊 🎈 awesome job @sectopod

[kvark]

🎉 🎉 🎉

[sectopod]

Thank you everyone for the support!

[retep998]

Now gfx can finally live up to its goal of being a a library that abstracts across multiple graphics APIs!