Auto generate tests #80

thargy · 2018-06-02T17:35:15Z

This pull request replaces the manual ShaderBuiltinsTest with a new auto-generated code test suite. The new test (TestBuiltins) accepts a list of methods that can run on both CPU & GPU and then builds the code for each available backend automatically. It then generates random test data and runs multiple iterations against the methods under test, storing the results. Finally, it analyses the differences between backends, and the CPU and produces a report similar to:

40 methods had failures out of 158 (25.32%).

◑ Single ShaderGen.ShaderBuiltins.SmoothStep(Single min, Single max, Single x) failed 523/1000 (52.3%).

SmoothStep(1.200526, 0.3233924, 0.8882229)
Direct3D11, OpenGLES, OpenGL, Vulkan = 0.2900398
CPU = NaN

SmoothStep(-0.1804996, -0.4084838, -0.713079)
Direct3D11, OpenGLES, OpenGL, Vulkan = 1
CPU = NaN

SmoothStep(3.165185, 0.03825688, 0.9647093)
Direct3D11, OpenGLES, OpenGL, Vulkan = 0.788668
CPU = NaN

SmoothStep(1.091138, 0.3107342, -0.7975433)
Direct3D11, OpenGLES, OpenGL, Vulkan = 1
CPU = NaN

SmoothStep(2.810593, -0.6635795, -0.5368141)
Direct3D11, OpenGLES, OpenGL, Vulkan = 0.996103
CPU = NaN

…

Which clearly shows an issue in the way the CPU code for SmoothStep is implemented.

Or:

◑ Vector3 ShaderGen.ShaderBuiltins.Clamp(Vector3 value, Single min, Single max) failed 512/1000 (51.2%).

Clamp(<-0.33386, 0.1280825, -1.845895>, 0.5461884, -0.183241)
Direct3D11, Vulkan = <-0.183241, -0.183241, -0.183241>
OpenGLES, OpenGL = <0.5461884, 0.5461884, 0.5461884>
CPU = <NaN, NaN, NaN>

Clamp(<-0.01004442, -0.02860214, 0.4585574>, 0.02712549, -1.671719)
Direct3D11, Vulkan = <-1.671719, -1.671719, -1.671719>
OpenGLES, OpenGL = <0.02712549, 0.02712549, 0.02712549>
CPU = <NaN, NaN, NaN>

Clamp(<0.1151791, -0.0695346, 1.40435>, 0.861564, -0.4885323)
Direct3D11, Vulkan = <-0.4885323, -0.4885323, -0.4885323>
OpenGLES, OpenGL = <0.861564, 0.861564, 0.861564>
CPU = <NaN, NaN, NaN>

Clamp(<-0.5609558, -0.4562162, -0.234859>, 0.1118884, -0.06781615)
Direct3D11, Vulkan = <-0.06781615, -0.06781615, -0.06781615>
OpenGLES, OpenGL = <0.1118884, 0.1118884, 0.1118884>
CPU = <NaN, NaN, NaN>

Clamp(<0.3038698, 1.671637, -0.04521934>, 0.0905877, -0.04086268)
Direct3D11, Vulkan = <-0.04086268, -0.04086268, -0.04086268>
OpenGLES, OpenGL = <0.0905877, 0.0905877, 0.0905877>
CPU = <NaN, NaN, NaN>

…

From which it much easier to start diagnosing where issues are occurring, particularly as it shows the values that resulted in failures occurring.

…o build into compute shaders for rapid testing and comparison between CPU & GPU backends.

Created outline structure for new way to create test data and test CPU implementations.

…g running tests.

…lts to show rate of inconsistency.

Corrrected format in TestTimer overloads to only show 2 decimal places. Improved formatting of TestBuiltins test output.

… readability.

…o count field.

Failure analysis now gives much more insightful information by grouping results together, making it much easier to see where a failure is occuring.

…possible.

…ide more control over tests. Removed old ShaderBuiltinTests code and cleanedup ready for pull request.

thargy · 2018-06-05T19:46:03Z

Hi @mellinoe, is there any more info you need for this?

mellinoe · 2018-06-05T19:50:28Z

@thargy Sorry, this has been on my list. I'll try to get to it later today or tomorrow.

thargy · 2018-06-05T19:53:01Z

No problems, just wasn't sure if you'd clocked it. I've merged the latest master changes you pushed.

mellinoe · 2018-06-05T21:28:31Z

src/ShaderGen.Tests/AutoGenerated/BuiltinsTests.cs

+                    testSets.Select(t => t.Backend).Where(b => b != null).ToArray(),
+                    null,
+                    null,
+                    csFunctionName);


The "function name" used here should actually just be the function name alone (not including type), because this is passed through to Veldrid. So it should just be "CS". This causes the Metal tests to fail.

Changing to just "CS" threw:

ShaderGen.ShaderGenerationException : The name passed to computeFunctionName must be a fully-qualified type and method.
at ShaderGen.ShaderGenerator..ctor(Compilation compilation, LanguageBackend[] languages, String vertexFunctionName, String fragmentFunctionName, String computeFunctionName, IShaderSetProcessor[] processors) in D:\Source Control\ShaderGen\src\ShaderGen\ShaderGenerator.cs:line 126
at ShaderGen.Tests.AutoGenerated.BuiltinsTests.TestBuiltins() in D:\Source Control\ShaderGen\src\ShaderGen.Tests\AutoGenerated\BuiltinsTests.cs:line 149

So this may be a problem.

Sorry, maybe I commented on the wrong place. ShaderGen indeed wants the fully-qualified name, because it needs all that info to locate the function. When you pass the actual shader code to Veldrid, on the other hand, you should just pass the actual function name ("CS"). I thought the line I commented on was the variable that ultimately goes into Veldrid.

Got you, I just spotted that and have fixed in latest push.

mellinoe · 2018-06-05T21:56:11Z

src/ShaderGen.Tests/AutoGenerated/TestSet.cs

+            string csFunctionName,
+            ITestOutputHelper output)
+        {
+            if (Interlocked.CompareExchange(ref _executed, 1, 0) > 0)


I don't see why this is necessary. It doesn't look like the tests are executed in parallel, or even more than once anyways. Is there something I'm missing?

Nope, soz, force of habit, most of the code I write is multi-threaded, so I'm just used to implementing barriers using this pattern.

This has been changed anyway to fix an issue on compilation failure.

mellinoe · 2018-06-05T22:02:09Z

src/ShaderGen.Tests/ShaderGen.Tests.csproj.DotSettings

@@ -0,0 +1,2 @@
+<wpf:ResourceDictionary xml:space="preserve" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:s="clr-namespace:System;assembly=mscorlib" xmlns:ss="urn:shemas-jetbrains-com:settings-storage-xaml" xmlns:wpf="http://schemas.microsoft.com/winfx/2006/xaml/presentation">


Remove this file

mellinoe · 2018-06-05T22:05:13Z

src/ShaderGen.Tests/Tools/ToolChain.cs

@@ -672,7 +672,7 @@ private static CompileResult FxcCompile(string code, Stage stage, string entryPo
        private static readonly Lazy<string> _glslvPath = new Lazy<string>(
            () =>
            {
-                // First, try to launch from the current environment.
+                // Execute, try to launch from the current environment.


Is this an intentional change?

mellinoe · 2018-06-05T22:06:17Z

src/ShaderGen.Tests/TestUtil.cs

+            for (int i = 0; i < floatCount; i++)
+            {
+                floats[i] = (float)((random.NextDouble() * 2.0 - 1.0) * Math.Pow(2.0, random.Next(minMantissa, maxMantissa)));
+                //floats[i] = (float)(random.NextDouble() * floatRange * 2f) - floatRange;


nit: dead code

mellinoe · 2018-06-05T22:07:08Z

src/ShaderGen.Tests/TestUtil.cs

+        /// <param name="decimalPlaces">The number of decimal places between 0 and 16 (ignored for bytes).</param>
+        /// <param name="breakPoint">The break point between 0 and 1024 (or 0D to base on decimal points).</param>
+        /// <returns>System.String.</returns>
+        public static string ToMemorySize(


Do we actually need all of these overloads? If not, let's just include the one(s) we actually use.

Sorry, it was a cut'n'paste from my other project, I figured I'd put them in here in case they were useful in future. I'll remove the unused overloads.

mellinoe · 2018-06-05T22:08:46Z

src/ShaderGen.Tests/Resource.resx

@@ -0,0 +1,178 @@
+<?xml version="1.0" encoding="utf-8"?>


I think this resource file is pretty overkill for what we need, and it's quite bloated (plus the backing file).

It's auto-generated in VS, and is the way I've always handled big strings. I'll submit an alternate design.

mellinoe · 2018-06-05T22:12:58Z

There's a bit more complexity here than I'd like, but hopefully that will pay off in the future as we work with the codebase more. If you fix the compute shader name (left a comment about it), then all the tests pass for me on my systems. There's a few other misc. things that should be fixed, but overall it looks to be in an okay state.

… makes it easier to implement a check that runs for each element in a vector. Added ShaderBuiltins.FMod. Mod made consistent across backends (as distinct from FMod), by implementing correctly in HLSL (and CPU). Made Pow and Cbrt implementations consistent by using Abs of 1st parameter. This is how Vulkan does it, whereas the other backends all return NaN. As we can't consistently return NaN on backends, I've adopted Vulkan's approach. Corrected CPU Clamp, Lerp and SmoothStep implementations by removing checks (makes consistent with GPU implementations). Corrected CPU Clamp implementation.

…ped from analysis (before it would cause it to show as different), allowing the tests to continue if at least two backends can be compared.

…ads. Minor tweaks as requested in mellinoeGH-80

…GH-80 comments.

thargy · 2018-06-05T23:36:27Z

There's a bit more complexity here than I'd like, but hopefully that will pay off in the future as we work with the codebase more. If you fix the compute shader name (left a comment about it), then all the tests pass for me on my systems. There's a few other misc. things that should be fixed, but overall it looks to be in an okay state.

Though it might appear a little complicated I've tried to make it encapsulated and it can be easily used to test other methods, and, eventually, extended to test GPU-only methods (comparing between backends only and not CPU).

I think it'll pay off as it has taken me about an hour to correctly identify and fix inconsistencies between the various backends, you can see the changes in f9a7b57. The Builtins are now consistent across all backends (and CPU). As it is now testing 1000 random iterations for each of the 165 methods that are implemented on GPU & CPU across all backends, writing manual tests would have been significantly more time consuming and prone to missing things. Only Atanh is currently showing problems on Direct3D.

…S method as CS is now not called from code directly.

thargy · 2018-06-06T00:07:10Z

@mellinoe I believe I've implemented all your suggestions, and with the fix of Atanh I've just pushed (4f40651), then the builtins are all showing as equivalent across all backends (albeit for relatively constrained input values). When you start loosening the constraints on the input floats then the Trig functions start to show inconsistencies (unsurprisingly).

I haven't, however, been able to test on metal, so you'll need to look at the TestBuiltins output to see if Metal shows any inconsistencies against the other backends. (Comparing against just the CPU is a decent start as this has been validated as being close enough to the GPU implemenations already).

mellinoe · 2018-06-06T00:34:49Z

There's a couple of problems on Metal with the latest changes.

I'm pretty sure the index should be computed like this:
int index = (int)ShaderGen.ShaderBuiltins.DispatchThreadID.X;

The cast (UInt3->int) fails on Metal otherwise.

FMod seems to be causing a lot of errors. The problem is this AddCheck method you're using, which ends up using invalid vector accessors. Metal is unusual in that it separates "packed" vectors from "regular" vectors. For our uses, struct fields are packed vectors. Packed vectors cannot be accessed using the ".x", ".y", etc. syntax. Instead, you need to use [0], [1], etc.

mellinoe · 2018-06-06T00:41:23Z

I'm also hitting a crash when the tests complete on Windows:

The active test run was aborted. Reason: Unhandled Exception: Unhandled Exception: System.ObjectDisposedException: Safe handle has been closed
   at System.Runtime.InteropServices.SafeHandle.DangerousAddRef(Boolean& success)
   at System.StubHelpers.StubHelpers.SafeHandleAddRef(SafeHandle pHandle, Boolean& success)
   at Microsoft.Win32.Win32Native.SetEvent(SafeWaitHandle handle)
   at System.Threading.EventWaitHandle.Set()
   at ShaderGen.Tests.Tools.ToolChain.<>c__DisplayClass44_1.<Execute>b__0(Object sender, DataReceivedEventArgs e) in E:\projects\ShaderGen\src\ShaderGen.Tests\Tools\ToolChain.cs:line 536

Fixed packed vector accesor logic in AddCheck on Metal.

mellinoe · 2018-06-06T00:45:41Z

Here's what my fix for AddCheck looks like:

        private static readonly string[] _vectorAccessors = { "0", "1", "2", "3" };
...
    return
        $"{shaderType}({string.Join(",", _vectorAccessors.Take(elementCount).Select(a => check.Replace("`", "[" + a + "]")))})";

thargy · 2018-06-06T00:53:54Z

Here's what my fix for AddCheck looks like:

I already pushed:

            return
                $"{shaderType}({string.Join(",", Enumerable.Range(0, elementCount).Select(a => check.Replace("`", $"[{a}]")))})";

Which should be equivalent (and dropped _vectorAccessors).

I'm also hitting a crash when the tests complete on Windows:

I've never seen that myself, however reviewing the code suggests a possibility. On exit, if one of the tools is still running the test process will start shutting down and disposing objects. The outputWaitHandle will be disposed, as will the errorWaitHandle. If the child process is now terminated/finishes, it may return data, and the OutputDataReceived event will fire, and you would see the error shown.

I've pushed a change that will make sure the process is disposed prior to the waithandles being disposed. Hopefully that will, at the very least, de-register the event handlers and prevent the error you're seeing be posisble.

mellinoe · 2018-06-06T00:59:51Z

I ran it a few times and, while it seems to occur less frequently, it still crashes occasionally (2/6 times currently).

Metal seems to be good now -- no problems there any more.

thargy · 2018-06-06T01:02:43Z

I ran it a few times and, while it seems to occur less frequently, it still crashes occasionally (2/6 times currently).

What test runner are you using on Windows, I'm just running in VS using the R# runner?

Metal seems to be good now -- no problems there any more.

Are there any failures reported in TestBuiltins (you'll still get a tick). If there are no issues the output should end:

Analysing results for failures took 1470.02ms
No failures detected!

mellinoe · 2018-06-06T01:05:07Z

I'm just running dotnet test ShaderGen.Tests.csproj from the command line.

mellinoe · 2018-06-06T01:10:38Z

@thargy Ah, I didn't realize there could be failures while the test still passed. Here's what it outputs:

Analysing results for failures took 824.3ms
7 methods had failures out of 165 (4.24%).

● Vector4 ShaderGen.ShaderBuiltins.Mod(Vector4 a, Vector4 b) failed 950/1000 (95%).

Mod(<-0.1473419, -0.06606695, 3.790649, -3.812556>, <-0.5484863, 1.955159, 0.2919904, -0.2313264>)
CPU = <-0.1473419, 1.889092, 0.2867646, -0.1113338>
Metal = <-0.1473419, -0.06606695, 0.2867646, -0.1113338>

Mod(<-3.672568, 3.733981, -3.982012, 0.04768488>, <0.08522116, -0.6115583, 2.423151, -0.1089887>)
CPU = <0.0771637, -0.5469265, 0.8642893, -0.06130387>
Metal = <-0.008057557, 0.06463158, -1.558861, 0.04768488>

Mod(<-3.337058, 3.664641, -0.04931281, -0.2034206>, <0.1302528, 0.3109234, -1.64931, -0.7285025>)
CPU = <0.04951525, 0.2444837, -0.04931281, -0.2034206>
Metal = <-0.08073749, 0.2444837, -0.04931281, -0.2034206>

Mod(<0.06146766, 0.2269077, -0.08715918, 0.7744274>, <-0.1101076, 0.1933157, -0.08376742, -1.092888>)
CPU = <-0.0486399, 0.033592, -0.003391758, -0.3184605>
Metal = <0.06146766, 0.033592, -0.003391758, 0.7744274>

Mod(<0.2180625, -0.6099011, -0.1890869, -0.1876781>, <0.09211984, -0.14133, 0.2206808, 0.320484>)
CPU = <0.03382286, -0.04458094, 0.03159387, 0.1328059>
Metal = <0.03382286, -0.04458094, -0.1890869, -0.1876781>

…

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

● Vector4 ShaderGen.ShaderBuiltins.Mod(Vector4 a, Single b) failed 936/1000 (93.6%).

Mod(<-1.332414, -0.1519372, 0.3580664, 0.2030616>, -0.2266406)
CPU = <-0.199211, -0.1519372, -0.09521487, -0.02357905>
Metal = <-0.1992111, -0.1519372, 0.1314258, 0.2030616>

Mod(<-0.2432121, -0.4899388, -0.1968114, 1.390035>, 0.7634823)
CPU = <0.5202702, 0.2735436, 0.566671, 0.6265526>
Metal = <-0.2432121, -0.4899388, -0.1968114, 0.6265526>

Mod(<1.875447, 0.08230057, 0.4035302, -0.04449578>, -1.059721)
CPU = <-0.2439942, -0.97742, -0.6561903, -0.04449578>
Metal = <0.8157263, 0.08230057, 0.4035302, -0.04449578>

Mod(<-0.9921335, 0.2405954, 0.02952472, -0.1325384>, -0.3359732)
CPU = <-0.320187, -0.09537782, -0.3064485, -0.1325384>
Metal = <-0.320187, 0.2405954, 0.02952472, -0.1325384>

Mod(<0.7359577, -1.616542, 0.4873287, -0.447501>, 0.6781166)
CPU = <0.05784118, 0.4178079, 0.4873287, 0.2306156>
Metal = <0.05784118, -0.2603086, 0.4873287, -0.447501>

…

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

● Vector3 ShaderGen.ShaderBuiltins.Mod(Vector3 a, Vector3 b) failed 887/1000 (88.7%).

Mod(<-0.06014774, -0.9836, 0.02765148>, <0.1230681, 0.8742101, -0.1192536>)
CPU = <0.06292035, 0.7648202, -0.09160215>
Metal = <-0.06014774, -0.1093899, 0.02765148>

Mod(<0.0284753, -0.1399699, 0.07027472>, <-0.6422678, -0.2835048, 0.1584362>)
CPU = <-0.6137925, -0.1399699, 0.07027472>
Metal = <0.0284753, -0.1399699, 0.07027472>

Mod(<3.830129, -0.2082705, -0.3100274>, <-0.7649199, 3.870798, -0.344797>)
CPU = <-0.7593904, 3.662527, -0.3100274>
Metal = <0.005529761, -0.2082705, -0.3100274>

Mod(<0.1018435, -0.1338812, 0.1493706>, <0.1147475, 3.891926, -1.851492>)
CPU = <0.1018435, 3.758044, -1.702121>
Metal = <0.1018435, -0.1338812, 0.1493706>

Mod(<-0.1772397, 0.1387893, -0.1407941>, <-0.8813902, 0.6458945, 0.04376643>)
CPU = <-0.1772397, 0.1387893, 0.03427166>
Metal = <-0.1772397, 0.1387893, -0.00949477>

…

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◕ Vector3 ShaderGen.ShaderBuiltins.Mod(Vector3 a, Single b) failed 868/1000 (86.8%).

Mod(<-1.485478, 1.124717, -0.3238766>, -0.259662)
CPU = <-0.1871682, -0.1735935, -0.06421456>
Metal = <-0.1871683, 0.08606851, -0.06421456>

Mod(<-0.05152696, 0.1360926, 0.1211376>, 0.1381756)
CPU = <0.08664868, 0.1360926, 0.1211376>
Metal = <-0.05152696, 0.1360926, 0.1211376>

Mod(<0.02352959, -0.5231687, 1.2423>, -0.1055625)
CPU = <-0.08203294, -0.1009186, -0.02445054>
Metal = <0.02352959, -0.1009186, 0.08111196>

Mod(<1.713617, 1.400487, -0.1511525>, 0.3490644)
CPU = <0.3173599, 0.004229307, 0.1979119>
Metal = <0.3173599, 0.004229307, -0.1511525>

Mod(<1.736485, -0.2102596, -1.810122>, -1.510112)
CPU = <-1.283738, -0.2102596, -0.30001>
Metal = <0.2263739, -0.2102596, -0.30001>

…

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◕ Vector2 ShaderGen.ShaderBuiltins.Mod(Vector2 a, Vector2 b) failed 757/1000 (75.7%).

Mod(<-0.06524262, 0.1453021>, <0.1248452, -0.1563947>)
CPU = <0.05960259, -0.01109263>
Metal = <-0.06524262, 0.1453021>

Mod(<-0.24629, -0.1658617>, <-0.1526749, 0.08190227>)
CPU = <-0.09361501, 0.07984507>
Metal = <-0.09361501, -0.002057195>

Mod(<0.1217519, -0.08054987>, <-0.06699717, 0.1960435>)
CPU = <-0.01224243, 0.1154936>
Metal = <0.05475474, -0.08054987>

Mod(<-0.02917885, 0.3808856>, <0.423089, -0.2923595>)
CPU = <0.3939101, -0.2038334>
Metal = <-0.02917885, 0.0885261>

Mod(<-1.068448, 0.1429756>, <0.2403389, -0.2420461>)
CPU = <0.1332461, -0.09907053>
Metal = <-0.1070929, 0.1429756>

…

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◕ Vector2 ShaderGen.ShaderBuiltins.Mod(Vector2 a, Single b) failed 747/1000 (74.7%).

Mod(<-0.4987893, 2.1514>, 0.3335225)
CPU = <0.1682557, 0.150265>
Metal = <-0.1652668, 0.150265>

Mod(<-0.04075132, 0.1077888>, -0.8046364)
CPU = <-0.04075132, -0.6968477>
Metal = <-0.04075132, 0.1077888>

Mod(<3.513057, -0.3537728>, 0.4259948)
CPU = <0.1050982, 0.07222205>
Metal = <0.1050982, -0.3537728>

Mod(<0.09067725, -1.519614>, 0.9782577)
CPU = <0.09067725, 0.4369016>
Metal = <0.09067725, -0.5413561>

Mod(<0.3394809, -0.05812054>, 0.1904949)
CPU = <0.148986, 0.1323744>
Metal = <0.148986, -0.05812054>

…

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◑ Single ShaderGen.ShaderBuiltins.Mod(Single a, Single b) failed 509/1000 (50.9%).

Mod(0.523179, -0.3522496)
CPU = -0.1813202
Metal = 0.1709294

Mod(-0.8329723, 0.1716798)
CPU = 0.02542657
Metal = -0.1462532

Mod(-0.71939, 0.09926502)
CPU = 0.07473016
Metal = -0.02453486

Mod(-0.9161512, 0.2987621)
CPU = 0.278897
Metal = -0.01986507

Mod(-0.2505285, 0.4783684)
CPU = 0.2278398
Metal = -0.2505285

…

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

thargy · 2018-06-06T01:18:17Z

I'm just running dotnet test ShaderGen.Tests.csproj from the command line.

If run this repeatedly now without any failures locally :(.

As for metal, can you try changing line 55 in MetalKnownFunctions:

 {nameof(ShaderBuiltins.Mod), SimpleNameTranslator("fmod")},

to

 {nameof(ShaderBuiltins.Mod), SimpleNameTranslator()},

in the first instance?

mellinoe · 2018-06-06T01:21:11Z

Plain old mod doesn't seem to be a standard Metal function:

 /var/folders/8t/vt43x2517k569rdz9_wbv5dr0000gn/T/tmp5Q37rV.tmp:502:23: error: use of undeclared identifier 'mod'
 parameters.Single_3 = mod(parameters.Single_1, parameters.Single_2);
                       ^

…dition. Add Metal mod operator.

thargy · 2018-06-06T01:26:37Z

Plain old mod doesn't seem to be a standard Metal function:

Apparently, there is a '%' operator that might be equivalent so I've pushed some code for that, if not, it will be similar to the FMod function except it should use floor in place of trunc.

If run this repeatedly now without any failures locally :(.

Finally got a failure! I've pushed a commit with try...catch around the sets for the rare race condition.

mellinoe · 2018-06-06T04:05:46Z

Metal's % operator doesn't seem to work with floats or vectors:

Error: 
/var/folders/8t/vt43x2517k569rdz9_wbv5dr0000gn/T/tmpuMIHxt.tmp:87:10: error: invalid operands to binary expression ('float' and 'float')
    r = f%2;
        ~^~
/var/folders/8t/vt43x2517k569rdz9_wbv5dr0000gn/T/tmpuMIHxt.tmp:88:23: error: invalid operands to binary expression ('float2' (aka 'vector_float2') and 'float2')
    r2 = float2(VH.V2)%float2(float2(2, 4));
         ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/var/folders/8t/vt43x2517k569rdz9_wbv5dr0000gn/T/tmpuMIHxt.tmp:89:23: error: invalid operands to binary expression ('float3' (aka 'vector_float3') and 'float3')
    r3 = float3(VH.V3)%float3(float3(2, 4, 6));
         ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~
/var/folders/8t/vt43x2517k569rdz9_wbv5dr0000gn/T/tmpuMIHxt.tmp:90:23: error: invalid operands to binary expression ('float4' (aka 'vector_float4') and 'float4')
    r4 = float4(VH.V4)%float4(float4(2, 4, 6, 8));
         ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~
4 errors generated.

mellinoe · 2018-06-06T04:08:26Z

it will be similar to the FMod function except it should use floor in place of trunc.

That seems to work well enough.

Analysing results for failures took 724.48ms
1 methods had failures out of 165 (.61%).

○ Vector2 ShaderGen.ShaderBuiltins.Mod(Vector2 a, Single b) failed 1/1000 (.1%).

Mod(<-0.2273049, -2.378017>, -3.592751E-06)
     CPU = <-2.324581E-06, 0>
   Metal = <-2.32918E-06, 8.981442E-08>

…o build into compute shaders for rapid testing and comparison between CPU & GPU backends. Added ToMemorySize() to TestUtils for nicer test output. Created outline structure for new way to create test data and test CPU implementations. Now creates test data, and a result set based on executing on the CPU. Added TestTimer to make it easier to output timing message during long running tests. Runs test on all backends and does naive byte wise comparison of results to show rate of inconsistency. Added some more utilities methods to improve test output. Corrrected format in TestTimer overloads to only show 2 decimal places. Improved formatting of TestBuiltins test output. Move code into AutoGenerated folder, and split out classes to improve readability. Renamed Mappings.Methods to Mappings.MethodMaps and changed Methods to count field. Major refactor to make more OO and easier to understand. Failure analysis now gives much more insightful information by grouping results together, making it much easier to see where a failure is occuring. Added more flexible equality that is based on float comparison where possible. Added float generation control, and float comparison control, to provide more control over tests. Removed old ShaderBuiltinTests code and cleanedup ready for pull request. Added new 'AddCheck' methods to known functions implementations. This makes it easier to implement a check that runs for each element in a vector. Added ShaderBuiltins.FMod. Mod made consistent across backends (as distinct from FMod), by implementing correctly in HLSL (and CPU). Made Pow and Cbrt implementations consistent by using Abs of 1st parameter. This is how Vulkan does it, whereas the other backends all return NaN. As we can't consistently return NaN on backends, I've adopted Vulkan's approach. Corrected CPU Clamp, Lerp and SmoothStep implementations by removing checks (makes consistent with GPU implementations). Corrected CPU Clamp implementation. A failure to compile a backend will not cause the test set to be skipped from analysis (before it would cause it to show as different), allowing the tests to continue if at least two backends can be compared. Fixed issues with Open GL ES floats, and support for MathF.Pow overloads. Minor tweaks as requested in GH-80 Fixed issue with Open GL ES float cast bracket position. Correct to only pass method name into execution code, as per GH-80 comments. Removed resources file as per GH-80 request. Also removed DoCS method as CS is now not called from code directly. Fixed atanh implementaion in HLSL. Fixed bug with extracting thread ID in compute shader. Fixed packed vector accesor logic in AddCheck on Metal. Ensure waithandles are disposed after process by changing using order. Wrapped waithandle set's in try..catch to prevent occasional race condition. Add Metal mod operator.

mellinoe · 2018-06-06T04:14:50Z

@thargy I merged this manually and applied the suggested implementation for Metal's Mod function. Feel free to open up another PR if there's further changes that should be made.

Thanks for the contribution!

thargy added 11 commits May 31, 2018 22:44

Initial commit of auto-generation code. This auto-generates C# code t…

9a50b1d

…o build into compute shaders for rapid testing and comparison between CPU & GPU backends.

Added ToMemorySize() to TestUtils for nicer test output.

eb08e25

Created outline structure for new way to create test data and test CPU implementations.

Now creates test data, and a result set based on executing on the CPU.

dc18c65

Added TestTimer to make it easier to output timing message during lon…

cd4acd0

…g running tests.

Runs test on all backends and does naive byte wise comparison of resu…

6b2c6df

…lts to show rate of inconsistency.

Added some more utilities methods to improve test output.

fa4471e

Corrrected format in TestTimer overloads to only show 2 decimal places. Improved formatting of TestBuiltins test output.

Move code into AutoGenerated folder, and split out classes to improve…

66861b3

… readability.

Renamed Mappings.Methods to Mappings.MethodMaps and changed Methods t…

a5f54a5

…o count field.

Major refactor to make more OO and easier to understand.

f1000ac

Failure analysis now gives much more insightful information by grouping results together, making it much easier to see where a failure is occuring.

Added more flexible equality that is based on float comparison where …

64f819a

…possible.

Added float generation control, and float comparison control, to prov…

0ccdec0

…ide more control over tests. Removed old ShaderBuiltinTests code and cleanedup ready for pull request.

Merge branch 'master' into AutoGenerateTests

14fef61

mellinoe reviewed Jun 5, 2018

View reviewed changes

thargy added 5 commits June 5, 2018 23:34

A failure to compile a backend will not cause the test set to be skip…

22bb0bf

…ped from analysis (before it would cause it to show as different), allowing the tests to continue if at least two backends can be compared.

Fixed issues with Open GL ES floats, and support for MathF.Pow overlo…

ccf1aa0

…ads. Minor tweaks as requested in mellinoeGH-80

Fixed issue with Open GL ES float cast bracket position.

d730c31

Correct to only pass method name into execution code, as per mellinoe…

80230d8

…GH-80 comments.

Removed resources file as per mellinoeGH-80 request. Also removed DoC…

564a26e

…S method as CS is now not called from code directly.

Fixed atanh implementaion in HLSL.

4f40651

Fixed bug with extracting thread ID in compute shader.

adc12a5

Fixed packed vector accesor logic in AddCheck on Metal.

Ensure waithandles are disposed after process by changing using order.

834cd02

Wrapped waithandle set's in try..catch to prevent occasional race con…

fecd9e7

…dition. Add Metal mod operator.

mellinoe closed this Jun 6, 2018

thargy deleted the AutoGenerateTests branch June 6, 2018 07:25

thargy mentioned this pull request Jun 6, 2018

[SUGGESTION] Control compatibility level for method implementations. #81

Open

		@@ -0,0 +1,2 @@
		<wpf:ResourceDictionary xml:space="preserve" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:s="clr-namespace:System;assembly=mscorlib" xmlns:ss="urn:shemas-jetbrains-com:settings-storage-xaml" xmlns:wpf="http://schemas.microsoft.com/winfx/2006/xaml/presentation">

Auto generate tests #80

Auto generate tests #80

Conversation

thargy commented Jun 2, 2018

thargy commented Jun 5, 2018

mellinoe commented Jun 5, 2018

thargy commented Jun 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mellinoe Jun 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mellinoe commented Jun 5, 2018

thargy commented Jun 5, 2018

thargy commented Jun 6, 2018

mellinoe commented Jun 6, 2018

mellinoe commented Jun 6, 2018

mellinoe commented Jun 6, 2018

thargy commented Jun 6, 2018

mellinoe commented Jun 6, 2018

thargy commented Jun 6, 2018

mellinoe commented Jun 6, 2018

mellinoe commented Jun 6, 2018

thargy commented Jun 6, 2018

mellinoe commented Jun 6, 2018

thargy commented Jun 6, 2018

mellinoe commented Jun 6, 2018

mellinoe commented Jun 6, 2018

mellinoe commented Jun 6, 2018

mellinoe Jun 5, 2018 •

edited

Loading