Add SoA raytracer as a CQ test for Intel hardware intrinsic #18839

fiigii · 2018-07-09T22:02:49Z

This PR ports the current SIMD benchmark RayTracer ( using Vector3) to a SoA algorithm using AVX/AVX2 intrinsics. The new benchmark keeps the same shading approach of the original raytracer as much as possible, so they can generate the same images and be compared directly.

Performance data (rendering a 2k image)

Execution time	Windows	Linux
Baseline (RayTracer )	6.00s	4.13s
PacketTracer	0.83s	0.93s
Performance Gains	7.23x	4.44x

Updated the performance data with #19663

The data collected on

Intel Core i9 7900X (Skylake-X) @ 3.3GHz, HT on, Turbo on, 16GB DDR4 2666MHz
Windows 10 and Ubuntu 16.04

VTune characterization (module level)

Windows

Linux

According to the execution time and the module-level VTune data, we can see that

The baseline has worse performance on Windows due to the CRT math library, which ucrtbase.dll takes ~50% execution time.
The Linux CRT math library libm-2.23.so has more reasonable performance (19.10%).
The baseline uses Math.Pow at https://github.com/dotnet/coreclr/blob/master/tests/src/JIT/Performance/CodeQuality/SIMD/RayTracer/Raytracer.cs#L153, which calls into CRT via P/Invoke.
Currently, the SoA ray-tracer (PacketTracer) does not need PInvoke into CRT, as we implemented a brief SIMD VectorMath.Pow in managed code.

The most obvious module-level difference between the baseline and SoA is that

the SoA ray-tracer has higher GC overhead (shown as coreclr.dll/libcoreclr.so/libc-2.23.so)
this issue is logged at https://github.com/dotnet/coreclr/issues/19116 for further discussion.

VTune characterization (managed code)

Windows

Linux

The codegen issues of RyuJIT have been logged at

https://github.com/dotnet/coreclr/issues/19129

VTune characterization (CoreCLR runtime)

Windows

Linux

Close https://github.com/dotnet/coreclr/issues/17798

fiigii · 2018-07-09T22:14:15Z

@CarolEidt @tannergooding @eerhardt @AndyAyersMS @mikedn

tannergooding · 2018-07-09T22:32:14Z

wrote a brief VectorMath library that provides vectorized Exp(), Log(), and Pow() functions needed by the SoA raytracer, but the result precision looks not good (@tannergooding could you help?)

@fiigii, You can find open-source/vectorized versions of these functions here (MIT Licensed, Microsoft Owned): https://github.com/Microsoft/DirectXMath/blob/master/Inc/DirectXMathVector.inl

They should be good starting points for speed/precision.

fiigii · 2018-07-09T22:38:45Z

@tannergooding Thank you so much! Will try to port it to C#.

eerhardt · 2018-07-09T23:20:22Z

the build configuration PacketTracer.csproj is incorrect (The type or namespace name 'Vector256<>' could not be found), @eerhardt could you help?

You need to add a PackageReference to System.Runtime.Intrinsics.Experimental. See

coreclr/tests/src/Common/test_dependencies/test_dependencies.csproj

Lines 25 to 27 in e0e2859

    
           <PackageReference Include="System.Runtime.Intrinsics.Experimental"> 
        
             <Version>$(MicrosoftPrivateCoreFxNETCoreAppPackageVersion)</Version> 
        
           </PackageReference>

as an example.

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/PacketTracer.cs

fiigii · 2018-07-11T22:57:35Z

The precision issue of vectorized Pow() is fixed, and this program no longer crashes after #18849 merged (Thank @tannergooding)

fiigii · 2018-07-11T23:24:10Z

tests/src/JIT/config/benchmark/benchmark.csproj

@@ -103,6 +103,9 @@
    <PackageReference Include="xunit.runner.utility">
      <Version>$(XunitPackageVersion)</Version>
    </PackageReference>
+      <PackageReference Include="System.Runtime.Intrinsics.Experimental"> 
+    <Version>$(MicrosoftPrivateCoreFxNETCoreAppPackageVersion)</Version> 


@eerhardt I added this reference, but this still cannot be built. Do I need to add a new csproj under tests\src\JIT\config? Or do something like #16378? cc @AndyAyersMS

AndyAyersMS · 2018-07-11T23:43:38Z

Benchmarks are still targeting netstandard1.4, and the new package is not compatible:

D:\repos\coreclr\tests\src\JIT\config\benchmark\benchmark.csproj : 
error NU1202: Package System.Runtime.Intrinsics.Experimental 4.6.0-preview1-26704-01 is not compatible with netstandard1.4 (.NETStandard,Version=v1.4). 
Package System.Runtime.Intrinsics.Experimental 4.6.0-preview1-26704-01 supports: netcoreapp2.1 (.NETCoreApp,Version=v2.1) 
[D:\repos\coreclr\tests\build.proj]

Seems like it ought to be simple to update the main benchmark dependence to netcoreapp2.1 or later, but this has proved difficult to fix. See #16126 for some attempts.

You might be able to clone the benchmark config and target netcoreapp2.1 in the cloned version and then refer to for your new test.

fiigii · 2018-07-12T01:25:15Z

tests/src/JIT/config/benchmark+intrinsic/benchmark+intrinsic.csproj

+    <Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
+    <CLRTestKind>BuildOnly</CLRTestKind>
+    <NugetTargetMoniker>.NETCoreApp,Version=v3.0</NugetTargetMoniker>
+    <NugetTargetMonikerShort>netcoreapp3.0</NugetTargetMonikerShort>


@AndyAyersMS Thank you so much for the help. I created a "benchmark+intrinsic.csproj" and target it to netcoreapp3.0., so it can be built now.

fiigii · 2018-07-13T00:35:48Z

All the issues have been addressed, I think this PR is ready to review. @CarolEidt @tannergooding @AndyAyersMS @mikedn @eerhardt PTAL

Now, the SoA raytracer outputs exactly same picture as the original AoS raytracer, and I will provide detailed profiling data later.

fiigii · 2018-07-13T00:37:37Z

@dotnet-bot test Windows_NT x64 Checked jitincompletehwintrinsic please
@dotnet-bot test Windows_NT x64 Checked jitx86hwintrinsicnoavx please
@dotnet-bot test Windows_NT x64 Checked jitx86hwintrinsicnoavx2 please
@dotnet-bot test Windows_NT x64 Checked jitx86hwintrinsicnosimd please
@dotnet-bot test Windows_NT x64 Checked jitnox86hwintrinsic please

@dotnet-bot test Windows_NT x86 Checked jitincompletehwintrinsic please
@dotnet-bot test Windows_NT x86 Checked jitx86hwintrinsicnoavx please
@dotnet-bot test Windows_NT x86 Checked jitx86hwintrinsicnoavx2 please
@dotnet-bot test Windows_NT x86 Checked jitx86hwintrinsicnosimd please
@dotnet-bot test Windows_NT x86 Checked jitnox86hwintrinsic please

@dotnet-bot test Ubuntu x64 Checked jitincompletehwintrinsic please
@dotnet-bot test Ubuntu x64 Checked jitx86hwintrinsicnoavx please
@dotnet-bot test Ubuntu x64 Checked jitx86hwintrinsicnoavx2 please
@dotnet-bot test Ubuntu x64 Checked jitx86hwintrinsicnosimd please
@dotnet-bot test Ubuntu x64 Checked jitnox86hwintrinsic please

tannergooding · 2018-07-13T01:13:43Z

This PR ports the current SIMD benchmark RayTracer ( using Vector3) to a SoA algorithm using AVX/AVX2 intrinsics.

It would be nice to have an AoS version (for direct comparison). The same could be said for a SoA version of the System.Numerics.Vector implementation.

Having an Sse version would also be nice, for benchmarking hardware without AVX/AVX2 support.

None of this required now, of course, but as future "up for grabs" work items.

tannergooding · 2018-07-13T01:17:09Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/Camera.cs

+// The .NET Foundation licenses this file to you under the MIT license.
+// See the LICENSE file in the project root for more information.
+//
+


General nit: It would be useful to format all these documents according to the recommended conventions....

Things like one statement per line, braces on their own line, no stray newlines, no trailing whitespace, a newline after the top level using statements and before the first namespace, etc....

I believe, VS, for the most part, should do this if you run the "Format Document" command

Thanks! Will do.

@fiigii, did this happen? It looks like we still have things like:

Stray or missing newlines

Unsorted usings

Methods entirely on a single line

etc

tannergooding · 2018-07-13T01:19:58Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/Intersections.cs

+    public Vector256<float> Distances;
+    public Vector256<int> ThingIndeces;
+
+    public static readonly Vector256<float> NullDistance = SetAllVector256<float>(float.MaxValue);


Why float.MaxValue, instead of something like NaN or Infinity?

Because we need to compare the NullDistance against the result of Intersect in method MinIntersect.

tannergooding · 2018-07-13T01:20:36Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/Intersections.cs

+internal struct Intersections
+{
+    public Vector256<float> Distances;
+    public Vector256<int> ThingIndeces;


nit: Indices

tannergooding · 2018-07-13T01:21:47Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/ColorPacket.cs

+    public static Int32RGBPacket256 ConvertToIntRGB(this VectorPacket256 colors)
+    {
+        var one = SetAllVector256<float>(1.0f);
+        var max = SetAllVector256<float>(255.0f);


Is a static readonly more efficient, since we don't have JIT support for this yet?

and, even if we did have JIT support, since the constants are per method, a static readonly might still be better...

Thanks, will cache them in static fields.

tannergooding · 2018-07-13T01:22:11Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/Camera.cs

+    public static Camera Create(VectorPacket256 pos, VectorPacket256 lookAt)
+    {
+        VectorPacket256 forward = (lookAt - pos).Normalize();
+        VectorPacket256 down = new VectorPacket256(SetAllVector256<float>(0), SetAllVector256<float>(-1), SetAllVector256<float>(0));


SetAllVector256<float>(0) should be SetZeroVector256?

tannergooding · 2018-07-13T01:22:47Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/Camera.cs

+        VectorPacket256 forward = (lookAt - pos).Normalize();
+        VectorPacket256 down = new VectorPacket256(SetAllVector256<float>(0), SetAllVector256<float>(-1), SetAllVector256<float>(0));
+        VectorPacket256 right = SetAllVector256<float>(1.5f) * VectorPacket256.CrossProduct(forward, down).Normalize();
+        VectorPacket256 up = SetAllVector256<float>(1.5f) * VectorPacket256.CrossProduct(forward, right).Normalize();


Might be more efficient to create a single instance of SetAllVector256(1.5f) and reuse it?

tannergooding · 2018-07-13T01:25:10Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/Intersections.cs

+    {
+        var cmp = Compare(dis, NullDistance, FloatComparisonMode.EqualOrderedNonSignaling);
+        var zero = SetZeroVector256<int>();
+        var mask = Avx2.CompareEqual(zero, zero);


Why are we getting a mask of ones this way?

I think this is the most efficient way, some C++ compilers also generate it.

A comment here would be useful then.

tannergooding · 2018-07-13T01:26:16Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/ObjectPool.cs

+        public ObjectPool(Func<T> generator, IProducerConsumerCollection<T> collection)
+            : base(collection)
+        {
+            if (generator == null) throw new ArgumentNullException("generator");


generator is null?

nameof(generator)?

tannergooding · 2018-07-13T01:26:37Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/ObjectPool.cs

+        public T GetObject()
+        {
+            T value;
+            return base.TryTake(out value) ? value : _generator();


out T value?

tannergooding · 2018-07-13T01:27:14Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/ObjectPool.cs

+        {
+            var items = new List<T>();
+            T value;
+            while (base.TryTake(out value)) items.Add(value);


out T value?

tannergooding · 2018-07-13T01:28:01Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/ObjectPool.cs

+
+        protected override bool TryAdd(T item)
+        {
+            PutObject(item);


Hmmm.... PutObject itself calls base.TryAdd, which returns a bool....

Seems we should have a TryPutObject and shouldn't just always return true

tannergooding · 2018-07-13T01:28:20Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/ObjectPool.cs

+
+        protected override bool TryTake(out T item)
+        {
+            item = GetObject();


Same here...

This file is directly copied from AoS raytracer, dot not need to change.

If it is an existing file, I agree it doesn't need to be changed in this PR.

It would still be nice to get it cleaned up separately, however.

tannergooding · 2018-07-13T01:29:28Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/PacketTracer.cs

+
+    public Packet256Tracer(int _width, int _hight)
+    {
+        if (_width % VectorPacket256.Packet256Size != 0)


Parenthesis around mathematical expressions are helpful 😄

(not everyone may agree, but that is my preference)

tannergooding · 2018-07-13T01:30:10Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/PacketTracer.cs

+internal class Packet256Tracer
+{
+    public int Width { get; private set; }
+    public int Hight { get; private set; }


nit: Height?

tannergooding · 2018-07-27T20:39:04Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/Color.cs

+// See the LICENSE file in the project root for more information.
+//
+
+using System.Runtime.Intrinsics;


These usings are all unused.

tannergooding · 2018-07-27T20:43:15Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/ObjectPacket.cs

+
+internal abstract class ObjectPacket256
+{
+    public Surface Surface { get; private set; }


Don't need private set

tannergooding · 2018-07-27T20:45:54Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/PacketTracer.cs

+    {
+        if ((_width % VectorPacket256.Packet256Size) != 0)
+        {
+            _width += VectorPacket256.Packet256Size - _width % VectorPacket256.Packet256Size;


nit: My preference is to add parenthesis where they help improve readability or where they help clarify operator precedence.

tannergooding · 2018-07-27T20:46:38Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/PacketTracer.cs

+
+    private static readonly Vector256<float> SevenToZero = SetVector256(7f, 6f, 5f, 4f, 3f, 2f, 1f, 0f);
+
+    public Packet256Tracer(int _width, int _height)


Parameters names should not start with _

tannergooding · 2018-07-27T20:47:26Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/PacketTracer.cs

+                Vector256<float> Xs = Add(SetAllVector256(fx), SevenToZero);
+                var dirs = GetPoints(Xs, SetAllVector256<float>(y), camera);
+                var rayPacket256 = new RayPacket256(camera.Pos, dirs);
+                var SoAcolors = TraceRay(rayPacket256, scene, 0);


nit: Make 0 a named parameter to help improve readability

0 seems already very clear as the first value of depth 😄

I was indicating that, if you were to just read TraceRay(rayPacket256, scene, 0);, it isn't obvious that 0 is for depth (you have to go look at the TraceRay signature).

While TraceRay(rayPacket256, scene, depth: 0); makes this immediately obvious

Oh, I see, misunderstood your words...

tannergooding · 2018-07-27T20:48:41Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/PacketTracer.cs

+
+                // `FastTranspose` returns an "incomplete" AoS structure,
+                // which can be written into memory 16-byte by 16-byte.
+                // Now, .NET Core does not guarantee the 32-byte alignment, 


I didn't think .NET Core guaranteed 16-byte alignment either (just that the first local was 16-byte aligned, or something similar)...

tannergooding · 2018-07-27T20:49:29Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/PacketTracer.cs

+                    Avx2.ExtractVector128(output + 20, intAoS.Bs, 1);
+                }
+
+                /* 


This looks like you commented out some unused code?

tannergooding · 2018-07-27T20:54:17Z

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/PacketTracer.cs

+        float heightRate1 = Height / 2.0f;
+        float heightRate2 = Height * 2.0f;
+
+        var recenteredX = Divide(Subtract(x, SetAllVector256(widthRate1)), SetAllVector256(widthRate2));


Is this more or less perforamnt than a SetAllVector256(Width) and SetAllVector256(2) along with a Multiple and Divide call?

fiigii · 2018-07-30T21:39:44Z

@tannergooding Thank you so much for the review, I have addressed your feedback.

fiigii · 2018-07-30T21:40:53Z

logged the CRT issue at https://github.com/dotnet/coreclr/issues/19203

fiigii · 2018-07-31T20:58:47Z

Removed the package dependency on S.R.Intrinsic.Experimental to match the recent CoreFX update.

tannergooding

Overall LGTM.

I would still really like to see some of the document formatting fixed/improved (even if it is an up-for-grabs bug). There are a number of stray or missing newlines, multiple statements on a single line, unused or unsorted usings, code simplifications (sometimes from new language features, sometimes where the code was unnecessarily verbose), etc.

4creators · 2018-08-01T21:47:20Z

@dotnet-bot test OSX10.12 x64 Checked CoreFX Tests
@dotnet-bot test OSX10.12 x64 Checked Innerloop Build and Test

tannergooding · 2018-08-10T14:53:08Z

@dotnet-bot test OSX10.12 x64 Checked CoreFX Tests

tannergooding · 2018-08-10T14:54:07Z

@CarolEidt, @AndyAyersMS, @eerhardt, @fiigii. Unless someone has feedback saying otherwise, I plan on merging this after the tests show as all green.

CarolEidt · 2018-08-10T18:39:57Z

I am fine with merging this. I had hoped to take time to review, but just haven't found the time, and would indeed like to see this added.

I second Tanner's general comments about improving the documentation and formatting, and am fine with making that a future work item. One specific comment I had was that I hadn't (yet) found any attribution about where this came from. The existing RayTracer was derived from a sample program that was available on MSDN, and if this was also derived from that it would be good to note it somewhere.

fiigii · 2018-08-10T18:50:53Z

@CarolEidt Thanks for the comment. I will continue to improve the code/doc of this benchmark in future PRs. Especially, I have made a prototype of struct promotion to address the GC overhead from VectorPacket, so this benchmark may be changed to struct based.

and if this was also derived from that it would be good to note it somewhere.

This benchmark is not derived from the current RayTracer. I am just using its input data (models) and some top-level framework code with a totally new algorithm, so that these two benchmarks can be directly compared. I will add some comments to note the reused code in the next PR. Thanks!

tannergooding · 2018-08-10T20:25:59Z

Merging. The CoreFX tests aren't impacted by this change and are failing with unrelated issues.

mikedn reviewed Jul 10, 2018

View reviewed changes

tests/src/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/PacketTracer.cs Outdated Show resolved Hide resolved

fiigii force-pushed the soabenchmark branch from fd1f17d to 72804ec Compare July 11, 2018 18:51

fiigii commented Jul 11, 2018

View reviewed changes

fiigii force-pushed the soabenchmark branch from 72804ec to 9dea497 Compare July 12, 2018 01:19

fiigii commented Jul 12, 2018

View reviewed changes

fiigii force-pushed the soabenchmark branch from 9dea497 to 8fca2c9 Compare July 13, 2018 00:28

fiigii changed the title ~~[WIP] Add SoA raytracer as a CQ test for Intel hardware intrinsic~~ Add SoA raytracer as a CQ test for Intel hardware intrinsic Jul 13, 2018

tannergooding reviewed Jul 13, 2018

View reviewed changes

tannergooding reviewed Jul 27, 2018

View reviewed changes

fiigii force-pushed the soabenchmark branch from 43e4cbb to 8a554f9 Compare July 30, 2018 21:28

fiigii force-pushed the soabenchmark branch from 8a554f9 to ac2a05f Compare July 31, 2018 20:57

fiigii force-pushed the soabenchmark branch from ac2a05f to 45badbb Compare July 31, 2018 21:25

tannergooding approved these changes Jul 31, 2018

View reviewed changes

Add SoA raytracer as a CQ test for Intel hardware intrinsic

c48e743

fiigii force-pushed the soabenchmark branch from 45badbb to c48e743 Compare August 6, 2018 22:11

tannergooding merged commit b9028b3 into dotnet:master Aug 10, 2018

adamsitnik mentioned this pull request Aug 13, 2018

Port new SoA benchmarks from CoreCLR dotnet/performance#113

Closed

fiigii deleted the soabenchmark branch August 28, 2018 23:58


		private static readonly Vector256<float> SevenToZero = SetVector256(7f, 6f, 5f, 4f, 3f, 2f, 1f, 0f);

		public Packet256Tracer(int _width, int _height)

Add SoA raytracer as a CQ test for Intel hardware intrinsic #18839

Add SoA raytracer as a CQ test for Intel hardware intrinsic #18839

Conversation

fiigii commented Jul 9, 2018 • edited Loading

Performance data (rendering a 2k image)

VTune characterization (module level)

Windows

Linux

VTune characterization (managed code)

Windows

Linux

VTune characterization (CoreCLR runtime)

Windows

Linux

fiigii commented Jul 9, 2018

tannergooding commented Jul 9, 2018

fiigii commented Jul 9, 2018

eerhardt commented Jul 9, 2018

fiigii commented Jul 11, 2018 • edited Loading

Choose a reason for hiding this comment

AndyAyersMS commented Jul 11, 2018

Choose a reason for hiding this comment

fiigii commented Jul 13, 2018

fiigii commented Jul 13, 2018

tannergooding commented Jul 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tannergooding Jul 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fiigii commented Jul 30, 2018

fiigii commented Jul 30, 2018

fiigii commented Jul 31, 2018

tannergooding left a comment

Choose a reason for hiding this comment

4creators commented Aug 1, 2018

tannergooding commented Aug 10, 2018

tannergooding commented Aug 10, 2018

CarolEidt commented Aug 10, 2018

fiigii commented Aug 10, 2018

tannergooding commented Aug 10, 2018

fiigii commented Jul 9, 2018 •

edited

Loading

fiigii commented Jul 11, 2018 •

edited

Loading

tannergooding Jul 27, 2018 •

edited

Loading