[wasm-mt] Issues with enabling threads in .NET 7 RC 1 #74654

lambdageek · 2022-08-26T15:11:04Z

Using the wasm-experimental workload it shoudl be possible to create multi-threaded wasmbrowser projects in .NET 7 RC1

Issues:

First group - fixed on main by [wasm-mt] Fix pack/build issues in threaded builds #75162 and on release/7.0 by [release/7.0][wasm-mt] Fix pack/build issues in threaded builds #75171
- WasmGenerateAppBundle doesn't create multi-threaded builds with WasmEnableThreads - only WasmBuildNative does - due to dotnet.worker.js missing from the runtime pack
- WorkloadManifest.targets.in uses WasmEnableThreading instead of WasmEnableThreads
- The CoreLibs in the multithread and perftrace build variants don't define the correct feature flags.
- dotnet new wasmbrowser creates browser.csproj, not <dirname>.csproj
~~dotnet new wasmbrowser does not create runtimeconfig.template.json required for dotnet run~~ this looks like a local caching issue
[wasm-mt] Emscripten SDK runtime pack is missing multi-threaded static libraries #75263

Detailed reproduction steps

Download a net7 RC 1 nightly tar.gz from dotnet/installer (I used the "Release/7.0.1xx-rc1 (7.0.x Runtime)" column)
Unpack into ${HOME}/work/net7-nightly

Set DOTNET_ROOT and PATH:

export DOTNET_ROOT="${HOME}/work/net7-nightly"
export PATH="${DOTNET_ROOT}:${PATH}"

Create the directory ${HOME}/work/net7-playground and add the following into ${HOME}/work/net7-playground/NuGet.config

<configuration>
  <packageSources>
    <add key="dotnet7" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet7/nuget/v3/index.json" />
  </packageSources>
</configuration>

From the net7-playground directory (so that NuGet.config is in the current dir), install the wasm-experimental workload:
```
cd ~/work/net7-playground
dotnet workload install wasm-experimental
```
Make a sample project directory ${HOME}/work/net7-playground/hithread and create a wasmbrowser project:
```
cd ~/work/net7-playground
mkdir hithread
cd hithread
dotnet new wasmbrowser
```
Bug 1 note that there is browser.csproj not hithread.csproj - other templates create a .csproj file with the same name as the directory.

Bug 2 create the missing runtimeconfig.template.json with teh following content

{
    "wasmHostProperties": {
        "perHostConfig": [
            {
                "name": "browser",
                "html-path": "index.html",
                "Host": "browser"
            }
        ]
    }
}

Add the follwoing to your browser.csproj

<PropertyGroup>
  <WasmEnableThreads>true</WasmEnableThreads>
  <WasmEnableThreading>true</WasmEnableThreading>
  <WasmBuildNative>true</WasmBuildNative>
</PropertyGroup>

Bug 3 note that WorkloadManifest.targets.in has a typo and uses WasmEnableThreading instead of WasmEnableThreads

Run dotnet build

$ dotnet build
MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
Determining projects to restore...
Restored /Users/alklig/work/net7-playground/hithread/browser.csproj (in 2 ms).
/Users/alklig/work/net7-nightly/sdk/7.0.100-rc.2.22425.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(219,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/Users/alklig/work/net7-playground/hithread/browser.csproj]
browser -> /Users/alklig/work/net7-playground/hithread/bin/Debug/net7.0/browser-wasm/browser.dll
Compiling native assets with emcc with -O0. This may take a while ...
[2/3] corebindings.c -> corebindings.o [took 0.384s]
[1/3] pinvoke.c -> pinvoke.o [took 0.384s]
[3/3] driver.c -> driver.o [took 0.416s]
Linking for initial memory $(EmccInitialHeapSize)=536870912 bytes. Set this msbuild property to change the value.
Linking with emcc with -O0. This may take a while ...
...
Exception: FROZEN_CACHE is set, but cache file is missing: "sysroot/lib/wasm32-emscripten/libGL-mt.a" (in cache root path "/Users/alklig/work/net7-nightly/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.osx-x64/8.0.0-alpha.1.22415.5/tools/emscripten/cache")
/Users/alklig/work/net7-nightly/packs/Microsoft.NET.Runtime.WebAssembly.Sdk/7.0.0-rc.1.22422.12/Sdk/WasmApp.Native.targets(422,5): error MSB3073: The command "emcc "@/Users/alklig/work/net7-nightly/packs/Microsoft.NETCore.App.Runtime.Mono.multithread.browser-wasm/7.0.0-rc.1.22422.12/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/Users/alklig/work/net7-nightly/packs/Microsoft.NETCore.App.Runtime.Mono.multithread.browser-wasm/7.0.0-rc.1.22422.12/runtimes/browser-wasm/native/src/emcc-link.rsp" "@/Users/alklig/work/net7-playground/hithread/obj/Debug/net7.0/browser-wasm/wasm/for-build/emcc-link.rsp"" exited with code 1. [/Users/alklig/work/net7-playground/hithread/browser.csproj]

Bug 4 We removed the *-mt.a static libraries in our EMSDK pack
9. Copy *-mt.a from an upstream EMSDK install into the emsdk pack:

cp ~/work/dotnet-runtime/runtime/src/mono/wasm/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/*-mt.a ~/work/net7-nightly/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.osx-x64/8.0.0-alpha.1.22415.5/tools/emscripten/cache/sysroot/lib/wasm32-emscripten

dotnet build should now succeed.
dotnet run shoudl serve the app. Open the URL in Chrome and open dev tools and hit reload. You should see something like this in the console:

MONO_WASM: worker initializing essential C exports and APIs
MONO_WASM: worker initializing essential C exports and APIs
MONO_WASM: worker initializing essential C exports and APIs
MONO_WASM: worker initializing essential C exports and APIs
dotnet.js:5 MONO_WASM: afterLoadWasmModuleToWorker added message event handler Worker {onmessage: ƒ, onerror: ƒ}
dotnet.js:5 MONO_WASM: afterLoadWasmModuleToWorker added message event handler Worker {onmessage: ƒ, onerror: ƒ}
dotnet.js:5 MONO_WASM: afterLoadWasmModuleToWorker added message event handler Worker {onmessage: ƒ, onerror: ƒ}
dotnet.js:5 MONO_WASM: afterLoadWasmModuleToWorker added message event handler Worker {onmessage: ƒ, onerror: ƒ}
dotnet.js:5 MONO_WASM: worker initializing essential C exports and APIs
dotnet.js:5 MONO_WASM: worker initializing essential C exports and APIs
dotnet.js:5 MONO_WASM: worker initializing essential C exports and APIs
dotnet.js:5 MONO_WASM: worker initializing essential C exports and APIs
dotnet.js:3 mono_wasm_runtime_ready fe00e07a-5519-4dfe-b35a-f867dbaf2e28
dotnet.js:2013 Hello, Console!

Clean the bin and obj directories.
Change the browser.csproj like this:

<PropertyGroup>
  <WasmEnableThreads>true</WasmEnableThreads>
  <WasmEnableThreading>true</WasmEnableThreading>
  <WasmGenerateAppBundle>true</WasmGenerateAppBundle>
</PropertyGroup>

That is, replace WasmBuildNative by WasmGenerateAppBundle
12. run dotnet build again. Note that dotnet.worker.js isn't in the AppBundle directory. When running
the app prints 404s when trying to load it. From the Chrome DevTools console:

GET http://127.0.0.1:9000/dotnet.worker.js 404 (Not Found)
dotnet.worker.js:1          GET http://127.0.0.1:9000/dotnet.worker.js 404 (Not Found)
dotnet.worker.js:1          GET http://127.0.0.1:9000/dotnet.worker.js 404 (Not Found)
dotnet.worker.js:1          GET http://127.0.0.1:9000/dotnet.worker.js 404 (Not Found)
dotnet.js:5 MONO_WASM: afterLoadWasmModuleToWorker added message event handler Worker {onmessage: ƒ, onerror: ƒ}
dotnet.js:5 MONO_WASM: afterLoadWasmModuleToWorker added message event handler Worker {onmessage: ƒ, onerror: ƒ}
dotnet.js:5 MONO_WASM: afterLoadWasmModuleToWorker added message event handler Worker {onmessage: ƒ, onerror: ƒ}
dotnet.js:5 MONO_WASM: afterLoadWasmModuleToWorker added message event handler Worker {onmessage: ƒ, onerror: ƒ}

The text was updated successfully, but these errors were encountered:

ghost · 2022-08-26T15:11:21Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Using the wasm-experimental workload it shoudl be possible to create multi-threaded wasmbrowser projects in .NET 7 RC1

Issues:

dotnet new wasmbrowser creates browser.csproj, not <dirname>.csproj
dotnet new wasmbrowser does not create runtimeconfig.template.json required for dotnet run
WorkloadManifest.targets.in uses WasmEnableThreading instead of WasmEnableThreads
EMSDK nuget is missing *-mt.a libs needed for multi-threaded builds due to Delete selected libs from cache to shrink size emsdk#43
WasmGenerateAppBundle doesn't create multi-threaded builds with WasmEnableThreads - only WasmBuildNative does

Detailed reproduction steps

Download a net7 RC 1 nightly tar.gz from dotnet/installer (I used the "Release/7.0.1xx-rc1 (7.0.x Runtime)" column)
Unpack into ${HOME}/work/net7-nightly

Set DOTNET_ROOT and PATH:

export DOTNET_ROOT="${HOME}/work/net7-nightly"
export PATH="${DOTNET_ROOT}:${PATH}"

Create the directory ${HOME}/work/net7-playground and add the following into ${HOME}/work/net7-playground/NuGet.config

<configuration>
  <packageSources>
    <add key="dotnet7" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet7/nuget/v3/index.json" />
  </packageSources>
</configuration>

From the net7-playground directory (so that NuGet.config is in the current dir), install the wasm-experimental workload:
```
cd ~/work/net7-playground
dotnet workload install wasm-experimental
```
Make a sample project directory ${HOME}/work/net7-playground/hithread and create a wasmbrowser project:
```
cd ~/work/net7-playground
mkdir hithread
cd hithread
dotnet new wasmbrowser
```
Bug 1 note that there is browser.csproj not hithread.csproj - other templates create a .csproj file with the same name as the directory.

Bug 2 create the missing runtimeconfig.template.json with teh following content

{
    "wasmHostProperties": {
        "perHostConfig": [
            {
                "name": "browser",
                "html-path": "index.html",
                "Host": "browser"
            }
        ]
    }
}

Add the follwoing to your browser.csproj

<PropertyGroup>
  <WasmEnableThreads>true</WasmEnableThreads>
  <WasmEnableThreading>true</WasmEnableThreading>
  <WasmBuildNative>true</WasmBuildNative>
</PropertyGroup>

Bug 3 note that WorkloadManifest.targets.in has a typo and uses WasmEnableThreading instead of WasmEnableThreads

Run dotnet build

$ dotnet build
MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
Determining projects to restore...
Restored /Users/alklig/work/net7-playground/hithread/browser.csproj (in 2 ms).
/Users/alklig/work/net7-nightly/sdk/7.0.100-rc.2.22425.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(219,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/Users/alklig/work/net7-playground/hithread/browser.csproj]
browser -> /Users/alklig/work/net7-playground/hithread/bin/Debug/net7.0/browser-wasm/browser.dll
Compiling native assets with emcc with -O0. This may take a while ...
[2/3] corebindings.c -> corebindings.o [took 0.384s]
[1/3] pinvoke.c -> pinvoke.o [took 0.384s]
[3/3] driver.c -> driver.o [took 0.416s]
Linking for initial memory $(EmccInitialHeapSize)=536870912 bytes. Set this msbuild property to change the value.
Linking with emcc with -O0. This may take a while ...
...
Exception: FROZEN_CACHE is set, but cache file is missing: "sysroot/lib/wasm32-emscripten/libGL-mt.a" (in cache root path "/Users/alklig/work/net7-nightly/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.osx-x64/8.0.0-alpha.1.22415.5/tools/emscripten/cache")
/Users/alklig/work/net7-nightly/packs/Microsoft.NET.Runtime.WebAssembly.Sdk/7.0.0-rc.1.22422.12/Sdk/WasmApp.Native.targets(422,5): error MSB3073: The command "emcc "@/Users/alklig/work/net7-nightly/packs/Microsoft.NETCore.App.Runtime.Mono.multithread.browser-wasm/7.0.0-rc.1.22422.12/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/Users/alklig/work/net7-nightly/packs/Microsoft.NETCore.App.Runtime.Mono.multithread.browser-wasm/7.0.0-rc.1.22422.12/runtimes/browser-wasm/native/src/emcc-link.rsp" "@/Users/alklig/work/net7-playground/hithread/obj/Debug/net7.0/browser-wasm/wasm/for-build/emcc-link.rsp"" exited with code 1. [/Users/alklig/work/net7-playground/hithread/browser.csproj]

Bug 4 We removed the *-mt.a static libraries in our EMSDK pack
9. Copy *-mt.a from an upstream EMSDK install into the emsdk pack:

cp ~/work/dotnet-runtime/runtime/src/mono/wasm/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/*-mt.a ~/work/net7-nightly/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.osx-x64/8.0.0-alpha.1.22415.5/tools/emscripten/cache/sysroot/lib/wasm32-emscripten

dotnet build should now succeed.
dotnet run shoudl serve the app. Open the URL in Chrome and open dev tools and hit reload. You should see something like this in the console:

MONO_WASM: worker initializing essential C exports and APIs
MONO_WASM: worker initializing essential C exports and APIs
MONO_WASM: worker initializing essential C exports and APIs
MONO_WASM: worker initializing essential C exports and APIs
dotnet.js:5 MONO_WASM: afterLoadWasmModuleToWorker added message event handler Worker {onmessage: ƒ, onerror: ƒ}
dotnet.js:5 MONO_WASM: afterLoadWasmModuleToWorker added message event handler Worker {onmessage: ƒ, onerror: ƒ}
dotnet.js:5 MONO_WASM: afterLoadWasmModuleToWorker added message event handler Worker {onmessage: ƒ, onerror: ƒ}
dotnet.js:5 MONO_WASM: afterLoadWasmModuleToWorker added message event handler Worker {onmessage: ƒ, onerror: ƒ}
dotnet.js:5 MONO_WASM: worker initializing essential C exports and APIs
dotnet.js:5 MONO_WASM: worker initializing essential C exports and APIs
dotnet.js:5 MONO_WASM: worker initializing essential C exports and APIs
dotnet.js:5 MONO_WASM: worker initializing essential C exports and APIs
dotnet.js:3 mono_wasm_runtime_ready fe00e07a-5519-4dfe-b35a-f867dbaf2e28
dotnet.js:2013 Hello, Console!

Clean the bin and obj directories.
Change the browser.csproj like this:

<PropertyGroup>
  <WasmEnableThreads>true</WasmEnableThreads>
  <WasmEnableThreading>true</WasmEnableThreading>
  <WasmGenerateAppBundle>true</WasmGenerateAppBundle>
</PropertyGroup>

That is, replace WasmBuildNative by WasmGenerateAppBundle
12. run dotnet build again. Note that you get a single-threaded app
(dotnet.worker.js isn't in the AppBundle directory, when running
the app it doesn't print messages about initializing workers to
the console)
Bug 5 WasmGenerateAppBundle doesn't generate a multi-threaded app.

Author:	lambdageek
Assignees:	-
Labels:	`arch-wasm`, `untriaged`, `area-Build-mono`
Milestone:	-

lambdageek · 2022-08-26T15:11:45Z

@lewing @steveisok @radical @radekdoulik

lambdageek · 2022-08-26T15:17:20Z

@radekdoulik @lewing I guess our biggest problem is going to be

EMSDK nuget is missing *-mt.a libs needed for multi-threaded builds

because that runs up against the nuget size limit?

radical · 2022-08-26T15:18:23Z

@radekdoulik @lewing I guess our biggest problem is going to be

EMSDK nuget is missing *-mt.a libs needed for multi-threaded builds

because that runs up against the nuget size limit?

I can take care of the rest.

lambdageek · 2022-08-26T15:26:35Z

A bit more detail on teh last point: it seems like WasmGenerateAppBundle does work - except that the dotnet.worker.js file is missing from the multithread (and perftrace, I guess) runtime pack, so it doesn't get copied into the AppBundle

lambdageek · 2022-08-26T15:33:37Z

Maybe for net7 RC2 we can focus on WasmGenerateAppBundle+WasmEnableThreads as the "golden path" - that way we don't have to put the *-mt.a libs back in the Emscripten nuget yet. Anyone who wants to P/Invoke native code will be broken, but simple experiments with threading will work.

for net7 RTM we probably should have the *-mt.a stuff back in the Emscripten nuget, but if that's hard, maybe WasmBuildNative+WasmEnableThreads will have to wait for net8

lambdageek · 2022-09-06T15:58:51Z

Ah there's one more issue:

public class MyClass {
    [MethodImpl(MethodImplOptions.NoInlining)]
    public static string CallMeFromJS()
    {
        new Thread(Loop).Start();

        return "Hello, World!";
    }

    public static void Loop() {
        for (int i = 0; i < 5; ++i) {
            Console.WriteLine ("Thread ping!");
            Thread.Sleep (100);
        }
    }
}

results in

Which means we didn't build CoreLib for threading. Probably needs to check for MonoWasmBuildVariant when defining FeatureWasmThreads in the CoreLib proj. I'll start a PR

lambdageek · 2022-09-06T20:40:57Z

I'm not sure how to fix

dotnet new wasmbrowser does not create runtimeconfig.template.json required for dotnet run

As far as I can tell the file is in the template directory https://github.com/dotnet/runtime/tree/main/src/mono/wasm/templates/templates/browser

Note also that README.md didn't get installed either. Do we need to do something to .template.config\template.json ? I think the default sources.include should have copied us. And neither *.md or runtimeconfig.template.json is in sources.exclude by default.

lambdageek · 2022-09-06T20:47:48Z

@sayedihashimi Do you know why dotnet new wasmbrowser wouldn't install runtimeconfig.template.json from our template https://github.com/dotnet/runtime/tree/main/src/mono/wasm/templates/templates/browser by default? Should we be using sources.copyOnly ?

sayedihashimi · 2022-09-06T21:04:22Z

@lambdageek do you have a nuget package that I can try out? Or is there some specific steps that I should follow to try it out?

sayedihashimi · 2022-09-06T21:25:53Z

@lambdageek I cloned the repo and ran dotnet pack in the runtime\src\mono\wasm\templates folder to create the nuget package. I used dotnet new --install to install and try out the wasmbrowser template. It appears to be dropping that file, see the gif at the end of this. You may be running into a caching issue. There are two levels of caches when working with NuGet packages, the NuGet cache and the Template Engine cache. What I would recommend is to change the version of the package, create a new package and then try it. If it works that confirms there is a caching issue. From there we can engage the Template Engine team. They have made some updates in .net 7 to decrease the caching issues.

lambdageek · 2022-09-06T23:10:02Z

@lambdageek I cloned the repo and ran dotnet pack in the runtime\src\mono\wasm\templates folder to create the nuget package. I used dotnet new --install to install and try out the wasmbrowser template. It appears to be dropping that file, see the gif at the end of this. You may be running into a caching issue. There are two levels of caches when working with NuGet packages, the NuGet cache and the Template Engine cache. What I would recommend is to change the version of the package, create a new package and then try it. If it works that confirms there is a caching issue. From there we can engage the Template Engine team. They have made some updates in .net 7 to decrease the caching issues.

Thanks @sayedihashimi, I'll try your simplified repro steps. It's entirely possible that it's some old cache on my machine. (Our existing repro steps involve a local .net sdk install followed by workload installation from artifacts/, so it may be that my local install has some older bits in it)

sayedihashimi · 2022-09-06T23:33:03Z

@lambdageek after you give it a try, let me know if you have any more questions.

lambdageek · 2022-09-07T01:03:05Z

@lambdageek after you give it a try, let me know if you have any more questions.

I'm all set. it works on another machine. I think I had an older build

lambdageek · 2022-09-09T20:29:25Z

Looks like we will probably get AOT in RC2, also 🎉

lewing · 2022-09-14T14:30:57Z

are we done here?

lambdageek · 2022-09-14T14:47:42Z

We're done. Using an RC2 installer from https://github.com/dotnet/installer, we now have RC2 runtime packs where -p:WasmEnableThreads=true works. AOT (-p:WasmEnableThreads=true -p:RunAOTCompilation=true) works too.

$ dotnet workload install wasm-experimental
...
Installing pack Microsoft.NET.Runtime.Emscripten.3.1.12.Cache.osx-x64 version 7.0.0-rc.2.22459.3...
...
Installing pack Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.browser-wasm version 7.0.0-rc.2.22463.21...

dotnet-issue-labeler bot added the area-Build-mono label Aug 26, 2022

ghost added the untriaged New issue has not been triaged by the area owner label Aug 26, 2022

lambdageek added the arch-wasm WebAssembly architecture label Aug 26, 2022

lambdageek added this to the 7.0.0 milestone Aug 26, 2022

lambdageek removed the untriaged New issue has not been triaged by the area owner label Aug 26, 2022

radical self-assigned this Aug 26, 2022

radical removed their assignment Sep 6, 2022

lambdageek self-assigned this Sep 6, 2022

lambdageek mentioned this issue Sep 6, 2022

[wasm-mt] Fix pack/build issues in threaded builds #75162

Merged

lambdageek mentioned this issue Sep 6, 2022

[release/7.0][wasm-mt] Fix pack/build issues in threaded builds #75171

Merged

lambdageek mentioned this issue Sep 8, 2022

[wasm-mt] Emscripten SDK runtime pack is missing multi-threaded static libraries #75263

Closed

lambdageek closed this as completed Sep 14, 2022

ghost locked as resolved and limited conversation to collaborators Oct 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wasm-mt] Issues with enabling threads in .NET 7 RC 1 #74654

[wasm-mt] Issues with enabling threads in .NET 7 RC 1 #74654

lambdageek commented Aug 26, 2022 •

edited

Loading

ghost commented Aug 26, 2022

Detailed reproduction steps

lambdageek commented Aug 26, 2022 •

edited

Loading

lambdageek commented Aug 26, 2022

radical commented Aug 26, 2022

lambdageek commented Aug 26, 2022

lambdageek commented Aug 26, 2022 •

edited

Loading

lambdageek commented Sep 6, 2022

lambdageek commented Sep 6, 2022

lambdageek commented Sep 6, 2022

sayedihashimi commented Sep 6, 2022

sayedihashimi commented Sep 6, 2022

lambdageek commented Sep 6, 2022

sayedihashimi commented Sep 6, 2022

lambdageek commented Sep 7, 2022

lambdageek commented Sep 9, 2022

lewing commented Sep 14, 2022

lambdageek commented Sep 14, 2022

[wasm-mt] Issues with enabling threads in .NET 7 RC 1 #74654

[wasm-mt] Issues with enabling threads in .NET 7 RC 1 #74654

Comments

lambdageek commented Aug 26, 2022 • edited Loading

Detailed reproduction steps

ghost commented Aug 26, 2022

Detailed reproduction steps

lambdageek commented Aug 26, 2022 • edited Loading

lambdageek commented Aug 26, 2022

radical commented Aug 26, 2022

lambdageek commented Aug 26, 2022

lambdageek commented Aug 26, 2022 • edited Loading

lambdageek commented Sep 6, 2022

lambdageek commented Sep 6, 2022

lambdageek commented Sep 6, 2022

sayedihashimi commented Sep 6, 2022

sayedihashimi commented Sep 6, 2022

lambdageek commented Sep 6, 2022

sayedihashimi commented Sep 6, 2022

lambdageek commented Sep 7, 2022

lambdageek commented Sep 9, 2022

lewing commented Sep 14, 2022

lambdageek commented Sep 14, 2022

lambdageek commented Aug 26, 2022 •

edited

Loading

lambdageek commented Aug 26, 2022 •

edited

Loading

lambdageek commented Aug 26, 2022 •

edited

Loading