Enable building shortfin's Rust dependencies on Windows

[marbre]

When using ilammy/msvc-dev-cmd to set up MSVC and dtolnay/rust-toolchain to set up Rust, building with SHORTFIN_ENABLE_TOKENIZERS set to ON fails on Windows with:

error: linking with `link.exe` failed: exit code: 1
  |
  = note: "C:\\Program Files\\Git\\usr\\bin\\link.exe" "/NOLOGO" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcx6IhZz\\symbols.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\crossbeam-utils-476df0f36b4e4560\\build_script_build-476df0f36b4e4560.build_script_build.81593264f59a8625-cgu.0.rcgu.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\crossbeam-utils-476df0f36b4e4560\\build_script_build-476df0f36b4e4560.build_script_build.81593264f59a8625-cgu.1.rcgu.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\crossbeam-utils-476df0f36b4e4560\\build_script_build-476df0f36b4e4560.build_script_build.81593264f59a8625-cgu.2.rcgu.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\crossbeam-utils-476df0f36b4e4560\\build_script_build-476df0f36b4e4560.30tdflta5f29rs5gf55oknqee.rcgu.o" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd-2df1f22abef96888.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libpanic_unwind-7fa781213a0698f8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libwindows_targets-2440cb72ce7deb9b.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_demangle-f04b9120076f20fa.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd_detect-b521ee511095af2f.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libhashbrown-124aa6c4c6ef4b4c.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_alloc-c86a42f7194744c8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libunwind-a4160695964[735](https://github.com/nod-ai/shark-ai/actions/runs/12051210540/job/33601745584?pr=619#step:10:736)08.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcfg_if-e246a9218bd1ed0e.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liballoc-8f9b5fcbcd27c22e.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_core-65178e86c6c71ba8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcore-fbeb171b69c59b37.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcompiler_builtins-e3a3e7896142045d.rlib" "kernel32.lib" "kernel32.lib" "advapi32.lib" "ntdll.lib" "userenv.lib" "ws2_32.lib" "dbghelp.lib" "/defaultlib:msvcrt" "/NXCOMPAT" "/OUT:D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\crossbeam-utils-476df0f36b4e4560\\build_script_build-476df0f36b4e4560.exe" "/OPT:REF,NOICF" "/DEBUG" "/PDBALTPATH:%_PDB%" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\intrinsic.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libcore.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libstd.natvis"
  = note: /usr/bin/link: extra operand 'D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\crossbeam-utils-476df0f36b4e4560\\build_script_build-476df0f36b4e4560.build_script_build.81593264f59a8625-cgu.0.rcgu.o'
          Try '/usr/bin/link --help' for more information.
          

error: linking with `link.exe` failed: exit code: 1
  |
  = note: "C:\\Program Files\\Git\\usr\\bin\\link.exe" "/NOLOGO" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustc9oR3ZA\\symbols.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\proc-macro2-aa639117d62021a0\\build_script_build-aa639117d62021a0.build_script_build.2e37346e7cb754b8-cgu.0.rcgu.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\proc-macro2-aa639117d62021a0\\build_script_build-aa639117d62021a0.build_script_build.2e37346e7cb754b8-cgu.1.rcgu.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\proc-macro2-aa639117d62021a0\\build_script_build-aa639117d62021a0.eo77g25asojiq7m7449m2h1zo.rcgu.o" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd-2df1f22abef96888.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libpanic_unwind-7fa781213a0698f8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libwindows_targets-2440cb72ce7deb9b.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_demangle-f04b9120076f20fa.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd_detect-b521ee511095af2f.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libhashbrown-124aa6c4c6ef4b4c.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_alloc-c86a42f7194744c8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libunwind-a416069596473508.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcfg_if-e246a9218bd1ed0e.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liballoc-8f9b5fcbcd27c22e.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_core-65178e86c6c71ba8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcore-fbeb171b69c59b37.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcompiler_builtins-e3a3e7896142045d.rlib" "kernel32.lib" "kernel32.lib" "advapi32.lib" "ntdll.lib" "userenv.lib" "ws2_32.lib" "dbghelp.lib" "/defaultlib:msvcrt" "/NXCOMPAT" "/OUT:D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\proc-macro2-aa639117d62021a0\\build_script_build-aa639117d62021a0.exe" "/OPT:REF,NOICF" "/DEBUG" "/PDBALTPATH:%_PDB%" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\intrinsic.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libcore.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libstd.natvis"
  = note: /usr/bin/link: extra operand 'D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\proc-macro2-aa639117d62021a0\\build_script_build-aa639117d62021a0.build_script_build.2e37346e7cb754b8-cgu.0.rcgu.o'
          Try '/usr/bin/link --help' for more information.
          

note: `link.exe` returned an unexpected error

note: the Visual Studio build tools may need to be repaired using the Visual Studio installer

note: or a necessary component may be missing from the "C++ build tools" workload

   Compiling shlex v1.3.0
   Compiling serde v1.0.215
error: could not compile `proc-macro2` (build script) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
error: could not compile `crossbeam-utils` (build script) due to 1 previous error
error: linking with `link.exe` failed: exit code: 1
  |
  = note: "C:\\Program Files\\Git\\usr\\bin\\link.exe" "/NOLOGO" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcKHwh9K\\symbols.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\serde-9df6f3f78162638f\\build_script_build-9df6f3f78162638f.build_script_build.678f91059671b4c4-cgu.0.rcgu.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\serde-9df6f3f78162638f\\build_script_build-9df6f3f78162638f.build_script_build.678f91059671b4c4-cgu.1.rcgu.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\serde-9df6f3f78162638f\\build_script_build-9df6f3f78162638f.bfk8wi8a449l6t5jxbcehj784.rcgu.o" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd-2df1f22abef96888.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libpanic_unwind-7fa781213a0698f8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libwindows_targets-2440cb72ce7deb9b.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_demangle-f04b9120076f20fa.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd_detect-b521ee511095af2f.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libhashbrown-124aa6c4c6ef4b4c.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_alloc-c86a42f7194[744](https://github.com/nod-ai/shark-ai/actions/runs/12051210540/job/33601745584?pr=619#step:10:745)c8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libunwind-a416069596473508.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcfg_if-e246a9218bd1ed0e.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liballoc-8f9b5fcbcd27c22e.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_core-65178e86c6c71ba8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcore-fbeb171b69c59b37.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcompiler_builtins-e3a3e7896142045d.rlib" "kernel32.lib" "kernel32.lib" "advapi32.lib" "ntdll.lib" "userenv.lib" "ws2_32.lib" "dbghelp.lib" "/defaultlib:msvcrt" "/NXCOMPAT" "/OUT:D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\serde-9df6f3f78162638f\\build_script_build-9df6f3f78162638f.exe" "/OPT:REF,NOICF" "/DEBUG" "/PDBALTPATH:%_PDB%" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\intrinsic.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libcore.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libstd.natvis"
  = note: /usr/bin/link: extra operand 'D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\serde-9df6f3f78162638f\\build_script_build-9df6f3f78162638f.build_script_build.678f91059671b4c4-cgu.0.rcgu.o'
          Try '/usr/bin/link --help' for more information.
          

error: could not compile `serde` (build script) due to 1 previous error
ninja: build stopped: subcommand failed.

This is not necessarily related to using the actiondtolnay/rust-toolchain and is probably rather a problem with ilammy/msvc-dev-cmd + Rust. GitHub rusted runners come with rustup preinstalled but setting up a toolchain with rustup default stable instead of using dtolnay/rust-toolchain results in similar build failures. If no MSVC is set up before, rustup-init offers to take care and automatically install the prerequisites, see https://rust-lang.github.io/rustup/installation/windows-msvc.html, which is needs a closer look.