Coordinate porting to Windows

[straight-shoota]

UPDATE 2023-04-24:

All the main platform features for Windows are finished! 🚀

There are still some smaller stories pending. Project board: https://github.com/orgs/crystal-lang/projects/11/views/5

You can support the ongoing development financially: Windows suppport project on Open Collective.

UPDATE 2021-11-18:

By now, the compiler works pretty well on windows and most of the standard library has been ported as well. Besides a couple other smaller missing pieces, the major gap is in concurrency-related features held back by the missing event loop implementation.

---

With #5339 being merged, the first step of bringing Crystal to Windows has been accomplished!
With current master branch it is possible to cross-compile and run simple programs (alá Hello, World!) on native Windows.

Obviously we're still far away from porting the compiler or entire standard library to windows. This issue is supposed to coordinate and keep track of ongoing efforts to add support for more and more of the crystal standard library.

TODO

The primary focus should be on the core library (somewhat odered by priority):

• File descriptors "hello world" windows port #5339
• GC "hello world" windows port #5339
• Errno "hello world" windows port #5339
• Basic Exceptions "hello world" windows port #5339
• Regex "hello world" windows port #5339
• Random Implement Crystal::System::Random for win32 #5539
• Time Implement Crystal::System::Time for win32 #5422
• Raising Support exception handling on windows #6419
• File Implement File and Dir for win32 #5623
• Dir Implement File and Dir for win32 #5623
• ENV Extract platform-specifics from ENV to Crystal::System::Env and implement for win32 #6333
• Fibers Implement fibers for win32 #7995
• Event loop Windows: Event loop based on IOCP #12149
• Signal Port signals to Windows #7339
• Mutex Enable Mutex on Windows #12213
• Sockets Porting Socket to win32 #10696
• Process
• Backtraces Support call stacks on Windows #11461
• Iconv (Support custom encodings on Windows through GNU libiconv #11480)
• System Implement System for Win32 #6972
• Threads (Implement multithreading primitives on Windows #11647)
• Compiler Bootstrapping compiler on Windows (tracking bug) #9032

And of course a CI infrastructure once specs are running on windows.

Proceedings

The general course of action is to extract platform-specific implementations and then add the implementation for win32 platform.

A lot of work has already been done in #3582 but it needs to be updated and chopped into smaller pieces of changes.

The first goal should probably be to get specs running on windows to be able to verify the windows implementations. Essential for this are as far as I can see file, dir, exceptions (backtrace!), process and signal.

If you want to help with that effort, or want to "take" a piece of that work to work on, please comment below.

Useful References

• Wiki page Porting to Windows
• Support Windows as a first class platform (note: not WSL or emulation) #26 Support Windows as a first class platform (note: not WSL or emulation)
• [WIP] Windows #3582 [WIP] Windows by @lbguilherme
• "hello world" windows port #5339 "hello world" windows port by @RX14
• Win: define Win32 API functions #4832 Win: define Win32 API functions by @txe
• Remove c/winapi and rename some windows types #5448 Remove c/winapi and rename some windows types (some pieces from removed c/winapi can be salvaged for implementations)

Please feel free to add and edit this post as appropriate (core team).

[RX14]

@straight-shoota that's a good point. We need to figure out the basics of getting a blank spec suite working on windows (we should do this first without unwind support, just the raise==exit we have now). Then we need to create some kind of win32_spec.cr which requires only the parts of spec/std that work. I'll take on this task (tomorrow).

[RX14]

Another area of priority is looking at c168035 and working out ways of reducing that stubbing. For example: get iconv working so we can remove the stubbing in src/io/encoding.cr. The modified parts of the corelib are actually I think worth focus much more than the missing parts (for now).

[straight-shoota]

A very basic working spec shouldn't be too hard I think. I've tried it earlier to figure out what might be needed for that. It seems to be missing mostly some basic file and dir methods. After disabling all feature options I even got a simple spec to compile, allthough it resulted a LLVM error at codegen.

[s5bug]

I'll take Fibers, I'll be able to start work on them after January 2nd.

[RX14]

@straight-shoota looks like my spec PR has turned into a Dir PR first.

[RX14]

Also of importance: deciding what to do about windows's UTF-16 use. There should definitely be a really quick and easy way of going between String -> UTF-16 WCHAR* and the reverse. We should also have a policy of no A functions, only W functions even this early in the port.

This is an important issue: we shouldn't rely on iconv for this (I presume it's slow for small strings, and we don't have it in windows anyway). We need to write UTF-16 infrastructure for all platforms first.

[RX14]

Good news: spec compiles and works on my fork.

Bad news: it only works sometimes.

From this spec failure, looks like we have a codegen or ABI bug somewhere:

[RX14]

Here's my branch porting spec: https://github.com/RX14/crystal/commits/feature/windows-spec

I'll turn "Remove c/winbase" and "Add Crystal::System::Dir" into PRs.

[RX14]

Another idea before I forget it:

Consolidate all stubbing out into one huge windows_stubs.cr file so that we can keep track of them all.

[straight-shoota]

You mean to remove winapi.cr, not winbase.
And this commit should also rename DWord.

[RX14]

@straight-shoota yep, thanks, commit message is wrong. Shouldn't be working at 1am...

[RX14]

Made those PRs: #5447 and #5448. I hope you don't mind me editing the checklist in your issue @straight-shoota.

[straight-shoota]

Looks great!

I've deliberatly added a note at the end of the OP for that purpose ;)

[felipetesc]

Great news! Thank you for this.

[drhuffman12]

In https://github.com/crystal-lang/crystal/wiki/Porting-to-Windows, should:

Ongoing efforts will be tracked and coordinated in #

... be:

Ongoing efforts will be tracked and coordinated in #5430?

[straight-shoota]

@drhuffman12 thanks, fixed!

[drhuffman12]

@straight-shoota , yw

[s5bug]

Due to time constraints and last minute events I won't be able to work on Fibers as expected :(. If someone else wants to claim them, that would be amazing.

[RX14]

If you can get openssh server working on WSL (hint: you need to edit /etc/ssh/sshd_conf to set UsePrivilegeSeparation no and change Port 2222), you can use the following script to make the windows testing process seamless:

#!/bin/sh
set -euo pipefail

windows=192.168.xxx.xxx

bin/crystal build --progress --cross-compile --target x86_64-unknown-windows-msvc -o .build/cross "${@}"
scp -P 2222 .build/cross.o $windows:/mnt/c/crystal/cross.o

cat <<EOF | ssh -p 2222 $windows bash
export PATH="$PATH:/mnt/c/Windows/System32"
cmd.exe /S /c "C:\Program^ Files^ ^(x86^)\Microsoft^ Visual^ Studio\2017\BuildTools\VC\Auxiliary\Build\vcvarsall.bat x64 && cd \crystal && cl cross.o /Fecross pcre.lib gc.lib libcmt.lib"

cd /mnt/c/crystal

echo
./cross.exe
EOF

This script assumes you have gc.lib and pcre.lib in C:\crystal and that you're using the visual studio 2017 build tools. MSVC 2015 can surely just change the path to vcvarsall.bat.

[RX14]

A script to automatically generate a compilable manifest of the entire spec suite, with broken tests commented out with a vague reason:

cd spec
for spec in ./std/**/*_spec.cr; do
  (cd .. && crystal-windows spec/$spec) >/dev/null 2>/dev/null && echo "require \"$spec\"" ||
    (../bin/crystal build --no-codegen --target x86_64--windows-msvc $spec >/dev/null 2>/dev/null && echo "# require \"$spec\" (failed to run)" || echo "# require \"$spec\" (failed to compile)")
done

crystal-windows is the script from above.

[straight-shoota]

The script posted by @RX14 is intended to be run on an external system which connects to WSL through SSH to upload the object file and run the linker through Windows interoperability feature.
If you want to run the compiler locally on WSL, you don't need to go through SSH.
I have put together a crystal-windows wrapper for this: https://gist.github.com/straight-shoota/e4393c9e3f0febb7dc290470f92f5c0d

[txe]

As I see, there is no dir.cr in crystal/system/win32, so I'm going to make it.
Is there any idea what to do with UTF-16?

[txe]

About UTF-16: I suppose we need just a couple of functions to encode\decode utf-16 without any len, find and others functions and can keep utf-16 string as array of wchar.
FYI, Go has this code for that and Nim this

[asterite]

@txe Please coordinate with @RX14 , as far as I know he was porting File to Windows (not sure about Dir). For UTF16 we now have this: #5541 . But check the PR's comments, we might need to change it a bit to include a zero at the end when encoding to Slice(UInt16).

[RX14]

@txe Fantastic to see you back!

I'm working on File and Dir right now, they're both mostly complete but the new code isn't in my branch (uncomitted on my PC). UTF-16 also has an added tweak in my branch.

I have a lot of work not in master, perhaps committing my branch would help you have more of a base to work off of?

Namely, working in my branch but not master: File, Dir, Spec. I also have a spec/windows_spec.cr which lists all working specs.

[mdwagner]

@neatorobito Ah, you're right. Same issue.

[neatorobito]

@mdwagner I've tested and pushed a fix for this in the scoop repo, please give it a try when you get a chance.

[mdwagner]

@neatorobito I updated to the new version in Scoop, but still have the same issue.

[straight-shoota]

@mdwagner Can you please open a dedicated issue for this? It's out of scope for this meta issue. Please add details on your system setup (like install location of VS Build Tools).

[mominshaikhdevs]

Isn't it high time this + #26 + #7339 be closed ? (since as per this comment, #7339 is completed as #12224 + #12284 being completed) and move focus to other issues?

[straight-shoota]

No, it's not.
I'm not sure if the signal story is entirely finished. We just recently merged a bunch of changes. If that's all, yes we can close #7339. But we'll need to confirm that.

That would then mean the bulk of the Windows port is completed as per the TODO list in this issue. But there are still a lot of very rough edges, so #26 is still a way to go.

[zw963]

Hi, from the TODO list, it's seem like Signal should works on windows, but it build failed on my github action platform is ms-windows 2022.

Dependencies are satisfied
Building: translater
[1/13] Parse                             
[1/13] Parse                             
[2/13] Semantic (top level)              
[2/13] Semantic (top level)              
[3/13] Semantic (new)                    
[3/13] Semantic (new)                    
[4/13] Semantic (type declarations)      
[4/13] Semantic (type declarations)      
[5/13] Semantic (abstract def check)     
[5/13] Semantic (abstract def check)     
[6/13] Semantic (restrictions augmenter) 
[6/13] Semantic (restrictions augmenter) 
[7/13] Semantic (ivars initializers)     
[7/13] Semantic (ivars initializers)     
[8/13] Semantic (cvars initializers)     
[8/13] Semantic (cvars initializers)     
[9/13] Semantic (main)                   
Error target translater failed to compile:
Showing last frame. Use --error-trace for full trace.

In lib\selenium\src\selenium\service.cr:92:20
 92 | process.signal(Signal::KILL)

                     ^-----------
Error: undefined constant Signal::KILL
Did you mean 'Signal::ILL'?

Please have a look, following is log.

https://github.com/crystal-china/translater/actions/runs/4906504148/jobs/8761004602

[HertzDevil]

They should use Process#terminate, because Signal is not portable, and MSVC never defines Signal::KILL. This is by design

[Neltherion]

9 Years passed and still no Windows support. This is why Crystal will unfortunately always be a niche language for a small minority.

[mominshaikhdevs]

9 Years passed and still no Windows support. This is why Crystal will unfortunately always be a niche language for a small minority.

All I have seen here for the last 4 years is, New "Windows Related" Issues are piling up everyday. Instead of resolving the old ones. 😂

[cloewen8]

9 Years passed and still no Windows support. This is why Crystal will unfortunately always be a niche language for a small minority.

Not surprised, checked the funding and it just was not there. I've moved on to other languuages with better marketing. The ecosystem is moving on, without Crystal.

[Blacksmoke16]

I mean have any of you actually tried it? Windows support is, and has been for a while, at the point where you can pretty much install via the provided installer on the GH release and use it as you would on any other OS.

There are still some loose ends that need to be figured out, but I don't think it's fair to say "there is no windows support."

[mjblack]

9 Years passed and still no Windows support. This is why Crystal will unfortunately always be a niche language for a small minority.

You can compile on windows, you can create windows gui applications, you can create directx applications. What do you think is still missing?

[aolko]

9 Years passed and still no Windows support. This is why Crystal will unfortunately always be a niche language for a small minority.

You can compile on windows, you can create windows gui applications, you can create directx applications. What do you think is still missing?

modern web ui libs like electron/tauri/webview? Some libs for AI crowd like Torch, diffusers, transformers and other related dependencies?

[linuxct]

9 Years passed and still no Windows support. This is why Crystal will unfortunately always be a niche language for a small minority.

You can compile on windows, you can create windows gui applications, you can create directx applications. What do you think is still missing?

The interactive CLI, actually:

PS ..\Users\linuxct> crystal.exe i
Crystal was compiled without interpreter support

[Blacksmoke16]

modern web ui libs like electron/tauri/webview? Some libs for AI crowd like Torch, diffusers, transformers and other related dependencies?

There is a difference between the language missing something and the ecosystem missing something. All of these are examples of the latter. Unless you have specific windows specific reason as to why you can't make something like what you mention, then someone just needs to do it. You're not going to get these things in the stdlib. Could always start a discussion on the forums to see if you can find some others to help creating/maintaining these kind of libs.

The interactive CLI, actually:

This is not specific to windows either. The interpreter is still experimental, on all OSs. If you actually want to use it, you'd have to build from source.

[Tamnac]

then someone just needs to do it.

They're not going to, because windows is not yet a stable, feature complete platform. Why would they? I don't blame them. The ecosystem mirrors the language. If you can't complete the final stretch and finish Windows support, neither will anyone else. That's why you can't dismiss these concerns with "everything basically works". For practical purposes "incomplete windows support" == "no windows support"

I know the team is small and can only do so much, but even then, there are regular releases, and little mentioned of finishing windows support. The team is of course free to choose what to focus on. But you can't blame others for making decisions based on the perceived focus of the team. At this point it's been years and you'd have to be a fool to think Windows is going to be anything but a second class platform for Crystal.

When I got into crystal a little while back, I thought Windows support was around the corner. Only later after looking into it did I realize it's been "around the corner" for 9 years. I've lost motivation to sink any more time into it now. I think most people are like me. I really did want to use crystal more.

[Blacksmoke16]

because windows is not yet a stable, feature complete platform.

That's why you can't dismiss these concerns with "everything basically works".

I'm just going to repeat my comment from #5430 (comment). Do you have any concrete examples of something not working on Windows? If so, please say what it is, or better yet create an issue for it! Otherwise these claims that "there is no windows support" are baseless and are not helping, and seem to only be for trolling purposes...

EDIT: I also don't want it to come across as everything is perfect on Windows, but to my knowledge there is nothing left that works on other OSs, but just not in Windows. There will be of course bugs and things to improve, but that's no different than any of the other platforms.

[tmssngr]

If nothing is left to do on Windows, then better mention that Windows is officially supported now.

[felipetesc]

I've been paying attention to this topic for a while now, and I can't let it go unnoticed without thanking you for the effort to make Crystal functional on Windows. Finally, I can use it for general projects. So, thanks.

[rdp]

Might be nice to update the description here, it was confusing to me today that it has sections saying it's "missing event loop" and "far from porting the compiler", cheers! :)

[rdp]

Might be nice to name the installer exe something different from the portable name, ex: "crystal-1.13.1-windows-x86_64-msvc-unsupported-installer.exe" just to make it even more clear...An idea. Or even name the .zip as -portable.zip too. It's a fairly common windows convention :) Cheers!

[sdogruyol]

Bump, all TODOs are complete 👍 What's next?

[Blacksmoke16]

@sdogruyol Was some discussion on this in crystal-lang/crystal-book#772 (comment).

[crysbot]

This issue has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/crystal-core-developer-looking-for-work-in-germany-europe/7389/2