refs #16702 make DirSep + friends work with cross-compilation and VM

[timotheecour]

warning: I'm not necessarily endorsing this approach, but it's one of the options worth considering.

refs #16702

investigates one approach to solve cross-compilation problems involving semantics mismatch between CT and RT. Doesn't fix all issues obviously but this shows at least the most obvious issues can be fixed by simply adding a few key vmops.

import os

proc main()=
  echo "foo" / "bar"
  const s = currentSourcePath
  echo s
  echo currentSourcePath.parentDir
  echo s.parentDir
  echo currentSourcePath / "baz"

static: main()
main()

this works:

nim c -f --os:windows --skipparentcfg --skipusercfg --compileonly main

foo/bar
/Users/timothee/git_clone/nim/timn/tests/nim/all/t11799.nim
/Users/timothee/git_clone/nim/timn/tests/nim/all
/Users/timothee/git_clone/nim/timn/tests/nim/all
/Users/timothee/git_clone/nim/timn/tests/nim/all/t11799.nim/baz

this gives errors because I don't have a cross compilation toolchain for windows on osx

nim r -f --os:windows --skipparentcfg --skipusercfg main

on osx this shows:

/Users/timothee/.cache/nim/t11799_d/stdlib_io.nim.c:11:10: fatal error: 'io.h' file not found
#include <io.h>
         ^~~~~~
/Users/timothee/.cache/nim/t11799_d/stdlib_system.nim.c:13:10: fatal error: 'windows.h' file not found
#include <windows.h>
         ^~~~~~~~~~~
/Users/timothee/.cache/nim/t11799_d/stdlib_dynlib.nim.c:11:10: fatal error: 'windows.h' file not found
#include <windows.h>
         ^~~~~~~~~~~

which (ignoring errors because i don't have the windows headers on my system) shows that such an approach might work

@jangko

• could you please try this and run the generated code on windows to see if it has windows semantics at RT?
• *what's the simplest way to get a working cross compilation windows toolchain (headers, etc) working on osx?

future work

This could be further simplified by extending vmops to work in more contexts, including const, so that if a const is used in an already static context, it'd use the vmops definition if present.

[alaviss]

Would

const AutoSaveDir = "saves" / "automated" # Should be "saves/automated" on POSIX and r"saves\automated" on Windows

be compiled to the correct format of the target operating system with this PR? I don't see any tests for this.

[timotheecour]

@alaviss see test I added.

in particular, this test will pass on a posix host with --os:windows.

const dir = "foo" / "bar"
static: doAssert dir == "foo/bar" # this works and is tested in this PR's tests

let dir2 = "foo" / "bar"
doAssert dir2 == "foo\\bar" # I haven't tested this yet, but hopefully it works.

That's the most sensible semantics we can implement with current nim semantics IMO, making a clear distinction bw CT(honor host) and RT(honor target) code; if you want ability to use const's that follow target semantics, we'd need something like const {.target.} dir = expr, see #16702 (comment)

[alaviss]

This seems to be a band aid and a breaking change to me. Currently hostOS might yield windows but DirSep yields / at compile time. If you're implementing this, I would suggest having the {.host.} and {.target.} pragma implemented instead, so the behavior can be consistent between the stdlib and third party code.

We should define how the CTFE should actually work, and Nim should probably grow a build host/target distinction. I would say that this feature need an RFC, given that it is currently a breaking change.

[jangko]

could you please try this and run the generated code on windows to see if it has windows semantics at RT?

windows to windows, run on windows:
nim c -f --os:windows --skipparentcfg --skipusercfg --compileonly main
foo\bar
F:\projects\experiment\main.nim
F:\projects\experiment
F:\projects\experiment
F:\projects\experiment\main.nim\baz

windows to windows, run on windows:
nim r -f --os:windows --skipparentcfg --skipusercfg main
foo\bar
F:\projects\experiment\main.nim
F:\projects\experiment
F:\projects\experiment
F:\projects\experiment\main.nim\baz

linux to windows, run on linux:
nim c -f --os:windows --skipparentcfg --skipusercfg --compileonly main
foo/bar
/home/jangko/experiment/main.nim
/home/jangko/experiment
/home/jangko/experiment
/home/jangko/experiment/main.nim/baz

linux to windows, run on linux:
nim r -f --os:windows --skipparentcfg --skipusercfg main
foo\bar
/home/jangko/experiment/main.nim
\home\jangko\experiment
\home\jangko\experiment
\home\jangko\experiment\main.nim\baz

linux to windows, run on windows:
nim r -f --os:windows --skipparentcfg --skipusercfg main
foo\bar
/home/jangko/experiment/main.nim
\home\jangko\experiment
\home\jangko\experiment
\home\jangko\experiment\main.nim\baz

*what's the simplest way to get a working cross compilation windows toolchain (headers, etc) working on osx?

I have no idea, probably install mingw?

[Araq]

I would suggest having the {.host.} and {.target.} pragma implemented instead, so the behavior can be consistent between the stdlib and third party code.

I'm heavily against .host and .target pragmas. Whenever somebody uses static he needs to be aware of the gotchas that cross compiles implies. We need to ensure to use these pragmas consistently ourselves. We need to setup CIs that test cross compilation setups. It's an enourmous task for the benign problem that sometimes \\ is used instead of /. Btw Windows can deal fine with / anyway. Plus I don't see any benefits that cross compiling from Linux to Windows actually has, you setup a CI with Windows support, bam, done.

[timotheecour]

@Araq As I wrote in top post, I'm not endorsing this, but this was something worth exploring, I now don't mind if we close this WIP PR.

I'm now leaning instead towards timotheecour#598 vmhook instead, which amounts to user defined vmops (it has nothing to do with the .host and .target pragmas as previously discussed, and has none of its disadvantages).

vmhook offers a general solution to:

• cross compilation (not just windows <=> posix, but also issues related to 64bit host <=> 32bit target, or 64bit host <=> js target where int.sizeof differs, etc
• overcoming VM limitations (since we'd run native code at CT for a vmhook)
• performance in VM for hotspots

note: I've actually implemented this in a private branch (with auto-boxing from a non-generic proc declaration to a PNode vm wrapper and using --compiletimeFFI), and works pretty well.

[Araq]

has none of its disadvantages

well ... until you use vmhook to replicate the .host and .target distinction. ;-)