nim compiler built directly from bin/nim_csources is buggy

[timotheecour]

• nim compiler built directly from bin/nim_csources (ie, 1st order bootstrap) is buggy
• nim compiler built from nim compiler built from bin/nim_csources (ie, 2nd order bootstrap) is ok

Example

the --skipUserCfg --skipParentCfg are optional, just for isolation.

git clone https://github.com/nim-lang/Nim
cd Nim
sh build_all.sh # or whatever's needed to build `bin/nim_csources` from csources
bin/nim_csources c -o:bin/nim_temp4 -d:release -f --skipUserCfg --skipParentCfg compiler/nim.nim
bin/nim_temp4 c -r --skipUserCfg --skipParentCfg tests/actiontable/tactiontable.nim

Current Output

action 3 arg
/Users/timothee/git_clone/nim/Nim_devel/tests/actiontable/tactiontable.nim(37) tactiontable
/Users/timothee/git_clone/nim/Nim_devel/lib/pure/collections/tables.nim(264) []
Error: unhandled exception: key not found: C [KeyError]

Expected Output

action 3 arg
action 3 arg

Additional Information

git rev-parse HEAD
5848f00

directly using bin/nim_csources works:

bin/nim_csources c -r --skipUserCfg --skipParentCfg tests/actiontable/tactiontable.nim

2nd order bootstrapping works (bin/nim_csources => bin/nim_temp4 => bin/nim_temp5)

bin/nim_temp4 c -o:bin/nim_temp5 -d:release -f --skipUserCfg --skipParentCfg compiler/nim.nim
bin/nim_temp5 c -r --skipUserCfg --skipParentCfg tests/actiontable/tactiontable.nim

workaround

to build from a recent and stable nim release, use smthg like:
export nim_stable_X=$HOME/.choosenim/toolchains/nim-1.0.4/bin/nim
and use $nim_stable_X instead of bin/nim_csources

[Araq]

Well csources are frozen and are known to evaluate hash differently, it's later on built into the compiler. But the tarballs ship with C sources that are up to date, I hope.

[timotheecour]

maybe we could unfreeze csources and keep updating them as needed with new nim releases as it's been the case before, so that nim can bootstrap from csources in 1 step instead of (right now 2, later maybe more than 2) steps and so that compiler code can keep using recent nim features to benefit from those without resorting to too many conditionals when nim_is_old

at very least this should be documented (or even enforced) as it is a very subtle very bug-prone gotcha of the worst kind: silent subtle change in behavior that only triggers (without necessarily crashing, just different results) in some tests, so might fly under the radar in user code until it bugs in production code, in what user thought was a freshly built nim compiler

lots of ppl install nim via other ways than tartballs also

[Araq]

Well we cannot support an arbitrary set of build instructions that people think "will work", you wrote build_all.sh yourself iirc and it does do koch boot -d:release in the process. For good reasons as we noticed.

and so that compiler code can keep using recent nim features to benefit from those without resorting to too many conditionals when nim_is_old

We have a plan to solve that, a "csources2" repo.

[timotheecour]

ok, replacing bin/nim_csources by for example $HOME/.choosenim/toolchains/nim-1.0.4/bin/nim (installed by choosenim) works around the problem in this issue to have a recent-yet-stable base to rebuild development versions of nim (without bootstrapping from scratch each time which takes more time, or simply relying on nim which could point to unstable nim depending on user config)

We have a plan to solve that, a "csources2" repo.

curious what's the difference with csources?

[Araq]

curious what's the difference with csources?

There is no difference per se, the advantage is that every CI script with git clone --depth 1 https://github.com/nim-lang/csources.git continues to work and to build Nim version 1. csources are frozen for this reason. But we're not stuck with the requirement that the old csources.git must be able to build Nim version 2.

[timotheecour]

git clone --branch v0.20.0 --depth 1 https://github.com/nim-lang/csources.git achieves what you want without having to introduce another repo csources2

on top of that, implementing #11457 would make it all automatic so the tag (or hash) would be tied to a nim commit

[Araq]

It doesn't matter what "would achieve" the same when plenty of scripts do exactly git clone --depth 1 https://github.com/nim-lang/csources.git and we intend to not break them.

[ringabout]

https://github.com/nim-lang/csources_v2