nim dump is roughly 100x slower in 1.3 versus 1.2

[disruptek]

See title.

[Araq]

I'm pretty sure it's caused by our NimScript config evaluation.

[disruptek]

Confirmed. This is also the source of the extra output problem detailed in #13912.

@timotheecour

[timotheecour]

@Araq as always, the 1st thing to try is git bisect

3d2459bdc0b6d6236a2cd9209ed81c965ee411a5 is the first bad commit
commit 3d2459bdc0b6d6236a2cd9209ed81c965ee411a5
Author: Araq <rumpf_a@web.de>
Date:   Thu Apr 30 20:23:42 2020 +0200

    fixes the regression #12860 caused; hotfix

 lib/pure/os.nim          | 2 +-
 lib/system/nimscript.nim | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)
bisect run success

and indeed git bisect is correct once again:

3d2459b adds a import os dependency on nimscript which introduces the regression:

• nim dump . processes $nim/config/config.nims
• $nim/config/config.nims imports system => nimscript.nim => (since that commit) os.nim
• os.nim itself is expensive, causing a 6X slowdown

But that's not the end of the story.

Because of dom96/choosenim#193 (which I reported here #13986 (comment) as a confounding factor for another issue), $nim/config/config.nims is not copied over to $nim_toolchain_D/nim-1.2.0/config/config.nims, so for nim 1.2.0 installed via choosenim you instead get:

• nim dump . processes ... no nims file at all since processes $nim/config/config.nims doesn't exist

=> 23X difference

timings:

$nim_120_X dump --skipusercfg --hint:conf:on .  0.01s user 0.01s system 88% cpu 0.015 total
$nimb dump --skipusercfg --hint:conf:on .  0.05s user 0.01s system 97% cpu 0.062 total # right before 3d2459bdc0b6d6236a2cd9209ed81c965ee411a5
$nimb dump --skipusercfg --hint:conf:on .  0.33s user 0.02s system 99% cpu 0.352 total # right after 3d2459bdc0b6d6236a2cd9209ed81c965ee411a5

@disruptek

Confirmed. This is also the source of the extra output problem detailed in #13912.

unrelated AFAIU

proposal

• choosenim installs Nim/config/nim.cfg but not Nim/config/config.nims dom96/choosenim#193 is a bug that keeps creeping up and should be fixed, $nim/config/config.nims is meant to be read for correctness even if it currently doesn't contain much; users who know what they're doing can always pass --skipCfg to skip it
• 3d2459b should be reverted to avoid adding os dependency in nimscript
• Undefine paramCount & paramStr in nimscript.nim for *.nims #12860 should be reverted, and [superseded] fix #14142: no more clash with: import os + use of existsDir/dirExists/existsFile/fileExists/findExe in config.nims #14143 should be merged instead, it's the more general fix for import os + use of paramCount and paramStr in config.nims causes "ambiguous call' error #12835 since it also fixes import os + use of existsDir/dirExists/existsFile/fileExists/findExe in config.nims causes "ambiguous call' error #14142 etc

That will fix all above problems including #14142.

• I also have a PR that turns nimscript into an import module (not include file) and allows a nimscript module to skip auto-importing it, which makes nims code more like regular nim code without all the "baggage" like cd, etc. I just need to see how well it fits with new nimscript impl #14235.
  send pr_nimscript_module PR timotheecour/Nim#307

local imports

timotheecour#169
That's a topic for another discussion, but allowing local imports (eg at proc scope) would solve many dependency issues, including the one that causes this regression, making importing more lazy. I even have a working branch for that.

[genotrance]

dom96/choosenim#193 is a bug that keeps creeping up and should be fixed, $nim/config/config.nims is meant to be read for correctness even if it currently doesn't contain much; users who know what they're doing can always pass --skipCfg to skip it

This is not a choosenim issue which simply downloads the .zip or .xz file from nim-lang.org and extracts it. The file is missing in the archive posted on nim-lang.org.

[timotheecour]

The file is missing in the archive posted on nim-lang.org.

fixing that part in #14267