Remove compiler and use nimscript with nim e instead

[genotrance]

• Removed compiler dependency and use nim e instead
• Added nimscriptwrapper.nim to replace nimscriptsupport.nim (still in source control)
• Added OSX to CI
• Added devel & stable to CI
• Tests fixed to pass on Windows

[genotrance]

cc @Araq @dom96

[Araq]

LGTM.

[dom96]

Nice job overall. I've got some questions and some changes I'd like you to make though.

[genotrance]

I have made most of the changes as suggested. If there aren't any major concerns, I'd prefer merging this as is so that it can undergo some general testing by the community before 0.20 comes out.

[dom96]

I think this is much better. There are still some things I would change but we can do that later.

There is one thing that I need you to fix though, please do so and then we can merge this.

[dom96]

@genotrance as far as I'm concerned you need to revert this because it will break things, please do so (or explain why it must be this way).

[genotrance]

@genotrance as far as I'm concerned you need to revert this because it will break things, please do so (or explain why it must be this way).

See this job log for starters. Cache is restored but install is still rerun. Travis docs don't say that restoring cache will skip the install step, neither in the install nor cache documentation.

Even in nightlies we have to check for existence of nim binary to leverage caching.

Meanwhile, I tried caching the entire ~/.choosenim dir but ran into random Travis issues so backed that out for now. It can be done later as we get Windows CI working, once this PR is merged.

As for ~/.nimble, no doubt choosenim lives there but the redownload and nimble setup takes 1.2 seconds. It is likely it is already cached in the Travis http proxy. This way, we also get newer versions when they get released. Also, most tests are run against a custom nimbleDir but not generate so if for whatever reason that test fails and an older generate binary is in ~/.nimble/bin, the test might still pass.

Overall, caching isn't material to this PR. I will add Windows CI and ~/.choosenim caching separately. Please push this through and I can proceed onto other tasks.

[dom96]

Alright. I forgot that install was run anyway in Travis.

So I've checked out your branch and tested it. It looks like the caching mechanism doesn't work, it never gets invalidated. I tried to modify the Nimble file version from 0.9.0 to 0.10.0 and nimble dump keeps returning 0.9.0.

Then I removed the cache in /tmp/nimblecache and got this:

Seems there is still some work left to do :/ (That error message needs improvement too)

[genotrance]

So I've checked out your branch and tested it. It looks like the caching mechanism doesn't work, it never gets invalidated. I tried to modify the Nimble file version from 0.9.0 to 0.10.0 and nimble dump keeps returning 0.9.0.

Looks like I broke cache checking when I re-architected this. I've put in a fix and added a test case as well.

Then I removed the cache in /tmp/nimblecache and got this:

I wasn't able to reproduce this.

[dom96]

Merged manually.

[andreaferretti]

Apparently this PR broke all my packages tests :-D

All my libraries define the tasks nimble tests (a convention suggested in the nimscript docs ), and nimble test as an alias for that, since I usually forget the final s. For instance, in patty nimble file one can find

task tests, "run tests":
  --hints: off
  --linedir: on
  --stacktrace: on
  --linetrace: on
  --debuginfo
  --path: "."
  --run
  setCommand "c", "test.nim"

task test, "run tests":
  setCommand "tests"

After this PR, if I run nimble test on patty I get

andrea$ nimble test
  Executing task test in /Users/andrea/esperimenti/patty/patty.nimble
tests        run tests
test        run tests
   Warning: No tests found!

while running nimble tests gives an error

andrea$ nimble tests
  Executing task tests in /Users/andrea/esperimenti/patty/patty.nimble
../../../lib/pure/parsejson.nim(522) raiseParseErr
Error: unhandled exception: input(1, 153) Error: ] expected [JsonParsingError]

The same happens for all my other libraries. :-(

[genotrance]

@andreaferretti - I've made a fix in #642.

[andreaferretti]

@genotrance Thank you, now everything works again :-)

[andreaferretti]

@genotrance Actually, I still find a strange issue in Rosencrantz. Since the tests require a server and a client, I have a small shell script to launch both and stop the server after the tests are done:

set -e
set -u

nimble server
tests/rosencrantz &
PID="$!"
nimble client
kill "$PID"

It seems that the tests pass when launched manually. When I use the script, the tests do not pass (the returned value is not 0) and I get this strange line at the end:

(1, 22) Error: undeclared identifier: 'Users'
     Error: {"success": true, "command": "e", "flags": {"forceBuild": [""]}, "retVal": true}

which is even weirder cosindering that neither the server nor the client code contains the Users identifier anywhere

[andreaferretti]

@genotrance I get random issue while testing again :-(

For instance, on patty I have the following definition

# patty.nimble

task tests, "run tests":
  --hints: off
  --linedir: on
  --stacktrace: on
  --linetrace: on
  --debuginfo
  --path: "."
  --run
  setCommand "c", "test.nim"

task test, "run tests":
setCommand "tests"

If I run nimble tests everything works, but if I run nimble test (no s), which should be an alias, I get

djerzinski:patty andrea$ nimble test
  Executing task test in /Users/andrea/esperimenti/patty/patty.nimble
tests        run tests
test        run tests
   Warning: No tests found!

If I invert the definition of nimble test and nimble tests, both commands fail in the same way. Same thing happens with alea.

This is starting to get brittle. I have automated tests for my libraries, but I just cannot trust them anymore :-(

[dom96]

I noticed something similar for one of my tasks called server, if I renamed it to fooserver it worked, but for some reason server didn't (other tasks in the same file called things like client also worked).

We need to fix this and release a hotfix Nimble before Nim 0.20.0 is out

[genotrance]

I'll look at these issues asap.

[genotrance]

I've figured out the bug - it is fragile since I'm avoiding projectPath() recently added in Nim devel. I did that so this new nimble could support v0.19.x as well. I'll make it more robust tomorrow.

[dom96]

What are you using instead?

[genotrance]

I've created a fix - can you please verify on your end that it fixes your issues?

#659

[andreaferretti]

Sure, but I'm away from computers now, I can check on Monday

[andreaferretti]

@genotrance Works for me, thank you!