CI: Investigate why appveyor chokes in sass-specs test?

[am11]

sassc-win passes all the sass-specs tests and exits gracefully. It uses ruby runtime via sassc.

While node-sass builds libsass successfully on appveyor but the tests are interrupted for some unknown reasons.

Going by https://github.com/gruntjs/grunt/blob/master/appveyor.yml, I believe we can take some inferences from there and fix it for node-sass.

/cc @kevva, @deepak1556, @nschonni

[am11]

@FeodorFitsner, can you please provide some pointers here? It passes all tests locally on both Winx64 with Node-x86 and Win64 with Node-x64 (both cmd and ps), but fail to continue tests on Appveyor. We have 600+ tests (mostly coming from sass-spec.

Thanks in anticipation.

[FeodorFitsner]

Hey guys, in appveyor.yml try replacing test section with (remove ps: in the last line):

test_script:
  - node --version
  - npm --version
  - npm test

[am11]

@FeodorFitsner, Thanks for the tip!

I updated it, but unfortunately, the result is same (with or without cmd: or ps:). It terminates npm install without reporting any error. Also, it terminates at random tests. Locally it passes all the tests on Windows x64/x86.

See https://ci.appveyor.com/project/sass/node-sass/build/168.

It seems like, there is some kind of a time limit which Appveyor spare the job (such as npm test) to breath. We have long list of tests, hence the failure..

Please take a look into this issue.

[FeodorFitsner]

After googling I understood npm ERR! Test failed. See above for more details. could mean everything :)

Try running test command without npm, by directly calling node_modules/.bin/mocha test

[am11]

I made the change: am11@dde26eb and ran the test, but no luck: https://ci.appveyor.com/project/am11/node-sass/build/68/job/r1egub0279aevo6x

It seems like a timeout issue. Locally all tests run fine.

You can also give it a try. In PowerShell:

git clone https://github.com/sass/node-sass/ --recursive
cd node-sass
git submodule update --recursive --init
npm install

# at this point kill node.exe process from task manager
# see https://github.com/joyent/node/issues/7164#issuecomment-61814857

node scripts/build -f # rebuild the binaries to mimic the behavior of CI
npm test

[FeodorFitsner]

Timeout in some of the tests?

[am11]

I mean somehow the process is terminated by OS? Because it is terminating at random tests.

[FeodorFitsner]

Interesting, I'll give it a try by interactively logging in to build worker.

[am11]

👍

[kevva]

@FeodorFitsner, any news on this?

[FeodorFitsner]

Not yet, sorry.

[am11]

@FeodorFitsner, this might be a related case: sassc-win, which runs the same test suite as node-sass.

There is no issue with the project build. But, if you look at the history on Appveyor, 1.0.31, 1.0.32 and 1.0.33 didn't get any change made in the actual code, but only the README was updated, yet in 1.0.33 it passes Winx64 test and failing the same the rest of the builds. The error occurs when running the tests Command bin/sassc did not complete.

It seems more like timeout enforced on processes which breaks our tests.

[FeodorFitsner]

Hi guys!

It looks very similar to your issue (mocha with npm test): http://help.appveyor.com/discussions/problems/1091-npm-test-always-fails

[am11]

Hi @FeodorFitsner,

Thanks for the update! I am working on a separate branch, will test it out lately.

@stevenvachon, incidentally; can we use --no-exit with npm test which uses mocha?

[stevenvachon]

@am11 --no-exit is a mocha option, not an npm option and while I'm not sure if npm passes any arguments onto its scripts, I doubt it.

This is probably what you're looking for:

"scripts": {
    "test": "mocha test/ --no-exit"
}

npm test

[am11]

@steveluscher, thanks! I added --no-exit switch and now appveyor is not declaring the result (keep ticking indefinitely; the tests finished in 6 mins and 27 secs): https://ci.appveyor.com/project/sass/node-sass/build/250/job/0r75512tu1rbb9hf

Also, the two errors it reported, appeared quite earlier (before all tests are completed) at line 538.

[stevenvachon]

Yeah, I'd figured there would be side effects as it is not a real solution; more of a hopeful workaround. You could try using mocha's --bail option.

[am11]

Well, it failed with --bail as well as --bail --no-exit.

[am11]

This issue is fixed in mocha mochajs/mocha#1440 and the fix will be available in mocha's vNext.

This was actually a node.js issue, which is also fixed there in 0.11.x branch: nodejs/node-v0.x-archive#7196, but probably not be back-ported to v0.10.

This issue can be closed as concluded.