FreeBSD - tests/js/tconsole fails with different output

[euantorano]

This is a bit of a weird one. On FreeBSD, tests/js/tconsole.nim fails with the following:

Category: js
Name: tests/js/tconsole.nim JS -d:nodejs
Action: run
Result: reOutputsDiffer
-------- Expected -------
Hello, console
1 2 3
1 'hi' 1.1
--------- Given  --------
Hello, console
1 2 3
1 hi 1.1
-------------------------

Category: js
Name: tests/js/tconsole.nim JS -d:nodejs -d:release
Action: run
Result: reOutputsDiffer
-------- Expected -------
Hello, console
1 2 3
1 'hi' 1.1
--------- Given  --------
Hello, console
1 2 3
1 hi 1.1
-------------------------

Node is installed via pkg install node and reports the following version information: v12.4.0

[euantorano]

Just to test it out, I created a file index.js:

console.log("Hello, console")
console.log(1, 2, 3)
console.log(1, "hi", 1.1)

And ran it with node index.js:

Hello, console
1 2 3
1 hi 1.1

I did the same on the Mac (Node v12.10.0) and got the exact same output, but the test doesn't seem to fail on the Mac CI...

[Araq]

For me it produces the single quotes indeed, I'm on node --version v6.10.0

[euantorano]

There must have been some change in behaviour from then until now then I guess. I had a look at the Travis script and it doesn’t seem to install node on the Mac machine, so it must be pre-installed there but I’ve no idea what version it is. Interestingly, these tests don’t fail on OpenBSD either where I believe node v10.15.3 is used.

…

On Thu, 12 Sep 2019, at 09:52, Andreas Rumpf wrote: For me it produces the single quotes indeed, I'm on node --version v6.10.0 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#12182?email_source=notifications&email_token=AAFW24MXEZLYIOZTEBVLTSLQJH7LPA5CNFSM4IVZFKMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6RFACQ#issuecomment-530731018>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFW24JRZW2DVIUXPPILYZTQJH7LPANCNFSM4IVZFKMA>.

[euantorano]

I've just tested Node v10.16.0 via https://repl.it/languages/nodejs and it outputs:

Hello, console
1 2 3
1 'hi' 1.1

It definitely looks like Node have changed something from v10 - v12 😢

I'm not sure what to do with these tests in that case, but I'm going to contact somebody I know who's more involved with Node to see if they have any ideas when it changed.

[Araq]

Just disable the test for OpenBSD, it's not that important...

[euantorano]

@Araq Yep, we can do that for FreeBSD.

However, there may be a case in the future where these tests suddenly start failing elsewhere in CI due to Node updates. Just something that we should be wary of.

[euantorano]

Looks like the change happened in Node v12. Thanks to @bnb for opening an issue in the Node repository and helping track this down.

• Changelog - see treat format arguments equally
• PR

The AppVeyor script currently installs Node 8.

The Travis script doesn't install Node specifically, as Node is installed as part of the build environment. Currently, the xenial build environment installs both v11.0.0 and v8.12.0.

I've ignored the test on FreeBSD now, so this issue can be closed when the FreeBSD CI PR gets merged.

[stefantalpalaru]

I see the same test failures on Linux with Node.js 12.10.0.

[euantorano]

Yeah, the problem is due to a change in Node 12.0, so any systems running 12.0 or newer will fail these tests. The current solution is to ignore those failures, until all the CI systems are upgraded I guess.

…

On Mon, 23 Sep 2019, at 23:07, Ștefan Talpalaru wrote: I see the same test failures on Linux with Node.js 12.10.0. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#12182?email_source=notifications&email_token=AAFW24KQNCE6IJGZ2NOUU43QLE4ZNA5CNFSM4IVZFKMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7MNLFA#issuecomment-534304148>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFW24LRZAXKSMOSLQWSHF3QLE4ZNANCNFSM4IVZFKMA>.

[stefantalpalaru]

Why would you disable a test on FreeBSD when the output mismatch is also seen on Linux and it's due to a newer Node.js version?

[euantorano]

It was the easiest solution to the problem. We can always revisit it at a later date.

…

On Fri, 29 Nov 2019, at 23:12, Ștefan Talpalaru wrote: Why would you disable a test on FreeBSD when the output mismatch is also seen on Linux and it's due to a newer Node.js version? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#12182?email_source=notifications&email_token=AAFW24NRYDZQQVWMBWAAOU3QWGOV5A5CNFSM4IVZFKMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFPUU5A#issuecomment-559893108>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFW24PHLQ3HFG5H2AHYDI3QWGOV5ANCNFSM4IVZFKMA>.