Crashes when using ajv on Alpine Linux

[daveisfera]

• Version:
  Happens on 4.x, 6.x and 7.x.

• Platform:
  Alpine Linux 3.4

• Subsystem:
  v8

There's an issue open for docker-node here and it has a backtrace from gdb.

Running this script reproduces the crash:

#!/usr/bin/env node
'use strict';

const Ajv = require('ajv');
const validator = new Ajv({ allErrors: true, extendedRefs: false });

const STRING_KEY = {
    id: '/StringKey',
    type: 'string',
    maxLength: 10000,
};

validator.addSchema(STRING_KEY);

let schema = {
    type: 'object',
    properties: {},
};

const NUM_COLUMNS = parseInt(process.argv[2] || '81');
console.log(`Testing with ${NUM_COLUMNS} columns`);
let c;
for (c=0; c<NUM_COLUMNS; c++) {
    schema.properties[`s${c}`] = { $ref: '/StringKey'};
}
console.log('schema:', schema);
const validate = validator.compile(schema);

let value = {};
for (c=0; c<NUM_COLUMNS; c++) {
    const cS = `s${c}`;
    value[cS] = '';
}
console.log('value:', value);

const NUM_ROWS = parseInt(process.argv[3] || '394');
console.log(`Testing with ${NUM_ROWS} rows`);
let r;
for (r=0; r<NUM_ROWS; r++) {
    validate(value);
}

console.log('Done');

I used this Dockerfile to run the test (also crashes when using FROM node:4.8.0-alpine and FROM node:7.7.3-alpine:

FROM node:6.10.0-alpine

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

RUN yarn add ajv

COPY test_ajv.js /usr/src/app

CMD [ "node", "test_ajv.js" ]

And ran these commands:

docker build -t test_ajv .
docker run --rm -it test_ajv sh
node test_ajv.js

[mscdex]

Backtrace:

#0  0x0000564f3d67e63e in v8::internal::HGlobalValueNumberingPhase::CollectSideEffectsOnPathsToDominatedBlock(v8::internal::HBasicBlock*, v8::internal::HBasicBlock*) ()
#1  0x0000564f3d67e6da in v8::internal::HGlobalValueNumberingPhase::CollectSideEffectsOnPathsToDominatedBlock(v8::internal::HBasicBlock*, v8::internal::HBasicBlock*) ()
#2  0x0000564f3d67e6da in v8::internal::HGlobalValueNumberingPhase::CollectSideEffectsOnPathsToDominatedBlock(v8::internal::HBasicBlock*, v8::internal::HBasicBlock*) ()
...
#1287 0x0000564f3d67e6da in v8::internal::HGlobalValueNumberingPhase::CollectSideEffectsOnPathsToDominatedBlock(v8::internal::HBasicBlock*, v8::internal::HBasicBlock*) ()
#1288 0x0000564f3d67e6da in v8::internal::HGlobalValueNumberingPhase::CollectSideEffectsOnPathsToDominatedBlock(v8::internal::HBasicBlock*, v8::internal::HBasicBlock*) ()
#1289 0x0000564f3d67e6da in v8::internal::HGlobalValueNumberingPhase::CollectSideEffectsOnPathsToDominatedBlock(v8::internal::HBasicBlock*, v8::internal::HBasicBlock*) ()
#1290 0x0000564f3d68020c in v8::internal::HGlobalValueNumberingPhase::AnalyzeGraph() ()
#1291 0x0000564f3d6807dd in v8::internal::HGlobalValueNumberingPhase::Run() ()
#1292 0x0000564f3d6b13c5 in void v8::internal::HGraph::Run<v8::internal::HGlobalValueNumberingPhase>() ()
#1293 0x0000564f3d6be624 in v8::internal::HGraph::Optimize(v8::internal::BailoutReason*) ()
#1294 0x0000564f3d640aec in v8::internal::OptimizedCompileJob::OptimizeGraph() ()
#1295 0x0000564f3d8cc379 in v8::internal::OptimizingCompileDispatcher::CompileTask::Run() ()
#1296 0x0000564f3dc67329 in v8::platform::WorkerThread::Run() ()
#1297 0x0000564f3de923b0 in v8::base::ThreadEntry(void*) ()
#1298 0x00007f97cec3a655 in ?? () from /lib/ld-musl-x86_64.so.1
#1299 0x0000000000000000 in ?? ()

[mscdex]

/cc @nodejs/v8

[hashseed]

Looks like Crankshaft is running into stack overflow. Turbofan likely won't have this issue. I'll take a look whether there is an easy fix here.

[hashseed]

I can't actually reproduce it on my Ubuntu workstation. I installed ajv 1.3.10 via npm. The test case runs fine on v7.7.5-pre both release and debug mode, and on v7.7.3 debug mode.

I wonder whether Alpine Linux has different settings wrt stack limit. Maybe you can try with running with --no-use-gvn?

[daveisfera]

I can't actually reproduce it on my Ubuntu workstation. I installed ajv 1.3.10 via npm. The test case runs fine on v7.7.5-pre both release and debug mode, and on v7.7.3 debug mode.

I can only reproduce the crash when running on Alpine Linux.

I wonder whether Alpine Linux has different settings wrt stack limit. Maybe you can try with running with --no-use-gvn?

It does not crash when I run with --no-use-gvn.

[daveisfera]

I just tried with node 6.10.2 and it now crashes with a smaller number of columns (70) but the same number of rows is required (394).

[johan13]

I wonder whether Alpine Linux has different settings wrt stack limit.

Alpine Linux uses musl libc which has a default stack size of just 80 kB. Glibc for example has a default stack size of 8 MB.

[hashseed]

80kb sounds very limited. The correct fix would be to add stack checks to Crankshaft's GVN, but Crankshaft in upstream V8 is no longer under development.

[johan13]

It doesn't crash if you use the Turbofan compiler.
node --turbo test_ajv.js

[bnoordhuis]

Does node --stack_size=$kb test_ajv.js, where $kb is a value < 80, work? If yes, I suppose we could detect RLIMIT_STACK and adjust --stack_size accordingly.

Is the limit set my musl or alpine? I don't see anything in musl that suggests it modifies RLIMIT_STACK.

[hashseed]

@bnoordhuis that would be a good idea generally, but wouldn't help this case since the recursion in GVN doesn't perform stack checks. It would prevent crashes where we do though.

[bnoordhuis]

Good point. I think we could float a patch that adds stack checks in the appropriate places or make it non-recursive. CollectSideEffectsOnPathsToDominatedBlock appears to be the main offender.

[johan13]

@bnoordhuis, the limit is in musl: https://anonscm.debian.org/cgit/collab-maint/musl.git/tree/src/internal/pthread_impl.h#n144

[bnoordhuis]

@johan13 Can you try #12460?

the limit is in musl: https://anonscm.debian.org/cgit/collab-maint/musl.git/tree/src/internal/pthread_impl.h#n144

Ah, I understand now - I was looking at execve() and friends but it's a per-thread setting. In that case --noconcurrent_recompilation can probably be used as a workaround too, depending on the RLIMIT_STACK of the main thread (which hopefully is bigger.)

[johan13]

Yes, --noconcurrent_recompilation seems to do the trick as well.

I have never built node from source and I don't have a system with musl and build tools installed, so it would probably take me a little while to try the pull request.

[bnoordhuis]

Fixed in 30989d3 and coming to a release near you soon.

[daveisfera]

Awesome! Thanks for putting that fix in, but are there any other concerns about the stack being smaller with Alpine?

[bnoordhuis]

80 kB is on the slim side but it should be sufficient. Just file new issues if you hit more bugs and we'll take a look.

cc @jbergstroem - didn't you mention an alpine buildbot a while ago?