-
-
Notifications
You must be signed in to change notification settings - Fork 187
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MongoMemoryServer.create hanging in Docker #710
Comments
from what i can tell from the logs, it seems like the binary is found and tried to be started, but then it seems like the log gets cut-off, without any indication on what happened (MMS has logs for successful and error-ed child-processes) can you maybe confirm that the following are actually the last lines of the debug output? MongoMS:MongoInstance Mongo[33933]: prepareCommandArgs: final argument array:["--port","33933","--dbpath","/tmp/mongo-mem--5281-8pCZ60aOpRyc","--storageEngine","ephemeralForTest","--bind_ip","127.0.0.1","--noauth"] +1ms
MongoMS:MongoInstance Mongo[33933]: _launchKiller: Launching Killer Process (parent: 5281, child: 6853) +19ms |
Yes these are the last lines of the log. Jest timeout happens well after the last log. I also found this odd as it felt like there should be more. When I tried to debug within the package I could only get so far. I have also noticed the .cache does not get created in the |
it only attempts to create the folder when it is downloading, but as noted earlier the binary already exists at
just to be sure, could you try doing For now i will try to add a timeout to not leave it hanging (default of 10s timeout) |
After piping the output to a file I am getting the same results with the logs ending after
|
very weird, but because i dont have enough information to go on (or be able to reproduce the issue), i cannot really help with it the only thing that may be helpful is if you add |
I pulled the childProcess. I will keep digging around this area in my debugger. Attempted to log "childProcess1 <ref *1> ChildProcess {
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: undefined,
_closesNeeded: 3,
_closesGot: 0,
connected: false,
signalCode: null,
exitCode: null,
killed: false,
spawnfile: '/root/.cache/mongodb-binaries/mongod-x64-debian-5.0.8',
_handle: Process {
onexit: [Function (anonymous)],
pid: 37213,
[Symbol(owner_symbol)]: [Circular *1]
},
spawnargs: [
'/root/.cache/mongodb-binaries/mongod-x64-debian-5.0.8',
'--port',
'39543',
'--dbpath',
'/tmp/mongo-mem--90422-E4QtiEv7DndD',
'--storageEngine',
'ephemeralForTest',
'--bind_ip',
'127.0.0.1',
'--noauth'
],
pid: 37213,
stdin: <ref *2> Socket {
connecting: false,
_hadError: false,
_parent: null,
_host: null,
_readableState: ReadableState {
objectMode: false,
highWaterMark: 16384,
buffer: BufferList { head: null, tail: null, length: 0 },
length: 0,
pipes: [],
flowing: null,
ended: false,
endEmitted: false,
reading: false,
sync: true,
needReadable: false,
emittedReadable: false,
readableListening: false,
resumeScheduled: false,
errorEmitted: false,
emitClose: false,
autoDestroy: false,
destroyed: false,
errored: null,
closed: false,
closeEmitted: false,
defaultEncoding: 'utf8',
awaitDrainWriters: null,
multiAwaitDrain: false,
readingMore: false,
dataEmitted: false,
decoder: null,
encoding: null,
readable: false,
[Symbol(kPaused)]: null
},
_events: [Object: null prototype] { end: [Function: onReadableStreamEnd] },
_eventsCount: 1,
_maxListeners: undefined,
_writableState: WritableState {
objectMode: false,
highWaterMark: 16384,
finalCalled: false,
needDrain: false,
ending: false,
ended: false,
finished: false,
destroyed: false,
decodeStrings: false,
defaultEncoding: 'utf8',
length: 0,
writing: false,
corked: 0,
sync: true,
bufferProcessing: false,
onwrite: [Function: bound onwrite],
writecb: null,
writelen: 0,
afterWriteTickInfo: null,
buffered: [],
bufferedIndex: 0,
allBuffers: true,
allNoop: true,
pendingcb: 0,
prefinished: false,
errorEmitted: false,
emitClose: false,
autoDestroy: false,
errored: null,
closed: false,
closeEmitted: false
},
allowHalfOpen: false,
_sockname: null,
_pendingData: null,
_pendingEncoding: '',
server: null,
_server: null,
[Symbol(async_id_symbol)]: 888,
[Symbol(kHandle)]: Pipe { [Symbol(owner_symbol)]: [Circular *2] },
[Symbol(kSetNoDelay)]: false,
[Symbol(lastWriteQueueSize)]: 0,
[Symbol(timeout)]: null,
[Symbol(kBuffer)]: null,
[Symbol(kBufferCb)]: null,
[Symbol(kBufferGen)]: null,
[Symbol(kCapture)]: false,
[Symbol(kBytesRead)]: 0,
[Symbol(kBytesWritten)]: 0
},
stdout: <ref *3> Socket {
connecting: false,
_hadError: false,
_parent: null,
_host: null,
_readableState: ReadableState {
objectMode: false,
highWaterMark: 16384,
buffer: BufferList { head: null, tail: null, length: 0 },
length: 0,
pipes: [],
flowing: null,
ended: false,
endEmitted: false,
reading: true,
sync: false,
needReadable: true,
emittedReadable: false,
readableListening: false,
resumeScheduled: false,
errorEmitted: false,
emitClose: false,
autoDestroy: false,
destroyed: false,
errored: null,
closed: false,
closeEmitted: false,
defaultEncoding: 'utf8',
awaitDrainWriters: null,
multiAwaitDrain: false,
readingMore: false,
dataEmitted: false,
decoder: null,
encoding: null,
[Symbol(kPaused)]: null
},
_events: [Object: null prototype] {
end: [Function: onReadableStreamEnd],
close: [Function (anonymous)]
},
_eventsCount: 2,
_maxListeners: undefined,
_writableState: WritableState {
objectMode: false,
highWaterMark: 16384,
finalCalled: false,
needDrain: false,
ending: false,
ended: false,
finished: false,
destroyed: false,
decodeStrings: false,
defaultEncoding: 'utf8',
length: 0,
writing: false,
corked: 0,
sync: true,
bufferProcessing: false,
onwrite: [Function: bound onwrite],
writecb: null,
writelen: 0,
afterWriteTickInfo: null,
buffered: [],
bufferedIndex: 0,
allBuffers: true,
allNoop: true,
pendingcb: 0,
prefinished: false,
errorEmitted: false,
emitClose: false,
autoDestroy: false,
errored: null,
closed: false,
writable: false,
closeEmitted: false
},
allowHalfOpen: false,
_sockname: null,
_pendingData: null,
_pendingEncoding: '',
server: null,
_server: null,
[Symbol(async_id_symbol)]: 889,
[Symbol(kHandle)]: Pipe { reading: true, [Symbol(owner_symbol)]: [Circular *3] },
[Symbol(kSetNoDelay)]: false,
[Symbol(lastWriteQueueSize)]: 0,
[Symbol(timeout)]: null,
[Symbol(kBuffer)]: null,
[Symbol(kBufferCb)]: null,
[Symbol(kBufferGen)]: null,
[Symbol(kCapture)]: false,
[Symbol(kBytesRead)]: 0,
[Symbol(kBytesWritten)]: 0
},
stderr: <ref *4> Socket {
connecting: false,
_hadError: false,
_parent: null,
_host: null,
_readableState: ReadableState {
objectMode: false,
highWaterMark: 16384,
buffer: BufferList { head: null, tail: null, length: 0 },
length: 0,
pipes: [],
flowing: null,
ended: false,
endEmitted: false,
reading: true,
sync: false,
needReadable: true,
emittedReadable: false,
readableListening: false,
resumeScheduled: false,
errorEmitted: false,
emitClose: false,
autoDestroy: false,
destroyed: false,
errored: null,
closed: false,
closeEmitted: false,
defaultEncoding: 'utf8',
awaitDrainWriters: null,
multiAwaitDrain: false,
readingMore: false,
dataEmitted: false,
decoder: null,
encoding: null,
[Symbol(kPaused)]: null
},
_events: [Object: null prototype] {
end: [Function: onReadableStreamEnd],
close: [Function (anonymous)]
},
_eventsCount: 2,
_maxListeners: undefined,
_writableState: WritableState {
objectMode: false,
highWaterMark: 16384,
finalCalled: false,
needDrain: false,
ending: false,
ended: false,
finished: false,
destroyed: false,
decodeStrings: false,
defaultEncoding: 'utf8',
length: 0,
writing: false,
corked: 0,
sync: true,
bufferProcessing: false,
onwrite: [Function: bound onwrite],
writecb: null,
writelen: 0,
afterWriteTickInfo: null,
buffered: [],
bufferedIndex: 0,
allBuffers: true,
allNoop: true,
pendingcb: 0,
prefinished: false,
errorEmitted: false,
emitClose: false,
autoDestroy: false,
errored: null,
closed: false,
writable: false,
closeEmitted: false
},
allowHalfOpen: false,
_sockname: null,
_pendingData: null,
_pendingEncoding: '',
server: null,
_server: null,
[Symbol(async_id_symbol)]: 890,
[Symbol(kHandle)]: Pipe { reading: true, [Symbol(owner_symbol)]: [Circular *4] },
[Symbol(kSetNoDelay)]: false,
[Symbol(lastWriteQueueSize)]: 0,
[Symbol(timeout)]: null,
[Symbol(kBuffer)]: null,
[Symbol(kBufferCb)]: null,
[Symbol(kBufferGen)]: null,
[Symbol(kCapture)]: false,
[Symbol(kBytesRead)]: 0,
[Symbol(kBytesWritten)]: 0
},
stdio: [
<ref *2> Socket {
connecting: false,
_hadError: false,
_parent: null,
_host: null,
_readableState: [ReadableState],
_events: [Object: null prototype],
_eventsCount: 1,
_maxListeners: undefined,
_writableState: [WritableState],
allowHalfOpen: false,
_sockname: null,
_pendingData: null,
_pendingEncoding: '',
server: null,
_server: null,
[Symbol(async_id_symbol)]: 888,
[Symbol(kHandle)]: [Pipe],
[Symbol(kSetNoDelay)]: false,
[Symbol(lastWriteQueueSize)]: 0,
[Symbol(timeout)]: null,
[Symbol(kBuffer)]: null,
[Symbol(kBufferCb)]: null,
[Symbol(kBufferGen)]: null,
[Symbol(kCapture)]: false,
[Symbol(kBytesRead)]: 0,
[Symbol(kBytesWritten)]: 0
},
<ref *3> Socket {
connecting: false,
_hadError: false,
_parent: null,
_host: null,
_readableState: [ReadableState],
_events: [Object: null prototype],
_eventsCount: 2,
_maxListeners: undefined,
_writableState: [WritableState],
allowHalfOpen: false,
_sockname: null,
_pendingData: null,
_pendingEncoding: '',
server: null,
_server: null,
[Symbol(async_id_symbol)]: 889,
[Symbol(kHandle)]: [Pipe],
[Symbol(kSetNoDelay)]: false,
[Symbol(lastWriteQueueSize)]: 0,
[Symbol(timeout)]: null,
[Symbol(kBuffer)]: null,
[Symbol(kBufferCb)]: null,
[Symbol(kBufferGen)]: null,
[Symbol(kCapture)]: false,
[Symbol(kBytesRead)]: 0,
[Symbol(kBytesWritten)]: 0
},
<ref *4> Socket {
connecting: false,
_hadError: false,
_parent: null,
_host: null,
_readableState: [ReadableState],
_events: [Object: null prototype],
_eventsCount: 2,
_maxListeners: undefined,
_writableState: [WritableState],
allowHalfOpen: false,
_sockname: null,
_pendingData: null,
_pendingEncoding: '',
server: null,
_server: null,
[Symbol(async_id_symbol)]: 890,
[Symbol(kHandle)]: [Pipe],
[Symbol(kSetNoDelay)]: false,
[Symbol(lastWriteQueueSize)]: 0,
[Symbol(timeout)]: null,
[Symbol(kBuffer)]: null,
[Symbol(kBufferCb)]: null,
[Symbol(kBufferGen)]: null,
[Symbol(kCapture)]: false,
[Symbol(kBytesRead)]: 0,
[Symbol(kBytesWritten)]: 0
}
],
[Symbol(kCapture)]: false
}". |
I have found the exact code where it stalls but I have no idea why.
this.killerProcess = this._launchKiller(process.pid, this.mongodProcess.pid);
yield launch; // <---- This line
this.emit(MongoInstanceEvents.instanceStarted); |
from what i can tell it looks correctly started
from what i can tell now, is that the child-process gets correctly spawned and does not exit BUT does not output any stdout (or stderr for that matter), which MMS requires to fetch from to get when the database is correctly started, or has a error, see the following code: in case of a error or close, it should fire the but as can be seen from the logs, no stdout, stderr or events were fired, and thats the reason on why i cant really help debug further without being able to reproduce it just to be sure, you are running nodejs and not something like deno, correct? |
Correct. I am using CommonJS. Specifically: Docker Image
I know some of these are packages are in Debian. We recently transitioned from Alpine. I am going to setup a small repo that demonstrates the issue to see if it can be recreated. |
@hasezoey I have create a sample repo that recreates the issue along with instructions on my specific Docker setup and workflow in the Readme. https://github.com/AustinMutschler/debug-mms I was able to run this demo repo on an Intel Macbook using Docker (same steps as the README) and it works as intended but not on my M1 Macbook. This seems to be an issue with an M1 Macbook running a Docker container with platform Docker Engine 20.10.17 My setup: |
tired your repository locally and in docker, but i cannot reproduce this issue, for me it "just works" (Linux Manjaro x86_64, i dont have a Mac device to test on) because you are running on M1, is rosetta installed and working correctly? (i dont more to help with that) |
Rosetta is working fine. I have a feeling it has something to do with how Docker ARM running a linux/amd64 container uses STDOUT |
what you could also try is to: # term 1
docker-compose up --build
# wait for it to start
# term 2
docker ps
docker exec -it containerIdHere /bin/bash
# now in the container in term 2
mkdir /tmp/some-db-path
/root/.cache/mongodb-binaries/mongod-x64-debian-5.0.8 --port 43471 --dbpath /tmp/some-db-path --storageEngine ephemeralForTest --bind_ip 127.0.0.1 --noauth
# and see if it actually outputs something (Note: with |
if i understand correctly, github runners are also running in docker (or at least very similar to it), and gitlab runners do too sad to hear that it is affects multiple users, but i will not be able to debug this because i dont have a apple device (M1) to test on. for example, the following questions would need to be answered:
the place where the binary has been downloaded to can be found by starting your script (with MMS) in debug log mode and looking for a line similar to mkdir /tmp/some-db-path
/path/to/binary/mongod-x64-debian-5.0.8 --port 43471 --dbpath /tmp/some-db-path --storageEngine ephemeralForTest --bind_ip 127.0.0.1 |
@hasezoey according to Docker, to emulate amd64 on ARM machines they use a QEMU virtual machine. The good news for Mac users is they want to which to using Rosetta but have not begun development on that yet. https://docs.docker.com/build/building/multi-platform/ I am not sure if it is re-creatable in there. I will do some digging into the questions you asked above and get back to you. |
I'm not quite sure whether my problem is similar, but it is at least related: I'm trying to run in a docker image on macos with an M1 machine. I have prepared an amazonlinux image with node to be able to use mongodb-linux-aarch64-amazon2-4.4.6. (I'm using 4.4.6 since I also run that on an Gitlab instance with old CPUs...). The file is correctly computed from os and arch, but unfortunately the download takes more than 10 seconds. Unfortunately the 10 seconds timeout cannot be changed, can it? When creating an instance via
However, this configuration is not passed to MongoInstance later on due to these lines: That is, I'm using a workaround by simply downloading the file manually, extract bin/mongod from the file and rename it accordingly (at |
@jpilgrim you can also set Also has the benefit (if you're on Docker) of better caching, since it happens at an earlier layer. |
@kevincennis, I already have configured that (actually, I set the version in package.json via config/mongodbMemoryServer/version. This (the version of the mondod binary) is not the issue here. The problem is that downloading the binaries (via your aforementioned hook) has a (more or less) hard coded timeout. This is why I thought that my issue is related to this thread here. Oh, maybe I should add that the timeout also hinders the binary to be downloaded correctly (or, let's say, completely) during |
thanks for pointing this out, will fix it in a patch version soon (and will probably refactor that thing for the next major)
the timeout is for launching the binary not downloading the binary |
@hasezoey Thanks! So with "patch" you mean: Add a setting for timeout for downloading the binary ? |
no, with the actual download has no timeout (that i know of) |
@hasezoey Thanks for the fast fix! I'm happy to helping improve mms... even though I apparently have a different problem. Very weird, seems to be related to how I start docker... As a matter of fact during npm install the download is cancelled after 10 seconds. But I have no idea -- and this is not a docker forum anyway :-) |
weird, from what i know docker does not have timeout things, so maybe your package-manager somehow has a timeout for postinstall scripts? (though from what i know yarn and npm dont have such a thing) |
After creating the
and
Both commands hang until cancelled. |
thanks for that information, this would mean it is a mongodb's issues, nothing MMS could do about (that i know), a issue would have to be opened at https://jira.mongodb.org/browse/ (and for discoverability, i would request to have the url of the opened issue posted in this thread to) |
@hasezoey Update I replaced the mongod file from I then manually replaced the mongod binary with a Debian 11 5.0.13 binary and it failed with the hanging issue. Root problemI was reading a post in a MongoDB forum and they said that QEMU does not work with >= v5 binaries because of issues with "AVX Instructions". Future SolutionThe good news is, last week QEMU committed a change for AVX instructions which is said to potentially fix the issue. Hopefully Docker will update QEMU when it releases and eventually support newer versions of Mongo. For now, it looks like ARM processors debugging in Docker using AMD64 will not be able to use MMS for v5 or v6 MongoDB. I know this is probably a rare instance because usually I would run the container on ARM natively and all would be well; however, there is another application in the container (Oracle Instantclient) that requires amd64 because it has little support for ARM (plus we have devs with both types of machines). WorkaroundIf it is acceptable to use Mongo 4.4 for testing you just need to change the version MMS uses by specifying it in an ENV variable or in the create configuration. const mongod = await MongoMemoryServer.create(
{
binary: {
version: '4.4.17'
}
}
); |
thanks for finding this, i initially excluded the AVX problem, because normally mongodb (at least on windows and linux from what i can tell) print a failing message / error and are not left hanging, like a exit signal of SIGILL |
Marking Issue as stale, will be closed in 7 days if no more activity is seen |
Closing Issue because it is marked as stale |
Update: |
Versions
package: mongo-memory-server
What is the Problem?
When running my Express application in Docker for local development and running jest test command; mongodb-memory-server stalls at
const mongod = await MongoMemoryServer.create();
Code Example
Debug Output
Debug Output
Do you know why it happenes?
I have been trying to debug this for some time but with no luck. I have gone through the stack to see if I could spot the problem. At the surface level of the issue
await MongoMemoryServer.create()
is not returning anything.When diving into the code I noticed that
this._startUpInstance(forceSamePort)
is the last area of code being executed.The code below is never being called
As a result
return instance
is not called.At first glance, I do not see any errors in the logs which would be causing this issue.
This issue does not happen when running the tests locally on my M1 Macbook Pro.
Steps I have taken
apt-get install libaio1
The text was updated successfully, but these errors were encountered: