Docker: The display compositor is frequently crashing. Goodbye. #3774

franz101 · 2019-01-15T10:34:23Z

Hi,
I'm using a docker container with node:8-slim
and puppeteer 1.10.0. The container runs fine locally but on my Debian server I got the error Message:
gpu_data_manager_impl_private.cc(892)] The display compositor is frequently crashing. Goodbye.
I guess this might be chrome related and maybe caused through lack of memory.

.... Couldn't find anyone experiencing this error before.

franz101 · 2019-01-15T11:31:33Z

Found a solution: Set chrome to stable

nikhilo · 2019-03-14T11:26:26Z

This seems to have broken again in latest stable. Google Chrome 73.0.3683.75

sonnyt · 2019-03-14T14:22:35Z

@nikhilo same here, did you find a solution?

rafaelbnp · 2019-03-14T17:49:34Z

@nikhilo @sonnyt have been fighting this since yesterday. Any luck?

sonnyt · 2019-03-14T20:33:00Z

@rafael-paiva nope, just ended up downgrading to Chromium 72.0.3626.121.

rafaelbnp · 2019-03-15T16:43:29Z

I'm using headless Firefox so my tests are not blocked. @franz101 can we re-open this issue?

aslushnikov · 2019-03-15T17:01:29Z

Can you guys share your docker container so that we can see this happenning locally?

chrismllr · 2019-03-15T20:05:17Z

This is happening for me on node:10-browsers and node:lts-browsers, pre-built docker images we're using in Circle: https://circleci.com/docs/2.0/circleci-images/#nodejs

These install Chrome @ 73.0.3683.75

Downgrading to node:10.14-browsers did the trick in the meantime (installed Chrome ~71)

aslushnikov · 2019-03-15T21:50:22Z

@chrismllr do I understand correctly that it crashes for the Chrome that's bundled with the docker image? Does the Chromium version we bundle with Puppeteer crash the same way?

yhatt · 2019-03-18T09:28:32Z

It has also happened similar crashes while running tests for our project.
https://circleci.com/gh/marp-team/marp-cli/1193

By using Chromium@74.0.3723.0 (r637110: Puppeteer v1.13.0 bundled version) instead Chrome@73.0.3683.75, it gets working well.

Chrome 73 breaks headless tests on CircleCI (see puppeteer/puppeteer#3774). While it's being investigated, using an older image should get us running tests again.

chrismllr · 2019-03-18T12:45:13Z

@aslushnikov Correct, it was the chrome in the prebuilt docker image. I may have jumped in based on a problem I was having elsewhere, but the bug seemed to be identical.

Previously our E2E tests started failing with the following error. chrome not reachable 12:41:37 (Session info: headless chrome=73.0.3683.75) This issue was limited to the CI agent machines. In order to debug the issue, we SSH'd to ci-agent-8, became the jenkins user and changed to one of the workspace directories where the E2E tests are run. We than ran the E2E tests manually in order to reproduce the issue, as follows. make clone make pull make start docker-compose run publishing-e2e-tests bash Once inside the container where the tests are run, we were able to reproduce the issue with 'bundle exec rspec'. In order to investigate further, we then installed vim, in order to install the irb gem and start an irb console using 'bundle exec irb -Ispec'. Then we did require 'spec_helper' driver = Capybara.drivers[Capybara.current_driver].call driver.visit('https://google.com') driver.visit('https://google.com') Running the visit method twice yields the same error as when running the tests. Using 'docker exec' to start another bash console and inspect the running processes shows that Chrome itself is failing to start. root 252 245 11 17:26 pts/1 00:00:00 [chrome] <defunct> Running chrome manually with the options from the spec_helper then yields the following error, even though we hadn't changed these options. google-chrome-stable --disable-dev-shm-usage --disable-gpu --disable-web-security --disable-infobars --disable-notifications --headless --no-sandbox --window-size=1400,1400 https://google.com [0318/173719.964171:FATAL:gpu_data_manager_impl_private.cc(892)] The display compositor is frequently crashing. Goodbye. Searching online for this error indicates its related to a new version of Chrome, as per the following issue on the puppeteer repo. puppeteer/puppeteer#3774 Unfortunately it's not possible for us to downgrade Chrome, since Google only provide the latest version in their package repo, and the E2E tests are being run in transient containers, which have no older versions available to downgrade to. This is the point where we lost all hope. In Chapter 1 we experimented with running Chrome manually, based on https://developers.google.com/web/updates/2017/04/headless-chrome. google-chrome-stable --headless --no-sandbox https://google.com The success of this command indicated one of the options specified in the spec_helper was causing Chrome to crash, and experimentation showed this was '--disable-dev-shm-usage'. Removing this parameter fixed the error, but caused Chrome to crash for a different reason. /dev/shm is a tmpfs partition, but by default it is only 64M in size. Previously, we had specified the '--disable-dev-shm-usage' option to use /tmp instead, but the new release of Chrome makes this option unusable for some reason. The obvious remedy is to increase the size of /dev/shm. publishing-e2e-tests: shm_size: 2G <<< build: . The combination of removing the faulty option and specifying a larger size for /dev/shm meant we could then run the E2E tests successfully. And they all lived happily ever after.

Chapter 1: The Long Day ======================= Previously our E2E tests started failing with the following error. chrome not reachable 12:41:37 (Session info: headless chrome=73.0.3683.75) This issue was limited to the CI agent machines. In order to debug the issue, we SSH'd to ci-agent-8, became the jenkins user and changed to one of the workspace directories where the E2E tests are run. We than ran the E2E tests manually in order to reproduce the issue, as follows. make clone make pull make start docker-compose run publishing-e2e-tests bash Once inside the container where the tests are run, we were able to reproduce the issue with 'bundle exec rspec'. In order to investigate further, we then installed vim, in order to install the irb gem and start an irb console using 'bundle exec irb -Ispec'. Then we did require 'spec_helper' driver = Capybara.drivers[Capybara.current_driver].call driver.visit('https://google.com') driver.visit('https://google.com') Running the visit method twice yields the same error as when running the tests. Using 'docker exec' to start another bash console and inspect the running processes shows that Chrome itself is failing to start. root 252 245 11 17:26 pts/1 00:00:00 [chrome] <defunct> Running chrome manually with the options from the spec_helper then yields the following error, even though we hadn't changed these options. google-chrome-stable --disable-dev-shm-usage --disable-gpu --disable-web-security --disable-infobars --disable-notifications --headless --no-sandbox --window-size=1400,1400 https://google.com [0318/173719.964171:FATAL:gpu_data_manager_impl_private.cc(892)] The display compositor is frequently crashing. Goodbye. Searching online for this error indicates its related to a new version of Chrome, as per the following issue on the puppeteer repo. puppeteer/puppeteer#3774 Unfortunately it's not possible for us to downgrade Chrome, since Google only provide the latest version in their package repo, and the E2E tests are being run in transient containers, which have no older versions available to downgrade to. This is the point where we lost all hope. Chapter 2: The New Dawn ======================= In Chapter 1 we experimented with running Chrome manually, based on https://developers.google.com/web/updates/2017/04/headless-chrome. google-chrome-stable --headless --no-sandbox https://google.com The success of this command indicated one of the options specified in the spec_helper was causing Chrome to crash, and experimentation showed this was '--disable-dev-shm-usage'. Removing this parameter fixed the error, but caused Chrome to crash for a different reason. /dev/shm is a tmpfs partition, but by default it is only 64M in size. Previously, we had specified the '--disable-dev-shm-usage' option to use /tmp instead, but the new release of Chrome makes this option unusable for some reason. The obvious remedy is to increase the size of /dev/shm. publishing-e2e-tests: shm_size: 2G <<< build: . The combination of removing the faulty option and specifying a larger size for /dev/shm meant we could then run the E2E tests successfully. And they all lived happily ever after.

Ref the bug mentioned in puppeteer/puppeteer#3774

franz101 · 2019-03-21T13:39:05Z

This seems to have broken again in latest stable. Google Chrome 73.0.3683.75

franz101 · 2019-03-21T13:40:22Z

`FROM node:8-slim
EXPOSE 3000 9229
#FROM node:latest

See https://crbug.com/795759

RUN apt-get update && apt-get install -yq libgconf-2-4

Install latest chrome dev package and fonts to support major charsets (Chinese, Japanese, Arabic, Hebrew, Thai and a few others)

Note: this installs the necessary libs to make the bundled version of Chromium that Puppeteer

installs, work.

RUN apt-get update && apt-get install -y wget --no-install-recommends
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
&& apt-get update
&& apt-get install -y google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst ttf-freefont
--no-install-recommends
&& rm -rf /var/lib/apt/lists/*
&& apt-get purge --auto-remove -y curl
&& rm -rf /src/*.deb

It's a good idea to use dumb-init to help prevent zombie chrome processes.

ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 /usr/local/bin/dumb-init
RUN chmod +x /usr/local/bin/dumb-init

Uncomment to skip the chromium download when installing puppeteer. If you do,

you'll need to launch puppeteer with:

browser.launch({executablePath: 'google-chrome-unstable'})

ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true

RUN groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser
&& mkdir -p /home/pptruser/Downloads && mkdir -p /home/pptruser/app/img
&& chown -R pptruser:pptruser /home/pptruser
RUN chown -R pptruser:pptruser /home/pptruser
RUN chmod -R 777 /home/pptruser/app/img

WORKDIR /home/pptruser/app
COPY . /home/pptruser/app

Install puppeteer so it's available in the container.

RUN npm i puppeteer

Add user so we don't need --no-sandbox.

RUN npm i -f
RUN npm install

Run everything after as non-privileged user.

USER pptruser
CMD npm run docker-start

`

@aslushnikov

sonnyt · 2019-03-21T13:41:22Z

I switched over to google-chrome-unstable which is 74.0... it works!

franz101 · 2019-03-21T14:20:20Z

@rafael-paiva nope, just ended up downgrading to Chromium 72.0.3626.121.

how to you downgrade in the docker file? It's working now using a different docker container:
https://github.com/buildkite/docker-puppeteer

but how do you set the chrome version to fixed?

skamenetskiy · 2019-03-22T09:09:55Z

Having the same issue when running in ubuntu container under windows (not docker).

google-chrome --version
Google Chrome 74.0.3729.22 dev

22 03 2019 12:09:12.354:ERROR [launcher]: ChromeHeadless crashed.
DevTools listening on ws://127.0.0.1:9222/devtools/browser/a9e949f6-bb7e-498f-a4d7-e2164c42fddc
[0322/120912.025646:FATAL:gpu_data_manager_impl_private.cc(897)] The display compositor is frequently crashing. Goodbye.
Failed to generate minidump.[0322/120912.337254:ERROR:broker_posix.cc(43)] Invalid node channel message

mrodal · 2019-03-28T19:40:13Z

Im having the same issue on Google Chrome 74.0.3729.40 beta on WSL... Is this being worked by someone? should we create a bug report somewhere?

dpdudek · 2019-04-05T09:28:25Z

In my case I resolved it by adding option --disable-features=VizDisplayCompositor.

aslushnikov · 2019-04-20T08:29:18Z

WSL-related issues are unrelated to Docker. Other than that, this seems to be fixed.

MolloKhan · 2019-06-05T02:29:53Z

In my case I resolved it by adding option --disable-features=VizDisplayCompositor.

@dpdudek Man after a couple of hours of dealing with this problem that option saved my day. Thanks!
I have a quick question. Where can I check the list of features? I mean, how did you know that you have to disable that feature? It might be helpful in the future

dpdudek · 2019-06-05T09:04:48Z

@larzuk91 When I enabled verbose log
driver = webdriver.Chrome(executable_path=CHROME_DRIVER_PATH, options=chrome_options, service_args=["--verbose", "--log-path=/var/log/driver.log"])
then message from driver was "The display compositor is frequently crashing. Goodbye.". After that I found that message in gpu_data_manager_impl_private.cc. Looking in code I found that it was some inconsistency with disable gpu and VizDisplayCompositor feature in "if" expressions (I had disabled gpu by
--disable-gpu parameter). Somewhere else I found how to disable VizDisplayCompositor and it started working, so I stopped at that.

List of features is also available under chrome://flags, but now VizDisplayCompositor is not visible in my version of chrome ( 75.0.3770.80).

https://chromium.googlesource.com/chromium/src/+/fd6ee1143ba55b99a627f158ef61bb80b898fe97/content/browser/gpu/gpu_data_manager_impl_private.cc

MolloKhan · 2019-06-05T16:20:07Z

Ohh, it's clearer now. Thanks a lot for helping me understand what was happening. The funny part is by disabling that feature, it only crashes 3 times and after that it works

DevTools listening on ws://127.0.0.1:9222/devtools/browser/02600cca-cb6a-478f-be44-43b17ca624e6
[0605/111803.924207:WARNING:gpu_process_host.cc(1205)] The GPU process has crashed 1 time(s)
[0605/111804.278884:WARNING:gpu_process_host.cc(1205)] The GPU process has crashed 2 time(s)
[0605/111804.633481:WARNING:gpu_process_host.cc(1205)] The GPU process has crashed 3 time(s)

Not sure why but I'm happy that I can run my tests locally :)

cah-chase-spencer · 2019-11-26T17:39:12Z

In my case I resolved it by adding option --disable-features=VizDisplayCompositor.

THANK YOU!!

Bug-Reaper · 2022-02-20T09:16:11Z

Ran into the same error with an electron app I've had installed forever that sporadically broke. Solved with --no-sandbox of all things which seems super weird considering it's a compositor issue.

franz101 closed this as completed Jan 15, 2019

yhatt mentioned this issue Mar 18, 2019

Upgrade Node and dependent packages to the latest marp-team/marp-cli#77

Merged

jonathanperret mentioned this issue Mar 18, 2019

[TECH] Correction du build CircleCI en forçant Chrome 72 1024pix/pix#414

Merged

fatso83 added a commit to fatso83/sinon that referenced this issue Mar 20, 2019

Use pre-installed browsers (v72) instead of fetching latest

2a37b17

Ref the bug mentioned in puppeteer/puppeteer#3774

franz101 reopened this Mar 21, 2019

yhatt mentioned this issue Apr 7, 2019

Update a workaround for the stable chrome's crash in docker image marp-team/marp-cli#80

Merged

aslushnikov closed this as completed Apr 20, 2019

skyeagle mentioned this issue Apr 25, 2019

Does it run on hosted CI e.g. CircleCI, TravisCI? twalpole/apparition#10

Closed

flotwig mentioned this issue Oct 29, 2019

WSL | Electron 5.0 is producing gpu_data_manager_impl_private display compositor crashing cypress-io/cypress#5526

Closed

jennifer-shehane mentioned this issue Jan 15, 2020

wslbridge (Ubuntu on Windows) The display compositor is frequently crashing. cypress-io/cypress#5918

Open

garpu mentioned this issue Feb 22, 2022

--no-sandbox needed for glibc 2.35 Sn8z/Poddr#1111

Open

TDiblik mentioned this issue Apr 17, 2022

AppImage Fails to start getstation/desktop-app#82

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker: The display compositor is frequently crashing. Goodbye. #3774

Docker: The display compositor is frequently crashing. Goodbye. #3774

franz101 commented Jan 15, 2019

franz101 commented Jan 15, 2019

nikhilo commented Mar 14, 2019 •

edited

Loading

sonnyt commented Mar 14, 2019

rafaelbnp commented Mar 14, 2019

sonnyt commented Mar 14, 2019

rafaelbnp commented Mar 15, 2019

aslushnikov commented Mar 15, 2019

chrismllr commented Mar 15, 2019 •

edited

Loading

aslushnikov commented Mar 15, 2019

yhatt commented Mar 18, 2019

chrismllr commented Mar 18, 2019

franz101 commented Mar 21, 2019

franz101 commented Mar 21, 2019

sonnyt commented Mar 21, 2019

franz101 commented Mar 21, 2019

skamenetskiy commented Mar 22, 2019

mrodal commented Mar 28, 2019 •

edited

Loading

dpdudek commented Apr 5, 2019

aslushnikov commented Apr 20, 2019

MolloKhan commented Jun 5, 2019 •

edited

Loading

dpdudek commented Jun 5, 2019

MolloKhan commented Jun 5, 2019 •

edited

Loading

cah-chase-spencer commented Nov 26, 2019

Bug-Reaper commented Feb 20, 2022

Docker: The display compositor is frequently crashing. Goodbye. #3774

Docker: The display compositor is frequently crashing. Goodbye. #3774

Comments

franz101 commented Jan 15, 2019

franz101 commented Jan 15, 2019

nikhilo commented Mar 14, 2019 • edited Loading

sonnyt commented Mar 14, 2019

rafaelbnp commented Mar 14, 2019

sonnyt commented Mar 14, 2019

rafaelbnp commented Mar 15, 2019

aslushnikov commented Mar 15, 2019

chrismllr commented Mar 15, 2019 • edited Loading

aslushnikov commented Mar 15, 2019

yhatt commented Mar 18, 2019

chrismllr commented Mar 18, 2019

franz101 commented Mar 21, 2019

franz101 commented Mar 21, 2019

See https://crbug.com/795759

Install latest chrome dev package and fonts to support major charsets (Chinese, Japanese, Arabic, Hebrew, Thai and a few others)

Note: this installs the necessary libs to make the bundled version of Chromium that Puppeteer

installs, work.

It's a good idea to use dumb-init to help prevent zombie chrome processes.

Uncomment to skip the chromium download when installing puppeteer. If you do,

you'll need to launch puppeteer with:

browser.launch({executablePath: 'google-chrome-unstable'})

ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true

Install puppeteer so it's available in the container.

Add user so we don't need --no-sandbox.

Run everything after as non-privileged user.

sonnyt commented Mar 21, 2019

franz101 commented Mar 21, 2019

skamenetskiy commented Mar 22, 2019

mrodal commented Mar 28, 2019 • edited Loading

dpdudek commented Apr 5, 2019

aslushnikov commented Apr 20, 2019

MolloKhan commented Jun 5, 2019 • edited Loading

dpdudek commented Jun 5, 2019

MolloKhan commented Jun 5, 2019 • edited Loading

cah-chase-spencer commented Nov 26, 2019

Bug-Reaper commented Feb 20, 2022

nikhilo commented Mar 14, 2019 •

edited

Loading

chrismllr commented Mar 15, 2019 •

edited

Loading

mrodal commented Mar 28, 2019 •

edited

Loading

MolloKhan commented Jun 5, 2019 •

edited

Loading

MolloKhan commented Jun 5, 2019 •

edited

Loading