:test:fixtures:krb5kdc-fixture:composeUp failure

[gwbrown]

CI failed on 7.x with this error:

ERROR: for a5c3c1596d62b7b1842fcb2469c0c711_krb5kdc-fixture__hdfs_1  Cannot create container for service hdfs: error creating overlay mount to /var/lib/docker/overlay2/e40001620e6f9a60ea6001580ccd075835a2d8234c75c50bf056f66d9ba1cba7/merged: device or resource busy

Build scan

Looks like docker-compose had trouble creating an hdfs container. I'm not sure if this issue is more appropriate for Elasticsearch or Infra.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Build)

[rjernst]

This seems like an issue with the underlying docker image we are trying to start. @jbaiera can you have a look?

[alpar-t]

This particular failure is on ubuntu-1804 I know we'we had similar issues on SLES and maybe fedora 29 and we pulled them from the general worker pool, but didn't get to the bottom of the issue.

[droberts195]

Another one happened in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob+fast+part2/1688/consoleText:

Creating network "a5c3c1596d62b7b1842fcb2469c0c711_krb5kdc-fixture__default" with the default driver
Creating a5c3c1596d62b7b1842fcb2469c0c711_krb5kdc-fixture__hdfs_1 ... 
Creating a5c3c1596d62b7b1842fcb2469c0c711_krb5kdc-fixture__peppa_1 ... 
Creating a5c3c1596d62b7b1842fcb2469c0c711_krb5kdc-fixture__peppa_1 ... error

ERROR: for a5c3c1596d62b7b1842fcb2469c0c711_krb5kdc-fixture__peppa_1  Cannot create container for service peppa: error creating overlay mount to /var/lib/docker/overlay2/9d2967ca570b8c46643a30a2e801bcdde60e0dfbbed40c7f41936e41d75f6f57/merged: device or resource busy
Creating a5c3c1596d62b7b1842fcb2469c0c711_krb5kdc-fixture__hdfs_1  ... done

ERROR: for peppa  Cannot create container for service peppa: error creating overlay mount to /var/lib/docker/overlay2/9d2967ca570b8c46643a30a2e801bcdde60e0dfbbed40c7f41936e41d75f6f57/merged: device or resource busy
Encountered errors while bringing up the project.
Stopping a5c3c1596d62b7b1842fcb2469c0c711_krb5kdc-fixture__hdfs_1 ... 

The worker this happened on was elasticsearch-ci-immutable-ubuntu-1804-1571225771493903523. As it was ephemeral it's gone now.

[romseygeek]

This also just happened on a CI build on one of my PRs: https://gradle-enterprise.elastic.co/s/aexbtvi3nivxo/console-log?task=:test:fixtures:krb5kdc-fixture:composeUp

[alpar-t]

Both of those are on ubuntu 18, so I'm starting to believe there's something specific on that platform, possibly specific to newer versions of something ( kernel or docker ) and we got an upgrade in this version of Ubuntu.

I have a set of PRs that changes how we gather logs once the build completes to include system logs.
I'm hoping that there will be something useful in there. I'm holding off on removing the platform from the general pool for now.
I tried to reproduce something similar on SLES but I it never did, so might also have to do with the fact that the worker just started.

[mark-vieira]

Definitely seems related to ubuntu 18.04.

https://gradle-enterprise.elastic.co/scans/failures?failures.failureClassification=non_verification&failures.failureMessage=*%0A%3E%20Exit-code%201%20when%20calling%20/usr/local/bin/docker-compose,%20stdout:%20N/A*&list.offset=0&list.size=50&list.sortColumn=startTime&list.sortOrder=desc&search.buildToolType=gradle&search.buildToolType=maven&search.startTimeMax=1571342751552&search.startTimeMin=1570737951552&trends.section=overview&trends.timeResolution=day&viewer.tzOffset=-420

This Debian 10 build also failed for the same reason, which would make sense since Ubuntu 18.04 is based on Debian 10.

[alpar-t]

New failures should have a "GCP upload" link in the build scan. We have to wait for one of those to get additional insight on this.

[mark-vieira]

@atorok I just took a look at the logs uploaded to GCP for this failure and the only thing in there was the Gradle daemon log which is effectively identical to the console output. What kind of Docker-specific logging were we expecting to capture here?

[mark-vieira]

@atorok it looks like we aren't getting a journalctl log because we need to call supply() here:

elasticsearch/gradle/build-complete.gradle

Line 28 in 28b599d

}

[alpar-t]

I opened #48276 to address that