Container build fails after docker log limit reached

[TravisBowers]

Output of the ue4-docker info command:
ue4-docker.txt

Additional details:

• Are you accessing the network through a proxy server? No

I am attempting to build a ue4-engine image using ue4-docker running in a container built by the following Dockerfile:

FROM ubuntu:focal AS build
# set the timezone, add the docker repository, and install ue4-docker and dependencies
RUN ln -fs /usr/share/zoneinfo/America/Detroit /etc/localtime &&\
    apt-get update && apt-get install -y \
    apt-transport-https \
    ca-certificates \
    gnupg \
    curl \
    lsb-release &&\
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg &&\
    echo \
    "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
    $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null &&\
    apt-get update && apt-get install -y \
    docker-ce \
    docker-ce-cli \
    containerd.io \
    python3 \
    python3-dev \
    python3-pip  &&\
    pip3 install ue4-docker

CMD ue4-docker info

and invoked by the following gitlab-ci job.

build-unreal-engine:
  stage: build
  needs:
    - job: build-ue4-docker
      optional: true
  image: $REGISTRY_URL/ue4-docker:$CI_COMMIT_REF_SLUG
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - export UE4DOCKER_TAG_NAMESPACE=$REGISTRY_URL
    - ue4-docker build $VERSION -username=$GITHUB_USERNAME -password=$GITHUB_PASSWORD --cuda=10.2 --no-minimal
    - docker push $REGISTRY_URL/ue4-engine:$VERSION

This job fails after ~45 minutes. Unfortunately, it is impossible to tell what is failing because the docker logs are getting clipped ~26 minutes into the build with [output clipped, log limit 1MiB reached]

Is there any way to reduce the log output produced by ue4-docker?
Alternatively, what configuration changes can be made to increase the docker build log size limit?

[TBBle]

Per docker/buildx#484, that log clipping is coming from BuildKit, not Docker.

I see BUILDKIT_STEP_LOG_MAX_SIZE=10000000 in your ue4-docker.txt, which implies (to me) a 10MB log limit, so I'm not sure why that didn't work. Maybe that env-var is no longer supported?

You might need to poke around in the above-linked issue to work out how correctly tell BuildKit to allow a larger step log. It seems like maybe you need to create a new builder with the larger log limit? I don't have the code in front of me, but I don't recall if we have a way for the ue4-docker command to be told to use a different buildx builder than the default, and that issue didn't seem to have a way to change the default builder's config.

A workaround would be to use ue4-docker's "layout" mode to export the desired build as a Dockerfile, and then execute it using something like docker/buildx#484 (comment) (although that's going to create a new builder each time, so consider refining it if you go this way)

[slonopotamus]

I suspect that BUILDKIT_STEP_LOG_MAX_SIZE needs to be in env variables of docker daemon.

[slonopotamus]

To be honest, I completely forgot about this weird log thing when did #229.

[TravisBowers]

This issue was ultimately resolved by adding the following to /etc/systemd/system/docker.service.d/buildkit-environment.conf and restarting the daemon.

[Service]
Environment="DOCKER_BUILDKIT=1"
Environment="BUILDKIT_STEP_LOG_MAX_SIZE=10000000"
Environment="BUILDKIT_STEP_LOG_MAX_SPEED=10000000"