Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[prebuild][selfhosted] prebuilds freeze and does not fail nor finish. #8942

Closed
fredrikhgrelland opened this issue Mar 23, 2022 · 15 comments
Closed

[prebuild][selfhosted] prebuilds freeze and does not fail nor finish. #8942

fredrikhgrelland opened this issue Mar 23, 2022 · 15 comments
Labels
feature: prebuilds self-hosted: eks Self hosted support for AWS EKS self-hosted team: workspace Issue belongs to the Workspace team

Comments

@fredrikhgrelland
Copy link

fredrikhgrelland commented Mar 23, 2022 •
edited
Loading

Bug description

I am running a docker build as an init-task in a self-hosted gitpod on EKS.
This works fine every time when opening a new workspace.
I have given the pods 16gb ram and 4 cpus.

When starting a prebuild with with the magic-link https://gitpod.example.com/#prebuild/https://github/myrepo one of the following things will happen:

  • prebuild starts and says "Connecting to workspace logs..." forever
  • prebuild starts, shows log output and the log stops at a random process.

For both cases i can observe:

  • the prebuild pod is running and i can connect a terminal to it.
  • the init-task is apparently running with supervisor init
gitpod ~ $ ps -axf
    PID TTY      STAT   TIME COMMAND
   1136 pts/2    Ss     0:00 bash
   4287 pts/2    R+     0:00  \_ ps -axf
      1 ?        Ssl    0:00 /.supervisor/workspacekit ring0
     14 ?        Sl     0:08 /proc/self/exe ring1 --mapping-established
     28 ?        Sl     0:00  \_ supervisor init
     59 ?        Sl     0:08  |   \_ supervisor run
     78 pts/0    Ss     0:00  |       \_ /bin/bash
    402 pts/0    S+     0:00  |       |   \_ make build
    403 pts/0    S+     0:00  |       |       \_ /bin/bash -c cd aws-mwaa-local-runner && ./mwaa-local-env build-image docker image save amazon/mwaa-lo
    404 pts/0    S+     0:00  |       |           \_ /bin/bash ./mwaa-local-env build-image
    405 pts/0    Sl+    0:00  |       |               \_ docker build --rm --compress -t amazon/mwaa-local:2.2 ./docker
    417 pts/1    Ssl+   0:11  |       \_ /usr/bin/dockerd --experimental --rootless --data-root=/workspace/.docker-root --add-runtime gitpod=/usr/bin/r
    432 ?        Ssl    0:03  |           \_ containerd --config /var/run/docker/containerd/containerd.toml --log-level info
     50 ?        S      0:02  \_ /.supervisor/slirp4netns --configure --mtu=65520 --disable-host-loopback --api-socket /tmp/wskit-slirp4netns3587365082
  • the logfile tail -f /workspace/.gitpod/prebuild-log-1 seems to stop at a random place in the process. (the log below came from a prebuild that never showed any logs in the web-ui
{
cd .airflow; make build
}; exit
gitpod /workspace/aws-advh-dataplatform-gitpod (main) $ {
> cd .airflow; make build
> }; exit
cd aws-mwaa-local-runner && ./mwaa-local-env build-image
docker image save amazon/mwaa-local:2.2 -o /workspace/airflow.tar.gz
Sending build context to Docker daemon  23.76kB
Step 1/28 : FROM amazonlinux
latest: Pulling from library/amazonlinux
20e1cc2336fb: Pull complete 
Digest: sha256:b33b787cdb0e82495d2dc115745f68c7cd8d2585d9d83812fdc183ad39d1b753
Status: Downloaded newer image for amazonlinux:latest
 ---> 2aa63ff55c40
Step 2/28 : LABEL maintainer="amazon"
 ---> Running in 203f9df4ba92
Removing intermediate container 203f9df4ba92
 ---> c004d0c0733a
Step 3/28 : ARG AIRFLOW_VERSION=2.2.2
 ---> Running in 78149a2a1562
Removing intermediate container 78149a2a1562
 ---> 7d43f654d576
Step 4/28 : ARG WATCHTOWER_VERSION=1.0.6
 ---> Running in 31b0b1d870ab
Removing intermediate container 31b0b1d870ab
 ---> c00b0a4d6209
Step 5/28 : ARG PROVIDER_AMAZON_VERSION=2.4.0
 ---> Running in ad195792a0ad
Removing intermediate container ad195792a0ad
 ---> 9b068285af83
Step 6/28 : ARG AIRFLOW_USER_HOME=/usr/local/airflow
 ---> Running in 5ac5bb3631c8
Removing intermediate container 5ac5bb3631c8
 ---> 99c2493d01ff
Step 7/28 : ARG AIRFLOW_DEPS=""
 ---> Running in d66e0cfcc412
Removing intermediate container d66e0cfcc412
 ---> ea7d541231ac
Step 8/28 : ARG PYTHON_DEPS=""
 ---> Running in 2dcb6d4fa12d
Removing intermediate container 2dcb6d4fa12d
 ---> 16a5845cedff
Step 9/28 : ARG SYSTEM_DEPS=""
 ---> Running in f10c1be7d845
Removing intermediate container f10c1be7d845
 ---> 9c6973f31e44
Step 10/28 : ARG INDEX_URL=""
 ---> Running in 358c6ffd00b2
Removing intermediate container 358c6ffd00b2
 ---> e73746a0ae16
Step 11/28 : ENV AIRFLOW_HOME=${AIRFLOW_USER_HOME}
 ---> Running in bdfb343f7ade
Removing intermediate container bdfb343f7ade
 ---> c312340b4605
Step 12/28 : COPY script/bootstrap.sh /bootstrap.sh
 ---> fb6ccbf77656
Step 13/28 : COPY script/systemlibs.sh /systemlibs.sh
 ---> 6de8ffb20c70
Step 14/28 : COPY config/constraints.txt /constraints.txt
 ---> 284a68de72a6
Step 15/28 : COPY config/requirements.txt /requirements.txt
 ---> 23d67fa86e0f
Step 16/28 : COPY config/mwaa-base-providers-requirements.txt /mwaa-base-providers-requirements.txt
 ---> ee6975146786
Step 17/28 : RUN chmod u+x /systemlibs.sh && /systemlibs.sh
 ---> Running in 3175a226e13b
Loaded plugins: ovl, priorities
No packages marked for update
Loaded plugins: ovl, priorities
No package gcc-g++ available.
Resolving Dependencies
--> Running transaction check
---> Package gcc.x86_64 0:7.3.1-13.amzn2 will be installed
--> Processing Dependency: libgomp = 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64
--> Processing Dependency: cpp = 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64
--> Processing Dependency: libsanitizer >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64
--> Processing Dependency: libquadmath >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64
--> Processing Dependency: libmpx >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64
--> Processing Dependency: libitm >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64
--> Processing Dependency: libcilkrts >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64
--> Processing Dependency: libatomic >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64
--> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-7.3.1-13.amzn2.x86_64
--> Processing Dependency: binutils >= 2.24 for package: gcc-7.3.1-13.amzn2.x86_64
--> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-7.3.1-13.amzn2.x86_64
--> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-7.3.1-13.amzn2.x86_64
--> Processing Dependency: libgomp.so.1()(64bit) for package: gcc-7.3.1-13.amzn2.x86_64
---> Package python3.x86_64 0:3.7.10-1.amzn2.0.1 will be installed
--> Processing Dependency: python3-libs(x86-64) = 3.7.10-1.amzn2.0.1 for package: python3-3.7.10-1.amzn2.0.1.x86_64
--> Processing Dependency: python3-setuptools for package: python3-3.7.10-1.amzn2.0.1.x86_64
--> Processing Dependency: python3-pip for package: python3-3.7.10-1.amzn2.0.1.x86_64
--> Processing Dependency: libpython3.7m.so.1.0()(64bit) for package: python3-3.7.10-1.amzn2.0.1.x86_64
---> Package python3-devel.x86_64 0:3.7.10-1.amzn2.0.1 will be installed
--> Processing Dependency: system-rpm-config for package: python3-devel-3.7.10-1.amzn2.0.1.x86_64
--> Processing Dependency: python3-rpm-macros for package: python3-devel-3.7.10-1.amzn2.0.1.x86_64
--> Processing Dependency: /usr/bin/pkg-config for package: python3-devel-3.7.10-1.amzn2.0.1.x86_64
--> Running transaction check
---> Package binutils.x86_64 0:2.29.1-31.amzn2 will be installed
---> Package cpp.x86_64 0:7.3.1-13.amzn2 will be installed
---> Package glibc-devel.x86_64 0:2.26-57.amzn2 will be installed
--> Processing Dependency: glibc-headers = 2.26-57.amzn2 for package: glibc-devel-2.26-57.amzn2.x86_64
--> Processing Dependency: glibc-headers for package: glibc-devel-2.26-57.amzn2.x86_64
---> Package libatomic.x86_64 0:7.3.1-13.amzn2 will be installed
---> Package libcilkrts.x86_64 0:7.3.1-13.amzn2 will be installed
---> Package libgomp.x86_64 0:7.3.1-13.amzn2 will be installed
---> Package libitm.x86_64 0:7.3.1-13.amzn2 will be installed
---> Package libmpc.x86_64 0:1.0.1-3.amzn2.0.2 will be installed
---> Package libmpx.x86_64 0:7.3.1-13.amzn2 will be installed
---> Package libquadmath.x86_64 0:7.3.1-13.amzn2 will be installed
---> Package libsanitizer.x86_64 0:7.3.1-13.amzn2 will be installed
---> Package mpfr.x86_64 0:3.1.1-4.amzn2.0.2 will be installed
---> Package pkgconfig.x86_64 1:0.27.1-4.amzn2.0.2 will be installed
---> Package python3-libs.x86_64 0:3.7.10-1.amzn2.0.1 will be installed
--> Processing Dependency: libtirpc.so.1()(64bit) for package: python3-libs-3.7.10-1.amzn2.0.1.x86_64
---> Package python3-pip.noarch 0:20.2.2-1.amzn2.0.3 will be installed
---> Package python3-rpm-macros.noarch 0:3-60.amzn2.0.1 will be installed
--> Processing Dependency: python-srpm-macros >= 3-38 for package: python3-rpm-macros-3-60.amzn2.0.1.noarch
--> Processing Dependency: python-rpm-macros for package: python3-rpm-macros-3-60.amzn2.0.1.noarch
---> Package python3-setuptools.noarch 0:49.1.3-1.amzn2.0.2 will be installed
---> Package system-rpm-config.noarch 0:9.1.0-76.amzn2.0.10 will be installed
--> Processing Dependency: dwz >= 0.4 for package: system-rpm-config-9.1.0-76.amzn2.0.10.noarch
--> Processing Dependency: zip for package: system-rpm-config-9.1.0-76.amzn2.0.10.noarch
--> Processing Dependency: perl-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.10.noarch
--> Processing Dependency: perl(Getopt::Long) for package: system-rpm-config-9.1.0-76.amzn2.0.10.noarch
--> Processing Dependency: /usr/bin/perl for package: system-rpm-config-9.1.0-76.amzn2.0.10.noarch
--> Running transaction check
---> Package dwz.x86_64 0:0.11-3.amzn2.0.3 will be installed
---> Package glibc-headers.x86_64 0:2.26-57.amzn2 will be installed
--> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.26-57.amzn2.x86_64
--> Processing Dependency: kernel-headers for package: glibc-headers-2.26-57.amzn2.x86_64
---> Package libtirpc.x86_64 0:0.2.4-0.16.amzn2 will be installed
---> Package perl.x86_64 4:5.16.3-299.amzn2.0.1 will be installed
--> Processing Dependency: perl-libs = 4:5.16.3-299.amzn2.0.1 for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Socket) >= 1.3 for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Scalar::Util) >= 1.10 for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl-macros for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl-libs for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(threads::shared) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(threads) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(constant) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Time::Local) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Time::HiRes) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Storable) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Socket) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Scalar::Util) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Pod::Simple::XHTML) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Pod::Simple::Search) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Filter::Util::Call) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(File::Temp) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(File::Spec::Unix) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(File::Spec::Functions) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(File::Spec) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(File::Path) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Exporter) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Cwd) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Carp) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: libperl.so()(64bit) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
---> Package perl-Getopt-Long.noarch 0:2.40-3.amzn2 will be installed
--> Processing Dependency: perl(Pod::Usage) >= 1.14 for package: perl-Getopt-Long-2.40-3.amzn2.noarch
--> Processing Dependency: perl(Text::ParseWords) for package: perl-Getopt-Long-2.40-3.amzn2.noarch
---> Package perl-srpm-macros.noarch 0:1-8.amzn2.0.1 will be installed
---> Package python-rpm-macros.noarch 0:3-60.amzn2.0.1 will be installed
---> Package python-srpm-macros.noarch 0:3-60.amzn2.0.1 will be installed
---> Package zip.x86_64 0:3.0-11.amzn2.0.2 will be installed
--> Running transaction check
---> Package kernel-headers.x86_64 0:4.14.268-205.500.amzn2 will be installed
---> Package perl-Carp.noarch 0:1.26-244.amzn2 will be installed
---> Package perl-Exporter.noarch 0:5.68-3.amzn2 will be installed
---> Package perl-File-Path.noarch 0:2.09-2.amzn2 will be installed
---> Package perl-File-Temp.noarch 0:0.23.01-3.amzn2 will be installed
---> Package perl-Filter.x86_64 0:1.49-3.amzn2.0.2 will be installed
---> Package perl-PathTools.x86_64 0:3.40-5.amzn2.0.2 will be installed
---> Package perl-Pod-Simple.noarch 1:3.28-4.amzn2 will be installed
--> Processing Dependency: perl(Pod::Escapes) >= 1.04 for package: 1:perl-Pod-Simple-3.28-4.amzn2.noarch
--> Processing Dependency: perl(Encode) for package: 1:perl-Pod-Simple-3.28-4.amzn2.noarch
---> Package perl-Pod-Usage.noarch 0:1.63-3.amzn2 will be installed
--> Processing Dependency: perl(Pod::Text) >= 3.15 for package: perl-Pod-Usage-1.63-3.amzn2.noarch
--> Processing Dependency: perl-Pod-Perldoc for package: perl-Pod-Usage-1.63-3.amzn2.noarch
---> Package perl-Scalar-List-Utils.x86_64 0:1.27-248.amzn2.0.2 will be installed
---> Package perl-Socket.x86_64 0:2.010-4.amzn2.0.2 will be installed
---> Package perl-Storable.x86_64 0:2.45-3.amzn2.0.2 will be installed
---> Package perl-Text-ParseWords.noarch 0:3.29-4.amzn2 will be installed
---> Package perl-Time-HiRes.x86_64 4:1.9725-3.amzn2.0.2 will be installed
---> Package perl-Time-Local.noarch 0:1.2300-2.amzn2 will be installed
---> Package perl-constant.noarch 0:1.27-2.amzn2.0.1 will be installed
---> Package perl-libs.x86_64 4:5.16.3-299.amzn2.0.1 will be installed
---> Package perl-macros.x86_64 4:5.16.3-299.amzn2.0.1 will be installed
---> Package perl-threads.x86_64 0:1.87-4.amzn2.0.2 will be installed
---> Package perl-threads-shared.x86_64 0:1.43-6.amzn2.0.2 will be installed
--> Running transaction check
---> Package perl-Encode.x86_64 0:2.51-7.amzn2.0.2 will be installed
---> Package perl-Pod-Escapes.noarch 1:1.04-299.amzn2.0.1 will be installed
---> Package perl-Pod-Perldoc.noarch 0:3.20-4.amzn2 will be installed
--> Processing Dependency: perl(parent) for package: perl-Pod-Perldoc-3.20-4.amzn2.noarch
--> Processing Dependency: perl(HTTP::Tiny) for package: perl-Pod-Perldoc-3.20-4.amzn2.noarch
--> Processing Dependency: groff-base for package: perl-Pod-Perldoc-3.20-4.amzn2.noarch
---> Package perl-podlators.noarch 0:2.5.1-3.amzn2.0.1 will be installed
--> Running transaction check
---> Package groff-base.x86_64 0:1.22.2-8.amzn2.0.2 will be installed
---> Package perl-HTTP-Tiny.noarch 0:0.033-3.amzn2 will be installed
---> Package perl-parent.noarch 1:0.225-244.amzn2.0.1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                  Arch     Version                   Repository    Size
================================================================================
Installing:
 gcc                      x86_64   7.3.1-13.amzn2            amzn2-core    22 M
 python3                  x86_64   3.7.10-1.amzn2.0.1        amzn2-core    72 k
 python3-devel            x86_64   3.7.10-1.amzn2.0.1        amzn2-core   243 k
Installing for dependencies:
 binutils                 x86_64   2.29.1-31.amzn2           amzn2-core   5.8 M
 cpp                      x86_64   7.3.1-13.amzn2            amzn2-core   9.2 M
 dwz                      x86_64   0.11-3.amzn2.0.3          amzn2-core    98 k
 glibc-devel              x86_64   2.26-57.amzn2             amzn2-core   994 k
 glibc-headers            x86_64   2.26-57.amzn2             amzn2-core   514 k
 groff-base               x86_64   1.22.2-8.amzn2.0.2        amzn2-core   948 k
 kernel-headers           x86_64   4.14.268-205.500.amzn2    amzn2-core   1.2 M
 libatomic                x86_64   7.3.1-13.amzn2            amzn2-core    46 k
 libcilkrts               x86_64   7.3.1-13.amzn2            amzn2-core    85 k
 libgomp                  x86_64   7.3.1-13.amzn2            amzn2-core   204 k
 libitm                   x86_64   7.3.1-13.amzn2            amzn2-core    84 k
 libmpc                   x86_64   1.0.1-3.amzn2.0.2         amzn2-core    52 k
 libmpx                   x86_64   7.3.1-13.amzn2            amzn2-core    51 k
 libquadmath              x86_64   7.3.1-13.amzn2            amzn2-core   189 k
 libsanitizer             x86_64   7.3.1-13.amzn2            amzn2-core   641 k
 libtirpc                 x86_64   0.2.4-0.16.amzn2          amzn2-core    88 k
 mpfr                     x86_64   3.1.1-4.amzn2.0.2         amzn2-core   208 k
 perl                     x86_64   4:5.16.3-299.amzn2.0.1    amzn2-core   8.0 M
 perl-Carp                noarch   1.26-244.amzn2            amzn2-core    19 k
 perl-Encode              x86_64   2.51-7.amzn2.0.2          amzn2-core   1.5 M
 perl-Exporter            noarch   5.68-3.amzn2              amzn2-core    29 k
 perl-File-Path           noarch   2.09-2.amzn2              amzn2-core    27 k
 perl-File-Temp           noarch   0.23.01-3.amzn2           amzn2-core    56 k
 perl-Filter              x86_64   1.49-3.amzn2.0.2          amzn2-core    76 k
 perl-Getopt-Long         noarch   2.40-3.amzn2              amzn2-core    56 k
 perl-HTTP-Tiny           noarch   0.033-3.amzn2             amzn2-core    38 k
 perl-PathTools           x86_64   3.40-5.amzn2.0.2          amzn2-core    83 k
 perl-Pod-Escapes         noarch   1:1.04-299.amzn2.0.1      amzn2-core    52 k
 perl-Pod-Perldoc         noarch   3.20-4.amzn2              amzn2-core    87 k
 perl-Pod-Simple          noarch   1:3.28-4.amzn2            amzn2-core   216 k
 perl-Pod-Usage           noarch   1.63-3.amzn2              amzn2-core    27 k
 perl-Scalar-List-Utils   x86_64   1.27-248.amzn2.0.2        amzn2-core    36 k
 perl-Socket              x86_64   2.010-4.amzn2.0.2         amzn2-core    49 k
 perl-Storable            x86_64   2.45-3.amzn2.0.2          amzn2-core    78 k
 perl-Text-ParseWords     noarch   3.29-4.amzn2              amzn2-core    14 k
 perl-Time-HiRes          x86_64   4:1.9725-3.amzn2.0.2      amzn2-core    45 k
 perl-Time-Local          noarch   1.2300-2.amzn2            amzn2-core    24 k
 perl-constant            noarch   1.27-2.amzn2.0.1          amzn2-core    19 k
 perl-libs                x86_64   4:5.16.3-299.amzn2.0.1    amzn2-core   685 k
 perl-macros              x86_64   4:5.16.3-299.amzn2.0.1    amzn2-core    44 k
 perl-parent              noarch   1:0.225-244.amzn2.0.1     amzn2-core    12 k
 perl-podlators           noarch   2.5.1-3.amzn2.0.1         amzn2-core   112 k
 perl-srpm-macros         noarch   1-8.amzn2.0.1             amzn2-core   4.7 k
 perl-threads             x86_64   1.87-4.amzn2.0.2          amzn2-core    50 k
 perl-threads-shared      x86_64   1.43-6.amzn2.0.2          amzn2-core    39 k
 pkgconfig                x86_64   1:0.27.1-4.amzn2.0.2      amzn2-core    54 k
 python-rpm-macros        noarch   3-60.amzn2.0.1            amzn2-core    14 k
 python-srpm-macros       noarch   3-60.amzn2.0.1            amzn2-core    18 k
 python3-libs             x86_64   3.7.10-1.amzn2.0.1        amzn2-core   9.2 M
 python3-pip              noarch   20.2.2-1.amzn2.0.3        amzn2-core   2.0 M
 python3-rpm-macros       noarch   3-60.amzn2.0.1            amzn2-core    12 k
 python3-setuptools       noarch   49.1.3-1.amzn2.0.2        amzn2-core   1.1 M
 system-rpm-config        noarch   9.1.0-76.amzn2.0.10       amzn2-core    80 k
 zip                      x86_64   3.0-11.amzn2.0.2          amzn2-core   263 k

Transaction Summary
================================================================================
Install  3 Packages (+54 Dependent packages)

Total download size: 67 M
Installed size: 213 M
Downloading packages:
--------------------------------------------------------------------------------
Total                                               14 MB/s |  67 MB  00:04     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : mpfr-3.1.1-4.amzn2.0.2.x86_64                               1/57 
  Installing : libmpc-1.0.1-3.amzn2.0.2.x86_64                             2/57 
  Installing : cpp-7.3.1-13.amzn2.x86_64                                   3/57 
  Installing : libmpx-7.3.1-13.amzn2.x86_64                                4/57 
  Installing : groff-base-1.22.2-8.amzn2.0.2.x86_64                        5/57 
  Installing : perl-HTTP-Tiny-0.033-3.amzn2.noarch                         6/57 
  Installing : 1:perl-parent-0.225-244.amzn2.0.1.noarch                    7/57 
  Installing : perl-Pod-Perldoc-3.20-4.amzn2.noarch                        8/57 
  Installing : perl-podlators-2.5.1-3.amzn2.0.1.noarch                     9/57 
  Installing : 1:perl-Pod-Escapes-1.04-299.amzn2.0.1.noarch               10/57 
  Installing : perl-Text-ParseWords-3.29-4.amzn2.noarch                   11/57 
  Installing : perl-Encode-2.51-7.amzn2.0.2.x86_64                        12/57 
  Installing : perl-Pod-Usage-1.63-3.amzn2.noarch                         13/57 
  Installing : 4:perl-macros-5.16.3-299.amzn2.0.1.x86_64                  14/57 
  Installing : 4:perl-libs-5.16.3-299.amzn2.0.1.x86_64                    15/57 
  Installing : perl-Exporter-5.68-3.amzn2.noarch                          16/57 
  Installing : perl-Time-Local-1.2300-2.amzn2.noarch                      17/57 
  Installing : perl-constant-1.27-2.amzn2.0.1.noarch                      18/57 
  Installing : perl-Scalar-List-Utils-1.27-248.amzn2.0.2.x86_64           19/57 
  Installing : perl-PathTools-3.40-5.amzn2.0.2.x86_64                     20/57 
  Installing : 4:perl-Time-HiRes-1.9725-3.amzn2.0.2.x86_64                21/57 
  Installing : perl-Socket-2.010-4.amzn2.0.2.x86_64                       22/57 
  Installing : perl-Storable-2.45-3.amzn2.0.2.x86_64                      23/57 
  Installing : perl-Carp-1.26-244.amzn2.noarch                            24/57 
  Installing : perl-File-Temp-0.23.01-3.amzn2.noarch                      25/57 
  Installing : perl-File-Path-2.09-2.amzn2.noarch                         26/57 
  Installing : 1:perl-Pod-Simple-3.28-4.amzn2.noarch                      27/57 
  Installing : perl-Getopt-Long-2.40-3.amzn2.noarch                       28/57 
  Installing : perl-threads-shared-1.43-6.amzn2.0.2.x86_64                29/57 
  Installing : perl-threads-1.87-4.amzn2.0.2.x86_64                       30/57 
  Installing : perl-Filter-1.49-3.amzn2.0.2.x86_64                        31/57 
  Installing : 4:perl-5.16.3-299.amzn2.0.1.x86_64                         32/57 
  Installing : zip-3.0-11.amzn2.0.2.x86_64                                33/57 
  Installing : dwz-0.11-3.amzn2.0.3.x86_64                                34/57 
  Installing : libatomic-7.3.1-13.amzn2.x86_64                            35/57 
  Installing : 1:pkgconfig-0.27.1-4.amzn2.0.2.x86_64                      36/57 
  Installing : libcilkrts-7.3.1-13.amzn2.x86_64                           37/57 
  Installing : kernel-headers-4.14.268-205.500.amzn2.x86_64               38/57 
  Installing : glibc-headers-2.26-57.amzn2.x86_64                         39/57 
  Installing : glibc-devel-2.26-57.amzn2.x86_64                           40/57 
  Installing : perl-srpm-macros-1-8.amzn2.0.1.noarch                      41/57 
  Installing : system-rpm-config-9.1.0-76.amzn2.0.10.noarch               42/57 
  Installing : python-srpm-macros-3-60.amzn2.0.1.noarch                   43/57 
  Installing : python-rpm-macros-3-60.amzn2.0.1.noarch                    44/57 
  Installing : python3-rpm-macros-3-60.amzn2.0.1.noarch                   45/57 
  Installing : libitm-7.3.1-13.amzn2.x86_64                               46/57 
  Installing : libtirpc-0.2.4-0.16.amzn2.x86_64                           47/57 
  Installing : python3-setuptools-49.1.3-1.amzn2.0.2.noarch               48/57 
  Installing : python3-libs-3.7.10-1.amzn2.0.1.x86_64                     49/57 
  Installing : python3-3.7.10-1.amzn2.0.1.x86_64                          50/57 
  Installing : python3-pip-20.2.2-1.amzn2.0.3.noarch                      51/57 
  Installing : libgomp-7.3.1-13.amzn2.x86_64                              52/57 
  Installing : binutils-2.29.1-31.amzn2.x86_64                            53/57 
  Installing : libquadmath-7.3.1-13.amzn2.x86_64                          54/57
  • the last few log-lines of the pod logs are:
022-03-23T19:13:10.403583732Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-03-23T19:13:10.403375610Z] Layer sha256:ef0c7ae227b2ca9d656b7472fdf9625d0b4fb0d710446b3910689bc0b7e05926 cleaned up \r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:13:10Z"}
2022-03-23T19:13:11.424728801Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-03-23T19:13:11.424525877Z] Layer sha256:ef0c7ae227b2ca9d656b7472fdf9625d0b4fb0d710446b3910689bc0b7e05926 cleaned up \r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:13:11Z"}
2022-03-23T19:13:12.397423269Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-03-23T19:13:12.397243466Z] Layer sha256:ef0c7ae227b2ca9d656b7472fdf9625d0b4fb0d710446b3910689bc0b7e05926 cleaned up \r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:13:12Z"}
2022-03-23T19:13:13.401560619Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-03-23T19:13:13.401387801Z] Layer sha256:ef0c7ae227b2ca9d656b7472fdf9625d0b4fb0d710446b3910689bc0b7e05926 cleaned up \r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:13:13Z"}
2022-03-23T19:13:19.100783313Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"time=\"2022-03-23T19:13:19.100289175Z\" level=info msg=\"starting signal loop\" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/3175a226e13b96aca3666c1ff5da1fa89f0ff50eb375ab280dae8398876dc41f pid=774\r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:13:19Z"}
2022-03-23T19:13:19.273519835Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"\u001b[31mERRO\u001b[0m[0000] failed to disable IPv6 forwarding for container's interface all: open /proc/sys/net/ipv6/conf/all/disable_ipv6: read-only file system \r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:13:19Z"}
2022-03-23T19:13:19.277003322Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"\u001b[33mWARN\u001b[0m[2022-03-23T19:13:19.276839148Z] Failed to disable IPv6 on all interfaces on network namespace \"/var/run/docker/netns/0d57e6164179\": reexec to set IPv6 failed: exit status 4 \r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:13:19Z"}
2022-03-23T19:13:19.388745822Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"time=\"2022-03-23T19:13:19.388494215Z\" level=error msg=\"add cg to OOM monitor\" error=\"cgroups: memory cgroup not supported on this system\"\r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:13:19Z"}
2022-03-23T19:14:31.393949730Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-03-23T19:14:31.393663633Z] ignoring event                                \u001b[36mcontainer\u001b[0m=3175a226e13b96aca3666c1ff5da1fa89f0ff50eb375ab280dae8398876dc41f \u001b[36mmodule\u001b[0m=libcontainerd \u001b[36mnamespace\u001b[0m=moby \u001b[36mtopic\u001b[0m=/tasks/delete \u001b[36mtype\u001b[0m=\"*events.TaskDelete\"","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:14:31Z"}
2022-03-23T19:14:31.393993839Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"\r\n\u001b[36mINFO\u001b[0m[2022-03-23T19:14:31.393658438Z] shim disconnected                             \u001b[36mid\u001b[0m=3175a226e13b96aca3666c1ff5da1fa89f0ff50eb375ab280dae8398876dc41f\r\n\u001b[31mERRO\u001b[0m[2022-03-23T19:14:31.393791565Z] copy shim log                                 \u001b[31merror\u001b[0m=\"read /proc/self/fd/16: file already closed\"\r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:14:31Z"}

Steps to reproduce

Running a prebuild building a docker image with significant load?

Workspace affected

No response

Expected behavior

Prebuilds should fail or succed. Now stuck in limbo.

Example repository

No response

Anything else?

No response
@axonasif
Copy link
Member

Hey @fredrikhgrelland, is repo public? It would help to reproduce. (maybe one of your dockerfile commands requires user input?)
Prebuilds can not run longer than 1h, not sure if that's the case.

@fredrikhgrelland
Copy link
Author

fredrikhgrelland commented Mar 27, 2022 •
edited
Loading

@axonasif I made a reproducible repo public here: https://github.com/fredrikhgrelland/gitpod-public-sandbox.

Case:

@Pothulapati Pothulapati self-assigned this Apr 6, 2022
@Pothulapati
Copy link
Contributor

Hey @fredrikhgrelland, Thanks for the detailed issue, repro cases. Definitely helps make debugging easier. :)

So, I have a self-hosted instance running at tarun.gitpod-self-hosted.com and for me the prebuild seems to work fine and ran to completion just like gitpod-io, and you can find the logs here. I'm still finding similar error logs in the prebuild container though i.e

{"label":"4ea4451c-88f9-4d6b-aae4-b5efbff7f576","level":"info","message":"\u001b[31mERRO\u001b[0m[0000] failed to disable IPv6 forwarding for container's interface all: open /proc/sys/net/ipv6/conf/all/disable_ipv6: read-only file system \r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T06:27:23Z"}
{"label":"4ea4451c-88f9-4d6b-aae4-b5efbff7f576","level":"info","message":"\u001b[33mWARN\u001b[0m[2022-04-06T06:27:23.988344674Z] Failed to disable IPv6 on all interfaces on network namespace \"/var/run/docker/netns/05084a1153e9\": reexec to set IPv6 failed: exit status 4 \r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T06:27:23Z"}
{"label":"4ea4451c-88f9-4d6b-aae4-b5efbff7f576","level":"info","message":"time=\"2022-04-06T06:27:24.082745291Z\" level=error msg=\"add cg to OOM monitor\" error=\"open /sys/fs/cgroup/memory/cgroup.event_control: read-only file system\"\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T06:27:24Z"}
{"label":"4ea4451c-88f9-4d6b-aae4-b5efbff7f576","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-04-06T06:27:24.107962721Z] ignoring event                                \u001b[36mcontainer\u001b[0m=1956b3e5dbfb6d29e421c554a331e8dc341958b7b976e97ed53aefbd61560ffd \u001b[36mmodule\u001b[0m=libcontainerd \u001b[36mnamespace\u001b[0m=moby \u001b[36mtopic\u001b[0m=/tasks/delete \u001b[36mtype\u001b[0m=\"*events.TaskDelete\"","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T06:27:24Z"}
{"label":"4ea4451c-88f9-4d6b-aae4-b5efbff7f576","level":"info","message":"\r\n\u001b[36mINFO\u001b[0m[2022-04-06T06:27:24.108178124Z] shim disconnected                             \u001b[36mid\u001b[0m=1956b3e5dbfb6d29e421c554a331e8dc341958b7b976e97ed53aefbd61560ffd\r\n\u001b[31mERRO\u001b[0m[2022-04-06T06:27:24.108283005Z] copy shim log                                 \u001b[31merror\u001b[0m=\"read /proc/self/fd/16: file already closed\"\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T06:27:24Z"}
{"label":"4ea4451c-88f9-4d6b-aae4-b5efbff7f576","level":"info","message":"time=\"2022-04-06T06:27:25.896634250Z\" level=info msg=\"starting signal loop\" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/5d5aa37079ecb996943be96a83de875668f514a5957eee1233f6293e2da219b8 pid=1840\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T06:27:25Z"}

Debugging more on this, and will update on this once I find more!

but the prebuild should still run to completion (just like in my case) and I'm wondering why that is not the case in your environment 🤔

@fredrikhgrelland
Copy link
Author

fredrikhgrelland commented Apr 6, 2022 •
edited
Loading

Yeah, it is bit of a head-scratcher.
I have replicated this on a brand new cluster using the latest version of gitpod and I am still seeing it freeze-fail.
It is not failing, but freezing up, and that is part of the problem.

Are you seeing these kind of logs as well:
{"label":"9f1d28bc-8ffb-4f45-b286-7abbf0469cc0","level":"info","message":"time=\"2022-04-06T10:03:48.407286565Z\" level=error msg=\"add cg to OOM monitor\" error=\"open /sys/fs/cgroup/memory/cgroup.event_control: read-only file system\"\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T10:03:48Z"}
It stops with this log-line:
{"label":"9f1d28bc-8ffb-4f45-b286-7abbf0469cc0","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-04-06T10:05:00.319048376Z] ignoring event \u001b[36mcontainer\u001b[0m=c52bcf11adce366c81109ebec0dd73b3a3cb7a02fab41f38a57888bb889509a9 \u001b[36mmodule\u001b[0m=libcontainerd \u001b[36mnamespace\u001b[0m=moby \u001b[36mtopic\u001b[0m=/tasks/delete \u001b[36mtype\u001b[0m=\"*events.TaskDelete\"\r\n\u001b[31mERRO\u001b[0m[2022-04-06T10:05:00.319114925Z] copy shim log \u001b[31merror\u001b[0m=\"read /proc/self/fd/16: file already closed\"\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T10:05:00Z"}

@fredrikhgrelland
Copy link
Author

I have run some more tests and I have maybe found something by looking at the prebuild-pod logs in real-time.
The prebuild continues to run fine after these logs:

2022-04-06T11:22:39.097122703Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"time=\"2022-04-06T11:22:39.096690965Z\" level=info msg=\"starting signal loop\" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/cf2ce27e01a7732f763e28fb3af68072da6e3b763f45cdd7177beb437ae5a89d pid=966\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:22:39Z"}
2022-04-06T11:22:39.251712049Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"\u001b[31mERRO\u001b[0m[0000] failed to disable IPv6 forwarding for container's interface all: open /proc/sys/net/ipv6/conf/all/disable_ipv6: read-only file system \r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:22:39Z"}
2022-04-06T11:22:39.254483194Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"\u001b[33mWARN\u001b[0m[2022-04-06T11:22:39.254360210Z] Failed to disable IPv6 on all interfaces on network namespace \"/var/run/docker/netns/04aa21c0f7c2\": reexec to set IPv6 failed: exit status 4 \r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:22:39Z"}
2022-04-06T11:22:39.384084749Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"time=\"2022-04-06T11:22:39.383842402Z\" level=error msg=\"add cg to OOM monitor\" error=\"open /sys/fs/cgroup/memory/cgroup.event_control: read-only file system\"\r\n","serviceContext":

While logs are streaming running for ~25 seconds:

{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:22:39Z"}
2022-04-06T11:23:21.397757620Z {"alias":"1dcdd82d-03d0-4184-928d-8c1496251d31","level":"info","message":"new terminal client","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","time":"2022-04-06T11:23:21Z"}
2022-04-06T11:23:23.412642462Z {"alias":"1dcdd82d-03d0-4184-928d-8c1496251d31","level":"info","message":"new terminal client","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","time":"2022-04-06T11:23:23Z"}

Log-streaming freeze and this pops up in the logs:

2022-04-06T11:23:49.278962037Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-04-06T11:23:49.278657618Z] ignoring event                                \u001b[36mcontainer\u001b[0m=cf2ce27e01a7732f763e28fb3af68072da6e3b763f45cdd7177beb437ae5a89d \u001b[36mmodule\u001b[0m=libcontainerd \u001b[36mnamespace\u001b[0m=moby \u001b[36mtopic\u001b[0m=/tasks/delete \u001b[36mtype\u001b[0m=\"*events.TaskDelete\"\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:23:49Z"}
2022-04-06T11:23:49.279022911Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-04-06T11:23:49.278741941Z] shim disconnected                             \u001b[36mid\u001b[0m=cf2ce27e01a7732f763e28fb3af68072da6e3b763f45cdd7177beb437ae5a89d\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:23:49Z"}
2022-04-06T11:23:49.279031115Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"\u001b[31mERRO\u001b[0m[2022-04-06T11:23:49.278834472Z] copy shim log                                 \u001b[31merror\u001b[0m=\"read /proc/self/fd/16: file already closed\"\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:23:49Z"}
2022-04-06T11:23:57.144430694Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"time=\"2022-04-06T11:23:57.144019839Z\" level=info msg=\"starting signal loop\" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/ce8dc69ef9faacc392296cd07bed61be5977a0db254d4a8494d61136a979b0b9 pid=1453\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:23:57Z"}
2022-04-06T11:23:57.295293487Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"\u001b[31mERRO\u001b[0m[0000] failed to disable IPv6 forwarding for container's interface all: open /proc/sys/net/ipv6/conf/all/disable_ipv6: read-only file system \r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:23:57Z"}
2022-04-06T11:23:57.298205229Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"\u001b[33mWARN\u001b[0m[2022-04-06T11:23:57.298060250Z] Failed to disable IPv6 on all interfaces on network namespace \"/var/run/docker/netns/546576152c18\": reexec to set IPv6 failed: exit status 4 \r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:23:57Z"}
2022-04-06T11:23:57.431655965Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"time=\"2022-04-06T11:23:57.431393664Z\" level=error msg=\"add cg to OOM monitor\" error=\"open /sys/fs/cgroup/memory/cgroup.event_control: read-only file system\"\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:23:57Z"}

The prebuild will now hang for ~1 hour and then continue to a image-build stage until starting the workspace, the prebuild clearly not done.
To speed this up, I might kill the pod and repeat.

NOTE: Normally it gets in a freeze-failing state far earlier and before the web-app starts printing logs.

@fredrikhgrelland
Copy link
Author

I have some more observations that might help further:

The webapp log-streaming as well as attaching to the prebuild pod tailing from /workspace/.gitpod/prebuild-log-1 are equal.
While observing running processes inside of the docker-build of the prebuildpod using ps -axf and watching /workspace/.gitpod/prebuild-log-1 I can clearly see that the docker-build is continuing well after the logs have stopped streaming.

@corneliusludmann
Copy link
Contributor

  • prebuild starts and says "Connecting to workspace logs..." forever
  • prebuild starts, shows log output and the log stops at a random process.

That is a UX issue. I see it from time to time as well: Prebuilds are running but the logs are not shown in the UI. Reloading the window helps for me usually. I think it's issue #8684, correct me when I'm wrong, @geropl.

The prebuild will now hang for ~1 hour and then continue to a image-build stage until starting the workspace, the prebuild clearly not done.

For prebuilds, we have a timeout of 1 hour. How long does the build usually take?

@geropl
Copy link
Member

geropl commented Apr 8, 2022

think it's issue #8684, correct me when I'm wrong, @geropl.

Exactly. #8684 does overlap with other issues already merged (but not deployed yet), so: we're working on this part.

@fredrikhgrelland Thanks for your very detailed analysis! 🙏 As far as I understand, especially after reading this comment of yours, this is a supervisor/workspace problem:

Log-streaming freeze and this pops up in the logs:

ignoring event
shim disconnected
copy shim log

@kylos101 This sounds like a runtime issue. Can someone from workspace have a look at this? @sagor999 maybe? 🤔

@kylos101
Copy link
Contributor

kylos101 commented Apr 8, 2022

Hi @fredrikhgrelland 👋 ,

Can you tell if you're hitting this line?

If yes, may I ask you to confirm from a regular workspace, what version of docker-compose version that you have?

I ask because about a month ago we updated the version of docker inworkspace-full, but I'm not sure what's being used with your installation. These issues were fixed in docker-compose 2.2.3, but, if you're using 2.2.2, you might be bumping into these issues. [1][2]

@kylos101 kylos101 moved this to In Progress in 🌌 Workspace Team Apr 8, 2022
@kylos101 kylos101 self-assigned this Apr 8, 2022
@kylos101 kylos101 removed the priority: highest (user impact) Directly user impacting label Apr 12, 2022
@kylos101 kylos101 removed their assignment Apr 14, 2022
@kylos101 kylos101 moved this from In Progress to Scheduled in 🌌 Workspace Team Apr 14, 2022
@fredrikhgrelland
Copy link
Author

Hi @kylos101,

Can you tell if you're hitting this line?

Unless you are running the command steps during a prebuild, I am not. See this line

If yes, may I ask you to confirm from a regular workspace, what version of docker-compose version that you have?

I ask because about a month ago we updated the version of docker inworkspace-full, but I'm not sure what's being used with your installation. These issues were fixed in docker-compose 2.2.3, but, if you're using 2.2.2, you might be bumping into these issues. [1][2]

gitpod /workspace/gitpod-public-sandbox/.airflow (isolate-docker-compose) $ docker-compose version
Docker Compose version v2.2.3

@fredrikhgrelland
Copy link
Author

@geropl

Is there any more steps I can take in order to debug a potential supervisor/workspace-issue?
Before easter I was chasing a containerd isse that looked like it fit the bill, but am currently a bit stuck. containerd/containerd#6402

@geropl geropl added the self-hosted: eks Self hosted support for AWS EKS label Apr 19, 2022
@geropl
Copy link
Member

geropl commented Apr 19, 2022

@fredrikhgrelland What version of containerd are you on? 🤔

@fredrikhgrelland
Copy link
Author

@geropl I am running containerd github.com/containerd/containerd v1.5.8 1e5ef943eb76627a6d3b6de8cd1ef6537f393a71
This comes from the ami provided in the eks-guide https://github.com/gitpod-io/gitpod-eks-guide.
My plan was to update the build with a newer binary, but I am currently unsuccessful to build an ami with the code here: https://github.com/gitpod-io/gitpod-eks-guide/tree/main/ami because of missing files. I have tried to pillage files from the running amis, but gitpod seems to fail getting up and running on a on a cluster built on my ami from source.

@lucasvaltl lucasvaltl mentioned this issue May 31, 2022
Closed
11 tasks
@kylos101
Copy link
Contributor

kylos101 commented Jun 8, 2022

Hey @fredrikhgrelland , are you still having trouble with prebuilds freezing?

CC: @metcalfc

@kylos101
Copy link
Contributor

kylos101 commented Jun 8, 2022

Hi @fredrikhgrelland , I am going to close this for now, but, if it's still a problem, let us know and we'll happily reopen. 🙏

CC: @metcalfc 👀

@kylos101 kylos101 closed this as completed Jun 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature: prebuilds self-hosted: eks Self hosted support for AWS EKS self-hosted team: workspace Issue belongs to the Workspace team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@kylos101 @Pothulapati @corneliusludmann @geropl @axonasif @fredrikhgrelland