Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CUDA CI jobs are broken in master #4610

Closed
StrikerRUS opened this issue Sep 17, 2021 · 2 comments
Closed

CUDA CI jobs are broken in master #4610

StrikerRUS opened this issue Sep 17, 2021 · 2 comments
Assignees
@shiyu1994
Labels
maintenance

Comments

@StrikerRUS
Copy link
Collaborator

Initially reported in #4606 (comment).

docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown.
Full logs: 
2021-09-15T23:12:00.6215072Z Can't find any online and idle self-hosted runner in the current repository, account/organization and enterprise that matches the required labels: 'self-hosted , linux'
2021-09-15T23:12:00.7256411Z Waiting for a self-hosted runner to pickup this job...
2021-09-15T23:12:47.4157030Z Job is about to start running on the runner: nv6-01 (repository)
2021-09-15T23:12:52.0695982Z Current runner version: '2.282.0'
2021-09-15T23:12:52.0700468Z Runner name: 'nv6-01'
2021-09-15T23:12:52.0701070Z Runner group name: 'Default'
2021-09-15T23:12:52.0702313Z Machine name: 'nv6-01'
2021-09-15T23:12:52.0705689Z ##[group]GITHUB_TOKEN Permissions
2021-09-15T23:12:52.0706967Z Actions: write
2021-09-15T23:12:52.0707553Z Checks: write
2021-09-15T23:12:52.0708038Z Contents: write
2021-09-15T23:12:52.0708549Z Deployments: write
2021-09-15T23:12:52.0709026Z Discussions: write
2021-09-15T23:12:52.0709541Z Issues: write
2021-09-15T23:12:52.0710017Z Metadata: read
2021-09-15T23:12:52.0710512Z Packages: write
2021-09-15T23:12:52.0710989Z PullRequests: write
2021-09-15T23:12:52.0711583Z RepositoryProjects: write
2021-09-15T23:12:52.0712180Z SecurityEvents: write
2021-09-15T23:12:52.0712745Z Statuses: write
2021-09-15T23:12:52.0713361Z ##[endgroup]
2021-09-15T23:12:52.0716281Z Prepare workflow directory
2021-09-15T23:12:52.1453868Z Prepare all required actions
2021-09-15T23:12:52.1463663Z Getting action download info
2021-09-15T23:12:52.4152015Z Download action repository 'actions/checkout@v1' (SHA:50fbc622fc4ef5163becd7fab6573eac35f8462e)
2021-09-15T23:12:52.8677653Z ##[group]Run sudo apt-get update
2021-09-15T23:12:52.8678549Z �[36;1msudo apt-get update�[0m
2021-09-15T23:12:52.8679176Z �[36;1msudo apt-get install --no-install-recommends -y \�[0m
2021-09-15T23:12:52.8679846Z �[36;1m    apt-transport-https \�[0m
2021-09-15T23:12:52.8680400Z �[36;1m    ca-certificates \�[0m
2021-09-15T23:12:52.8680840Z �[36;1m    curl \�[0m
2021-09-15T23:12:52.8681232Z �[36;1m    git \�[0m
2021-09-15T23:12:52.8681647Z �[36;1m    gnupg-agent \�[0m
2021-09-15T23:12:52.8682251Z �[36;1m    software-properties-common�[0m
2021-09-15T23:12:52.8683135Z �[36;1mcurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -�[0m
2021-09-15T23:12:52.8684271Z �[36;1msudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" -y�[0m
2021-09-15T23:12:52.8685440Z �[36;1mcurl -sL https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -�[0m
2021-09-15T23:12:52.8686789Z �[36;1mcurl -sL https://nvidia.github.io/nvidia-docker/$(. /etc/os-release;echo $ID$VERSION_ID)/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list�[0m
2021-09-15T23:12:52.8687869Z �[36;1msudo apt-get update�[0m
2021-09-15T23:12:52.8688484Z �[36;1msudo apt-get install --no-install-recommends -y \�[0m
2021-09-15T23:12:52.8689106Z �[36;1m    containerd.io \�[0m
2021-09-15T23:12:52.8689561Z �[36;1m    docker-ce \�[0m
2021-09-15T23:12:52.8690033Z �[36;1m    docker-ce-cli \�[0m
2021-09-15T23:12:52.8690531Z �[36;1m    nvidia-docker2�[0m
2021-09-15T23:12:52.8691062Z �[36;1msudo chmod a+rw /var/run/docker.sock�[0m
2021-09-15T23:12:52.8691625Z �[36;1msudo systemctl restart docker�[0m
2021-09-15T23:12:52.8713272Z shell: /bin/bash -e {0}
2021-09-15T23:12:52.8713839Z env:
2021-09-15T23:12:52.8714263Z   github_actions: true
2021-09-15T23:12:52.8714687Z   os_name: linux
2021-09-15T23:12:52.8715092Z   task: cuda
2021-09-15T23:12:52.8715517Z   conda_env: test-env
2021-09-15T23:12:52.8715964Z ##[endgroup]
2021-09-15T23:12:52.9693936Z Hit:1 http://azure.archive.ubuntu.com/ubuntu bionic InRelease
2021-09-15T23:12:52.9695849Z Get:2 http://azure.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
2021-09-15T23:12:52.9697851Z Get:3 http://azure.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
2021-09-15T23:12:52.9912762Z Get:4 https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64  InRelease [1484 B]
2021-09-15T23:12:52.9917711Z Hit:5 https://download.docker.com/linux/ubuntu bionic InRelease
2021-09-15T23:12:52.9970738Z Get:6 https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/amd64  InRelease [1481 B]
2021-09-15T23:12:53.0093847Z Hit:7 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64  InRelease
2021-09-15T23:12:53.0317450Z Hit:8 http://packages.microsoft.com/repos/azurecore bionic InRelease
2021-09-15T23:12:53.1211953Z Hit:9 http://security.ubuntu.com/ubuntu bionic-security InRelease
2021-09-15T23:12:53.7836628Z Fetched 166 kB in 1s (230 kB/s)
2021-09-15T23:12:54.7220539Z Reading package lists...
2021-09-15T23:12:54.8237085Z Reading package lists...
2021-09-15T23:12:54.9680864Z Building dependency tree...
2021-09-15T23:12:54.9685268Z Reading state information...
2021-09-15T23:12:55.0854263Z ca-certificates is already the newest version (20210119~18.04.1).
2021-09-15T23:12:55.0856865Z curl is already the newest version (7.58.0-2ubuntu3.15).
2021-09-15T23:12:55.0858003Z git is already the newest version (1:2.17.1-1ubuntu0.9).
2021-09-15T23:12:55.0859277Z software-properties-common is already the newest version (0.96.24.32.14).
2021-09-15T23:12:55.0861009Z apt-transport-https is already the newest version (1.6.14).
2021-09-15T23:12:55.0862242Z gnupg-agent is already the newest version (2.2.4-1ubuntu1.4).
2021-09-15T23:12:55.0863256Z The following packages were automatically installed and are no longer required:
2021-09-15T23:12:55.0864652Z   linux-azure-5.4-cloud-tools-5.4.0-1031
2021-09-15T23:12:55.0865956Z   linux-azure-5.4-cloud-tools-5.4.0-1032
2021-09-15T23:12:55.0867645Z   linux-azure-5.4-cloud-tools-5.4.0-1034
2021-09-15T23:12:55.0868834Z   linux-azure-5.4-cloud-tools-5.4.0-1035
2021-09-15T23:12:55.0869926Z   linux-azure-5.4-cloud-tools-5.4.0-1036
2021-09-15T23:12:55.0871050Z   linux-azure-5.4-cloud-tools-5.4.0-1039
2021-09-15T23:12:55.0872126Z   linux-azure-5.4-cloud-tools-5.4.0-1040
2021-09-15T23:12:55.0874428Z   linux-azure-5.4-cloud-tools-5.4.0-1041
2021-09-15T23:12:55.0876302Z   linux-azure-5.4-cloud-tools-5.4.0-1044
2021-09-15T23:12:55.0878167Z   linux-azure-5.4-cloud-tools-5.4.0-1046
2021-09-15T23:12:55.0879615Z   linux-azure-5.4-cloud-tools-5.4.0-1047
2021-09-15T23:12:55.0881095Z   linux-azure-5.4-cloud-tools-5.4.0-1048
2021-09-15T23:12:55.0882602Z   linux-azure-5.4-cloud-tools-5.4.0-1051
2021-09-15T23:12:55.0884289Z   linux-azure-5.4-cloud-tools-5.4.0-1055 linux-azure-5.4-headers-5.4.0-1031
2021-09-15T23:12:55.0886185Z   linux-azure-5.4-headers-5.4.0-1032 linux-azure-5.4-headers-5.4.0-1034
2021-09-15T23:12:55.0890916Z   linux-azure-5.4-headers-5.4.0-1035 linux-azure-5.4-headers-5.4.0-1036
2021-09-15T23:12:55.0892481Z   linux-azure-5.4-headers-5.4.0-1039 linux-azure-5.4-headers-5.4.0-1040
2021-09-15T23:12:55.0893947Z   linux-azure-5.4-headers-5.4.0-1041 linux-azure-5.4-headers-5.4.0-1044
2021-09-15T23:12:55.0895312Z   linux-azure-5.4-headers-5.4.0-1046 linux-azure-5.4-headers-5.4.0-1047
2021-09-15T23:12:55.0896688Z   linux-azure-5.4-headers-5.4.0-1048 linux-azure-5.4-headers-5.4.0-1051
2021-09-15T23:12:55.0898029Z   linux-azure-5.4-headers-5.4.0-1055 linux-azure-5.4-tools-5.4.0-1031
2021-09-15T23:12:55.0899298Z   linux-azure-5.4-tools-5.4.0-1032 linux-azure-5.4-tools-5.4.0-1034
2021-09-15T23:12:55.0900548Z   linux-azure-5.4-tools-5.4.0-1035 linux-azure-5.4-tools-5.4.0-1036
2021-09-15T23:12:55.0901796Z   linux-azure-5.4-tools-5.4.0-1039 linux-azure-5.4-tools-5.4.0-1040
2021-09-15T23:12:55.0903047Z   linux-azure-5.4-tools-5.4.0-1041 linux-azure-5.4-tools-5.4.0-1044
2021-09-15T23:12:55.0904384Z   linux-azure-5.4-tools-5.4.0-1046 linux-azure-5.4-tools-5.4.0-1047
2021-09-15T23:12:55.0906346Z   linux-azure-5.4-tools-5.4.0-1048 linux-azure-5.4-tools-5.4.0-1051
2021-09-15T23:12:55.0907412Z   linux-azure-5.4-tools-5.4.0-1055
2021-09-15T23:12:55.0908160Z Use 'sudo apt autoremove' to remove them.
2021-09-15T23:12:55.1315489Z 0 upgraded, 0 newly installed, 0 to remove and 67 not upgraded.
2021-09-15T23:12:55.1941550Z Warning: apt-key output should not be parsed (stdout is not a terminal)
2021-09-15T23:12:55.3033324Z OK
2021-09-15T23:12:55.9840762Z Hit:1 http://azure.archive.ubuntu.com/ubuntu bionic InRelease
2021-09-15T23:12:55.9864618Z Get:2 http://azure.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
2021-09-15T23:12:55.9866401Z Get:3 http://azure.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
2021-09-15T23:12:56.0088056Z Get:4 https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64  InRelease [1484 B]
2021-09-15T23:12:56.0103947Z Hit:5 https://download.docker.com/linux/ubuntu bionic InRelease
2021-09-15T23:12:56.0140909Z Get:6 https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/amd64  InRelease [1481 B]
2021-09-15T23:12:56.0176024Z Hit:7 http://packages.microsoft.com/repos/azurecore bionic InRelease
2021-09-15T23:12:56.0193559Z Hit:8 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64  InRelease
2021-09-15T23:12:56.1341058Z Hit:9 http://security.ubuntu.com/ubuntu bionic-security InRelease
2021-09-15T23:12:58.7911047Z Fetched 166 kB in 1s (228 kB/s)
2021-09-15T23:12:59.7421970Z Reading package lists...
2021-09-15T23:12:59.8363920Z Warning: apt-key output should not be parsed (stdout is not a terminal)
2021-09-15T23:12:59.9533374Z OK
2021-09-15T23:12:59.9943407Z deb https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/$(ARCH) /
2021-09-15T23:12:59.9944794Z #deb https://nvidia.github.io/libnvidia-container/experimental/ubuntu18.04/$(ARCH) /
2021-09-15T23:12:59.9946139Z deb https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/$(ARCH) /
2021-09-15T23:12:59.9947885Z #deb https://nvidia.github.io/nvidia-container-runtime/experimental/ubuntu18.04/$(ARCH) /
2021-09-15T23:12:59.9949152Z deb https://nvidia.github.io/nvidia-docker/ubuntu18.04/$(ARCH) /
2021-09-15T23:13:00.0858628Z Hit:1 http://azure.archive.ubuntu.com/ubuntu bionic InRelease
2021-09-15T23:13:00.0860384Z Get:2 http://azure.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
2021-09-15T23:13:00.0871844Z Get:3 http://azure.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
2021-09-15T23:13:00.1094525Z Get:4 https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64  InRelease [1484 B]
2021-09-15T23:13:00.1106033Z Hit:5 https://download.docker.com/linux/ubuntu bionic InRelease
2021-09-15T23:13:00.1144289Z Get:6 https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/amd64  InRelease [1481 B]
2021-09-15T23:13:00.1194299Z Hit:7 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64  InRelease
2021-09-15T23:13:00.1203912Z Hit:8 http://packages.microsoft.com/repos/azurecore bionic InRelease
2021-09-15T23:13:00.3105003Z Hit:9 http://security.ubuntu.com/ubuntu bionic-security InRelease
2021-09-15T23:13:00.8956958Z Fetched 166 kB in 1s (230 kB/s)
2021-09-15T23:13:01.8442490Z Reading package lists...
2021-09-15T23:13:01.9464317Z Reading package lists...
2021-09-15T23:13:02.0984029Z Building dependency tree...
2021-09-15T23:13:02.0989655Z Reading state information...
2021-09-15T23:13:02.2110377Z containerd.io is already the newest version (1.4.9-1).
2021-09-15T23:13:02.2111411Z docker-ce-cli is already the newest version (5:20.10.8~3-0~ubuntu-bionic).
2021-09-15T23:13:02.2112329Z docker-ce is already the newest version (5:20.10.8~3-0~ubuntu-bionic).
2021-09-15T23:13:02.2113146Z nvidia-docker2 is already the newest version (2.6.0-1).
2021-09-15T23:13:02.2113885Z The following packages were automatically installed and are no longer required:
2021-09-15T23:13:02.2114878Z   linux-azure-5.4-cloud-tools-5.4.0-1031
2021-09-15T23:13:02.2115949Z   linux-azure-5.4-cloud-tools-5.4.0-1032
2021-09-15T23:13:02.2117175Z   linux-azure-5.4-cloud-tools-5.4.0-1034
2021-09-15T23:13:02.2118190Z   linux-azure-5.4-cloud-tools-5.4.0-1035
2021-09-15T23:13:02.2119167Z   linux-azure-5.4-cloud-tools-5.4.0-1036
2021-09-15T23:13:02.2120126Z   linux-azure-5.4-cloud-tools-5.4.0-1039
2021-09-15T23:13:02.2121097Z   linux-azure-5.4-cloud-tools-5.4.0-1040
2021-09-15T23:13:02.2122068Z   linux-azure-5.4-cloud-tools-5.4.0-1041
2021-09-15T23:13:02.2123023Z   linux-azure-5.4-cloud-tools-5.4.0-1044
2021-09-15T23:13:02.2123995Z   linux-azure-5.4-cloud-tools-5.4.0-1046
2021-09-15T23:13:02.2124956Z   linux-azure-5.4-cloud-tools-5.4.0-1047
2021-09-15T23:13:02.2125929Z   linux-azure-5.4-cloud-tools-5.4.0-1048
2021-09-15T23:13:02.2126886Z   linux-azure-5.4-cloud-tools-5.4.0-1051
2021-09-15T23:13:02.2128128Z   linux-azure-5.4-cloud-tools-5.4.0-1055 linux-azure-5.4-headers-5.4.0-1031
2021-09-15T23:13:02.2129550Z   linux-azure-5.4-headers-5.4.0-1032 linux-azure-5.4-headers-5.4.0-1034
2021-09-15T23:13:02.2131385Z   linux-azure-5.4-headers-5.4.0-1035 linux-azure-5.4-headers-5.4.0-1036
2021-09-15T23:13:02.2133007Z   linux-azure-5.4-headers-5.4.0-1039 linux-azure-5.4-headers-5.4.0-1040
2021-09-15T23:13:02.2134441Z   linux-azure-5.4-headers-5.4.0-1041 linux-azure-5.4-headers-5.4.0-1044
2021-09-15T23:13:02.2135810Z   linux-azure-5.4-headers-5.4.0-1046 linux-azure-5.4-headers-5.4.0-1047
2021-09-15T23:13:02.2137168Z   linux-azure-5.4-headers-5.4.0-1048 linux-azure-5.4-headers-5.4.0-1051
2021-09-15T23:13:02.2138488Z   linux-azure-5.4-headers-5.4.0-1055 linux-azure-5.4-tools-5.4.0-1031
2021-09-15T23:13:02.2139763Z   linux-azure-5.4-tools-5.4.0-1032 linux-azure-5.4-tools-5.4.0-1034
2021-09-15T23:13:02.2140995Z   linux-azure-5.4-tools-5.4.0-1035 linux-azure-5.4-tools-5.4.0-1036
2021-09-15T23:13:02.2142244Z   linux-azure-5.4-tools-5.4.0-1039 linux-azure-5.4-tools-5.4.0-1040
2021-09-15T23:13:02.2143485Z   linux-azure-5.4-tools-5.4.0-1041 linux-azure-5.4-tools-5.4.0-1044
2021-09-15T23:13:02.2144727Z   linux-azure-5.4-tools-5.4.0-1046 linux-azure-5.4-tools-5.4.0-1047
2021-09-15T23:13:02.2148122Z   linux-azure-5.4-tools-5.4.0-1048 linux-azure-5.4-tools-5.4.0-1051
2021-09-15T23:13:02.2151436Z   linux-azure-5.4-tools-5.4.0-1055
2021-09-15T23:13:02.2152205Z Use 'sudo apt autoremove' to remove them.
2021-09-15T23:13:02.2548533Z 0 upgraded, 0 newly installed, 0 to remove and 67 not upgraded.
2021-09-15T23:13:11.2838582Z ##[group]Run sudo rm -rf $GITHUB_WORKSPACE
2021-09-15T23:13:11.2839209Z �[36;1msudo rm -rf $GITHUB_WORKSPACE�[0m
2021-09-15T23:13:11.2859604Z shell: /bin/bash -e {0}
2021-09-15T23:13:11.2859946Z env:
2021-09-15T23:13:11.2860287Z   github_actions: true
2021-09-15T23:13:11.2860627Z   os_name: linux
2021-09-15T23:13:11.2860950Z   task: cuda
2021-09-15T23:13:11.2861283Z   conda_env: test-env
2021-09-15T23:13:11.2861643Z ##[endgroup]
2021-09-15T23:13:11.3816298Z ##[group]Run actions/checkout@v1
2021-09-15T23:13:11.3816706Z with:
2021-09-15T23:13:11.3817038Z   fetch-depth: 5
2021-09-15T23:13:11.3817410Z   submodules: true
2021-09-15T23:13:11.3817758Z   clean: true
2021-09-15T23:13:11.3818061Z env:
2021-09-15T23:13:11.3818375Z   github_actions: true
2021-09-15T23:13:11.3818727Z   os_name: linux
2021-09-15T23:13:11.3819029Z   task: cuda
2021-09-15T23:13:11.3819409Z   conda_env: test-env
2021-09-15T23:13:11.3819755Z ##[endgroup]
2021-09-15T23:13:11.7329559Z Syncing repository: microsoft/LightGBM
2021-09-15T23:13:11.7455807Z ##[command]git version
2021-09-15T23:13:11.7745974Z git version 2.17.1
2021-09-15T23:13:11.7856545Z ##[command]git init "/home/guoke/actions-runner/_work/LightGBM/LightGBM"
2021-09-15T23:13:11.7884021Z Initialized empty Git repository in /home/guoke/actions-runner/_work/LightGBM/LightGBM/.git/
2021-09-15T23:13:11.7898055Z ##[command]git remote add origin https://github.com/microsoft/LightGBM
2021-09-15T23:13:11.7924687Z ##[command]git config gc.auto 0
2021-09-15T23:13:11.7949418Z ##[command]git config --get-all http.https://github.com/microsoft/LightGBM.extraheader
2021-09-15T23:13:11.8017128Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --tags --prune --progress --no-recurse-submodules --depth=5 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/4606/merge:refs/remotes/pull/4606/merge
2021-09-15T23:13:12.3992361Z remote: Enumerating objects: 5037, done.        
2021-09-15T23:13:12.3997556Z remote: Counting objects:   0% (1/5037)        
2021-09-15T23:13:12.4000132Z remote: Counting objects:   1% (51/5037)        
2021-09-15T23:13:12.4002058Z remote: Counting objects:   2% (101/5037)        
2021-09-15T23:13:12.4002598Z remote: Counting objects:   3% (152/5037)        
2021-09-15T23:13:12.4003067Z remote: Counting objects:   4% (202/5037)        
2021-09-15T23:13:12.4003537Z remote: Counting objects:   5% (252/5037)        
2021-09-15T23:13:12.4004026Z remote: Counting objects:   6% (303/5037)        
2021-09-15T23:13:12.4013610Z remote: Counting objects:   7% (353/5037)        
2021-09-15T23:13:12.4014136Z remote: Counting objects:   8% (403/5037)        
2021-09-15T23:13:12.4014855Z remote: Counting objects:   9% (454/5037)        
2021-09-15T23:13:12.4015303Z remote: Counting objects:  10% (504/5037)        
2021-09-15T23:13:12.4015761Z remote: Counting objects:  11% (555/5037)        
2021-09-15T23:13:12.4016206Z remote: Counting objects:  12% (605/5037)        
2021-09-15T23:13:12.4018901Z remote: Counting objects:  13% (655/5037)        
2021-09-15T23:13:12.4019614Z remote: Counting objects:  14% (706/5037)        
2021-09-15T23:13:12.4020069Z remote: Counting objects:  15% (756/5037)        
2021-09-15T23:13:12.4020534Z remote: Counting objects:  16% (806/5037)        
2021-09-15T23:13:12.4020983Z remote: Counting objects:  17% (857/5037)        
2021-09-15T23:13:12.4021542Z remote: Counting objects:  18% (907/5037)        
2021-09-15T23:13:12.4022096Z remote: Counting objects:  19% (958/5037)        
2021-09-15T23:13:12.4022557Z remote: Counting objects:  20% (1008/5037)        
2021-09-15T23:13:12.4025911Z remote: Counting objects:  21% (1058/5037)        
2021-09-15T23:13:12.4026568Z remote: Counting objects:  22% (1109/5037)        
2021-09-15T23:13:12.4027076Z remote: Counting objects:  23% (1159/5037)        
2021-09-15T23:13:12.4027544Z remote: Counting objects:  24% (1209/5037)        
2021-09-15T23:13:12.4028134Z remote: Counting objects:  25% (1260/5037)        
2021-09-15T23:13:12.4028605Z remote: Counting objects:  26% (1310/5037)        
2021-09-15T23:13:12.4029174Z remote: Counting objects:  27% (1360/5037)        
2021-09-15T23:13:12.4029647Z remote: Counting objects:  28% (1411/5037)        
2021-09-15T23:13:12.4030109Z remote: Counting objects:  29% (1461/5037)        
2021-09-15T23:13:12.4030689Z remote: Counting objects:  30% (1512/5037)        
2021-09-15T23:13:12.4031153Z remote: Counting objects:  31% (1562/5037)        
2021-09-15T23:13:12.4031936Z remote: Counting objects:  32% (1612/5037)        
2021-09-15T23:13:12.4032434Z remote: Counting objects:  33% (1663/5037)        
2021-09-15T23:13:12.4033031Z remote: Counting objects:  34% (1713/5037)        
2021-09-15T23:13:12.4033503Z remote: Counting objects:  35% (1763/5037)        
2021-09-15T23:13:12.4034092Z remote: Counting objects:  36% (1814/5037)        
2021-09-15T23:13:12.4034660Z remote: Counting objects:  37% (1864/5037)        
2021-09-15T23:13:12.4035133Z remote: Counting objects:  38% (1915/5037)        
2021-09-15T23:13:12.4035581Z remote: Counting objects:  39% (1965/5037)        
2021-09-15T23:13:12.4036163Z remote: Counting objects:  40% (2015/5037)        
2021-09-15T23:13:12.4036631Z remote: Counting objects:  41% (2066/5037)        
2021-09-15T23:13:12.4037214Z remote: Counting objects:  42% (2116/5037)        
2021-09-15T23:13:12.4037680Z remote: Counting objects:  43% (2166/5037)        
2021-09-15T23:13:12.4038127Z remote: Counting objects:  44% (2217/5037)        
2021-09-15T23:13:12.4038593Z remote: Counting objects:  45% (2267/5037)        
2021-09-15T23:13:12.4039056Z remote: Counting objects:  46% (2318/5037)        
2021-09-15T23:13:12.4039505Z remote: Counting objects:  47% (2368/5037)        
2021-09-15T23:13:12.4039969Z remote: Counting objects:  48% (2418/5037)        
2021-09-15T23:13:12.4040415Z remote: Counting objects:  49% (2469/5037)        
2021-09-15T23:13:12.4041038Z remote: Counting objects:  50% (2519/5037)        
2021-09-15T23:13:12.4041488Z remote: Counting objects:  51% (2569/5037)        
2021-09-15T23:13:12.4041949Z remote: Counting objects:  52% (2620/5037)        
2021-09-15T23:13:12.4042409Z remote: Counting objects:  53% (2670/5037)        
2021-09-15T23:13:12.4042886Z remote: Counting objects:  54% (2720/5037)        
2021-09-15T23:13:12.4043485Z remote: Counting objects:  55% (2771/5037)        
2021-09-15T23:13:12.4044054Z remote: Counting objects:  56% (2821/5037)        
2021-09-15T23:13:12.4044526Z remote: Counting objects:  57% (2872/5037)        
2021-09-15T23:13:12.4045110Z remote: Counting objects:  58% (2922/5037)        
2021-09-15T23:13:12.4045558Z remote: Counting objects:  59% (2972/5037)        
2021-09-15T23:13:12.4046316Z remote: Counting objects:  60% (3023/5037)        
2021-09-15T23:13:12.4046763Z remote: Counting objects:  61% (3073/5037)        
2021-09-15T23:13:12.4047354Z remote: Counting objects:  62% (3123/5037)        
2021-09-15T23:13:12.4047817Z remote: Counting objects:  63% (3174/5037)        
2021-09-15T23:13:12.4048609Z remote: Counting objects:  64% (3224/5037)        
2021-09-15T23:13:12.4049079Z remote: Counting objects:  65% (3275/5037)        
2021-09-15T23:13:12.4049539Z remote: Counting objects:  66% (3325/5037)        
2021-09-15T23:13:12.4049984Z remote: Counting objects:  67% (3375/5037)        
2021-09-15T23:13:12.4050442Z remote: Counting objects:  68% (3426/5037)        
2021-09-15T23:13:12.4050917Z remote: Counting objects:  69% (3476/5037)        
2021-09-15T23:13:12.4051378Z remote: Counting objects:  70% (3526/5037)        
2021-09-15T23:13:12.4051847Z remote: Counting objects:  71% (3577/5037)        
2021-09-15T23:13:12.4052293Z remote: Counting objects:  72% (3627/5037)        
2021-09-15T23:13:12.4052757Z remote: Counting objects:  73% (3678/5037)        
2021-09-15T23:13:12.4053203Z remote: Counting objects:  74% (3728/5037)        
2021-09-15T23:13:12.4053660Z remote: Counting objects:  75% (3778/5037)        
2021-09-15T23:13:12.4054121Z remote: Counting objects:  76% (3829/5037)        
2021-09-15T23:13:12.4054568Z remote: Counting objects:  77% (3879/5037)        
2021-09-15T23:13:12.4055029Z remote: Counting objects:  78% (3929/5037)        
2021-09-15T23:13:12.4055479Z remote: Counting objects:  79% (3980/5037)        
2021-09-15T23:13:12.4055935Z remote: Counting objects:  80% (4030/5037)        
2021-09-15T23:13:12.4056384Z remote: Counting objects:  81% (4080/5037)        
2021-09-15T23:13:12.4058215Z remote: Counting objects:  82% (4131/5037)        
2021-09-15T23:13:12.4058840Z remote: Counting objects:  83% (4181/5037)        
2021-09-15T23:13:12.4059341Z remote: Counting objects:  84% (4232/5037)        
2021-09-15T23:13:12.4059803Z remote: Counting objects:  85% (4282/5037)        
2021-09-15T23:13:12.4077588Z remote: Counting objects:  86% (4332/5037)        
2021-09-15T23:13:12.4078126Z remote: Counting objects:  87% (4383/5037)        
2021-09-15T23:13:12.4078605Z remote: Counting objects:  88% (4433/5037)        
2021-09-15T23:13:12.4086904Z remote: Counting objects:  89% (4483/5037)        
2021-09-15T23:13:12.4087436Z remote: Counting objects:  90% (4534/5037)        
2021-09-15T23:13:12.4087900Z remote: Counting objects:  91% (4584/5037)        
2021-09-15T23:13:12.4088350Z remote: Counting objects:  92% (4635/5037)        
2021-09-15T23:13:12.4088812Z remote: Counting objects:  93% (4685/5037)        
2021-09-15T23:13:12.4089259Z remote: Counting objects:  94% (4735/5037)        
2021-09-15T23:13:12.4089718Z remote: Counting objects:  95% (4786/5037)        
2021-09-15T23:13:12.4090194Z remote: Counting objects:  96% (4836/5037)        
2021-09-15T23:13:12.4090757Z remote: Counting objects:  97% (4886/5037)        
2021-09-15T23:13:12.4091427Z remote: Counting objects:  98% (4937/5037)        
2021-09-15T23:13:12.4091967Z remote: Counting objects:  99% (4987/5037)        
2021-09-15T23:13:12.4092430Z remote: Counting objects: 100% (5037/5037)        
2021-09-15T23:13:12.4092898Z remote: Counting objects: 100% (5037/5037), done.        
2021-09-15T23:13:12.4124229Z remote: Compressing objects:   0% (1/3137)        
2021-09-15T23:13:12.4151584Z remote: Compressing objects:   1% (32/3137)        
2021-09-15T23:13:12.4342232Z remote: Compressing objects:   2% (63/3137)        
2021-09-15T23:13:12.4503541Z remote: Compressing objects:   3% (95/3137)        
2021-09-15T23:13:12.4554205Z remote: Compressing objects:   4% (126/3137)        
2021-09-15T23:13:12.4693305Z remote: Compressing objects:   5% (157/3137)        
2021-09-15T23:13:12.4771123Z remote: Compressing objects:   6% (189/3137)        
2021-09-15T23:13:12.4888125Z remote: Compressing objects:   7% (220/3137)        
2021-09-15T23:13:12.5218277Z remote: Compressing objects:   8% (251/3137)        
2021-09-15T23:13:12.5282053Z remote: Compressing objects:   9% (283/3137)        
2021-09-15T23:13:12.5416655Z remote: Compressing objects:  10% (314/3137)        
2021-09-15T23:13:12.5494703Z remote: Compressing objects:  11% (346/3137)        
2021-09-15T23:13:12.5528850Z remote: Compressing objects:  12% (377/3137)        
2021-09-15T23:13:12.5542493Z remote: Compressing objects:  13% (408/3137)        
2021-09-15T23:13:12.5617926Z remote: Compressing objects:  14% (440/3137)        
2021-09-15T23:13:12.5666161Z remote: Compressing objects:  15% (471/3137)        
2021-09-15T23:13:12.5753207Z remote: Compressing objects:  16% (502/3137)        
2021-09-15T23:13:12.5839699Z remote: Compressing objects:  17% (534/3137)        
2021-09-15T23:13:12.5904768Z remote: Compressing objects:  18% (565/3137)        
2021-09-15T23:13:12.5981691Z remote: Compressing objects:  19% (597/3137)        
2021-09-15T23:13:12.6041044Z remote: Compressing objects:  20% (628/3137)        
2021-09-15T23:13:12.6159677Z remote: Compressing objects:  21% (659/3137)        
2021-09-15T23:13:12.6310396Z remote: Compressing objects:  22% (691/3137)        
2021-09-15T23:13:12.6630778Z remote: Compressing objects:  23% (722/3137)        
2021-09-15T23:13:12.6727809Z remote: Compressing objects:  24% (753/3137)        
2021-09-15T23:13:12.6799891Z remote: Compressing objects:  25% (785/3137)        
2021-09-15T23:13:12.6975913Z remote: Compressing objects:  26% (816/3137)        
2021-09-15T23:13:12.7044111Z remote: Compressing objects:  27% (847/3137)        
2021-09-15T23:13:12.7129636Z remote: Compressing objects:  28% (879/3137)        
2021-09-15T23:13:12.7178770Z remote: Compressing objects:  29% (910/3137)        
2021-09-15T23:13:12.7200197Z remote: Compressing objects:  30% (942/3137)        
2021-09-15T23:13:12.7209561Z remote: Compressing objects:  31% (973/3137)        
2021-09-15T23:13:12.7256553Z remote: Compressing objects:  32% (1004/3137)        
2021-09-15T23:13:12.7349152Z remote: Compressing objects:  33% (1036/3137)        
2021-09-15T23:13:12.7395967Z remote: Compressing objects:  34% (1067/3137)        
2021-09-15T23:13:12.7397375Z remote: Compressing objects:  35% (1098/3137)        
2021-09-15T23:13:12.7422992Z remote: Compressing objects:  36% (1130/3137)        
2021-09-15T23:13:12.7451812Z remote: Compressing objects:  37% (1161/3137)        
2021-09-15T23:13:12.7620199Z remote: Compressing objects:  38% (1193/3137)        
2021-09-15T23:13:12.7693342Z remote: Compressing objects:  39% (1224/3137)        
2021-09-15T23:13:12.7701196Z remote: Compressing objects:  40% (1255/3137)        
2021-09-15T23:13:12.7741248Z remote: Compressing objects:  41% (1287/3137)        
2021-09-15T23:13:12.7777343Z remote: Compressing objects:  42% (1318/3137)        
2021-09-15T23:13:12.7809874Z remote: Compressing objects:  43% (1349/3137)        
2021-09-15T23:13:12.7814608Z remote: Compressing objects:  44% (1381/3137)        
2021-09-15T23:13:12.7840163Z remote: Compressing objects:  45% (1412/3137)        
2021-09-15T23:13:12.7866584Z remote: Compressing objects:  46% (1444/3137)        
2021-09-15T23:13:12.7884121Z remote: Compressing objects:  47% (1475/3137)        
2021-09-15T23:13:12.7906250Z remote: Compressing objects:  48% (1506/3137)        
2021-09-15T23:13:12.7960824Z remote: Compressing objects:  49% (1538/3137)        
2021-09-15T23:13:12.8252360Z remote: Compressing objects:  50% (1569/3137)        
2021-09-15T23:13:12.8253242Z remote: Compressing objects:  51% (1600/3137)        
2021-09-15T23:13:12.8253860Z remote: Compressing objects:  52% (1632/3137)        
2021-09-15T23:13:12.8254463Z remote: Compressing objects:  53% (1663/3137)        
2021-09-15T23:13:12.8277632Z remote: Compressing objects:  54% (1694/3137)        
2021-09-15T23:13:12.8350497Z remote: Compressing objects:  55% (1726/3137)        
2021-09-15T23:13:12.8382204Z remote: Compressing objects:  56% (1757/3137)        
2021-09-15T23:13:12.8405450Z remote: Compressing objects:  57% (1789/3137)        
2021-09-15T23:13:12.8488976Z remote: Compressing objects:  58% (1820/3137)        
2021-09-15T23:13:12.8607424Z remote: Compressing objects:  59% (1851/3137)        
2021-09-15T23:13:12.8681759Z remote: Compressing objects:  60% (1883/3137)        
2021-09-15T23:13:12.8744477Z remote: Compressing objects:  61% (1914/3137)        
2021-09-15T23:13:12.8794094Z remote: Compressing objects:  62% (1945/3137)        
2021-09-15T23:13:12.8829955Z remote: Compressing objects:  63% (1977/3137)        
2021-09-15T23:13:12.8885494Z remote: Compressing objects:  64% (2008/3137)        
2021-09-15T23:13:12.8917833Z remote: Compressing objects:  65% (2040/3137)        
2021-09-15T23:13:12.8923560Z remote: Compressing objects:  66% (2071/3137)        
2021-09-15T23:13:12.8932265Z remote: Compressing objects:  67% (2102/3137)        
2021-09-15T23:13:12.8933202Z remote: Compressing objects:  68% (2134/3137)        
2021-09-15T23:13:12.8933923Z remote: Compressing objects:  69% (2165/3137)        
2021-09-15T23:13:12.8934730Z remote: Compressing objects:  70% (2196/3137)        
2021-09-15T23:13:12.8935585Z remote: Compressing objects:  71% (2228/3137)        
2021-09-15T23:13:12.8943721Z remote: Compressing objects:  72% (2259/3137)        
2021-09-15T23:13:12.8953050Z remote: Compressing objects:  73% (2291/3137)        
2021-09-15T23:13:12.8965084Z remote: Compressing objects:  74% (2322/3137)        
2021-09-15T23:13:12.8965951Z remote: Compressing objects:  75% (2353/3137)        
2021-09-15T23:13:12.8966784Z remote: Compressing objects:  76% (2385/3137)        
2021-09-15T23:13:12.8976700Z remote: Compressing objects:  77% (2416/3137)        
2021-09-15T23:13:12.8985867Z remote: Compressing objects:  78% (2447/3137)        
2021-09-15T23:13:12.8988876Z remote: Compressing objects:  79% (2479/3137)        
2021-09-15T23:13:12.9002207Z remote: Compressing objects:  80% (2510/3137)        
2021-09-15T23:13:12.9003250Z remote: Compressing objects:  81% (2541/3137)        
2021-09-15T23:13:12.9004107Z remote: Compressing objects:  82% (2573/3137)        
2021-09-15T23:13:12.9004787Z remote: Compressing objects:  83% (2604/3137)        
2021-09-15T23:13:12.9007465Z remote: Compressing objects:  84% (2636/3137)        
2021-09-15T23:13:12.9011803Z remote: Compressing objects:  85% (2667/3137)        
2021-09-15T23:13:12.9016447Z remote: Compressing objects:  86% (2698/3137)        
2021-09-15T23:13:12.9019312Z remote: Compressing objects:  87% (2730/3137)        
2021-09-15T23:13:12.9020588Z remote: Compressing objects:  88% (2761/3137)        
2021-09-15T23:13:12.9034238Z remote: Compressing objects:  89% (2792/3137)        
2021-09-15T23:13:12.9042404Z remote: Compressing objects:  90% (2824/3137)        
2021-09-15T23:13:12.9064685Z remote: Compressing objects:  91% (2855/3137)        
2021-09-15T23:13:12.9072015Z remote: Compressing objects:  92% (2887/3137)        
2021-09-15T23:13:12.9072597Z remote: Compressing objects:  93% (2918/3137)        
2021-09-15T23:13:12.9095856Z remote: Compressing objects:  94% (2949/3137)        
2021-09-15T23:13:12.9110010Z remote: Compressing objects:  95% (2981/3137)        
2021-09-15T23:13:12.9119981Z remote: Compressing objects:  96% (3012/3137)        
2021-09-15T23:13:12.9121451Z remote: Compressing objects:  97% (3043/3137)        
2021-09-15T23:13:12.9126180Z remote: Compressing objects:  98% (3075/3137)        
2021-09-15T23:13:12.9133418Z remote: Compressing objects:  99% (3106/3137)        
2021-09-15T23:13:12.9134293Z remote: Compressing objects: 100% (3137/3137)        
2021-09-15T23:13:12.9135045Z remote: Compressing objects: 100% (3137/3137), done.        
2021-09-15T23:13:12.9162622Z Receiving objects:   0% (1/5037)   
2021-09-15T23:13:12.9170192Z Receiving objects:   1% (51/5037)   
2021-09-15T23:13:12.9177210Z Receiving objects:   2% (101/5037)   
2021-09-15T23:13:12.9182375Z Receiving objects:   3% (152/5037)   
2021-09-15T23:13:12.9187814Z Receiving objects:   4% (202/5037)   
2021-09-15T23:13:12.9190776Z Receiving objects:   5% (252/5037)   
2021-09-15T23:13:12.9191288Z Receiving objects:   6% (303/5037)   
2021-09-15T23:13:12.9192698Z Receiving objects:   7% (353/5037)   
2021-09-15T23:13:12.9195142Z Receiving objects:   8% (403/5037)   
2021-09-15T23:13:12.9201990Z Receiving objects:   9% (454/5037)   
2021-09-15T23:13:12.9202468Z Receiving objects:  10% (504/5037)   
2021-09-15T23:13:12.9212889Z Receiving objects:  11% (555/5037)   
2021-09-15T23:13:12.9215519Z Receiving objects:  12% (605/5037)   
2021-09-15T23:13:12.9221277Z Receiving objects:  13% (655/5037)   
2021-09-15T23:13:12.9223527Z Receiving objects:  14% (706/5037)   
2021-09-15T23:13:12.9227701Z Receiving objects:  15% (756/5037)   
2021-09-15T23:13:12.9236397Z Receiving objects:  16% (806/5037)   
2021-09-15T23:13:12.9247315Z Receiving objects:  17% (857/5037)   
2021-09-15T23:13:12.9250835Z Receiving objects:  18% (907/5037)   
2021-09-15T23:13:12.9261784Z Receiving objects:  19% (958/5037)   
2021-09-15T23:13:12.9263551Z Receiving objects:  20% (1008/5037)   
2021-09-15T23:13:12.9270579Z Receiving objects:  21% (1058/5037)   
2021-09-15T23:13:12.9276586Z Receiving objects:  22% (1109/5037)   
2021-09-15T23:13:12.9286436Z Receiving objects:  23% (1159/5037)   
2021-09-15T23:13:12.9297585Z Receiving objects:  24% (1209/5037)   
2021-09-15T23:13:12.9306092Z Receiving objects:  25% (1260/5037)   
2021-09-15T23:13:12.9326139Z Receiving objects:  26% (1310/5037)   
2021-09-15T23:13:12.9336790Z Receiving objects:  27% (1360/5037)   
2021-09-15T23:13:12.9368966Z Receiving objects:  28% (1411/5037)   
2021-09-15T23:13:12.9390302Z Receiving objects:  29% (1461/5037)   
2021-09-15T23:13:12.9455552Z Receiving objects:  30% (1512/5037)   
2021-09-15T23:13:12.9472642Z Receiving objects:  31% (1562/5037)   
2021-09-15T23:13:12.9483989Z Receiving objects:  32% (1612/5037)   
2021-09-15T23:13:12.9505720Z Receiving objects:  33% (1663/5037)   
2021-09-15T23:13:12.9539634Z Receiving objects:  34% (1713/5037)   
2021-09-15T23:13:12.9544546Z Receiving objects:  35% (1763/5037)   
2021-09-15T23:13:12.9546553Z Receiving objects:  36% (1814/5037)   
2021-09-15T23:13:12.9551263Z Receiving objects:  37% (1864/5037)   
2021-09-15T23:13:12.9555914Z Receiving objects:  38% (1915/5037)   
2021-09-15T23:13:12.9571344Z Receiving objects:  39% (1965/5037)   
2021-09-15T23:13:12.9581021Z Receiving objects:  40% (2015/5037)   
2021-09-15T23:13:12.9590686Z Receiving objects:  41% (2066/5037)   
2021-09-15T23:13:12.9608906Z Receiving objects:  42% (2116/5037)   
2021-09-15T23:13:12.9625789Z Receiving objects:  43% (2166/5037)   
2021-09-15T23:13:12.9642262Z Receiving objects:  44% (2217/5037)   
2021-09-15T23:13:12.9675483Z Receiving objects:  45% (2267/5037)   
2021-09-15T23:13:12.9752987Z Receiving objects:  46% (2318/5037)   
2021-09-15T23:13:13.0898606Z Receiving objects:  47% (2368/5037)   
2021-09-15T23:13:13.1307710Z Receiving objects:  48% (2418/5037)   
2021-09-15T23:13:13.1729760Z Receiving objects:  49% (2469/5037)   
2021-09-15T23:13:13.1757416Z Receiving objects:  50% (2519/5037)   
2021-09-15T23:13:13.1766793Z Receiving objects:  51% (2569/5037)   
2021-09-15T23:13:13.1783088Z Receiving objects:  52% (2620/5037)   
2021-09-15T23:13:13.1789883Z Receiving objects:  53% (2670/5037)   
2021-09-15T23:13:13.1795925Z Receiving objects:  54% (2720/5037)   
2021-09-15T23:13:13.1810030Z Receiving objects:  55% (2771/5037)   
2021-09-15T23:13:13.1816930Z Receiving objects:  56% (2821/5037)   
2021-09-15T23:13:13.1834165Z Receiving objects:  57% (2872/5037)   
2021-09-15T23:13:13.1847598Z Receiving objects:  58% (2922/5037)   
2021-09-15T23:13:13.1856749Z Receiving objects:  59% (2972/5037)   
2021-09-15T23:13:13.1869431Z Receiving objects:  60% (3023/5037)   
2021-09-15T23:13:13.1878490Z Receiving objects:  61% (3073/5037)   
2021-09-15T23:13:13.1890843Z Receiving objects:  62% (3123/5037)   
2021-09-15T23:13:13.1903610Z Receiving objects:  63% (3174/5037)   
2021-09-15T23:13:13.1916243Z Receiving objects:  64% (3224/5037)   
2021-09-15T23:13:13.1929872Z Receiving objects:  65% (3275/5037)   
2021-09-15T23:13:13.1941935Z Receiving objects:  66% (3325/5037)   
2021-09-15T23:13:13.1946580Z Receiving objects:  67% (3375/5037)   
2021-09-15T23:13:13.1953796Z Receiving objects:  68% (3426/5037)   
2021-09-15T23:13:13.1960387Z Receiving objects:  69% (3476/5037)   
2021-09-15T23:13:13.1971997Z Receiving objects:  70% (3526/5037)   
2021-09-15T23:13:13.1982863Z Receiving objects:  71% (3577/5037)   
2021-09-15T23:13:13.2005172Z Receiving objects:  72% (3627/5037)   
2021-09-15T23:13:13.2018027Z Receiving objects:  73% (3678/5037)   
2021-09-15T23:13:13.2026369Z Receiving objects:  74% (3728/5037)   
2021-09-15T23:13:13.2080966Z Receiving objects:  75% (3778/5037)   
2021-09-15T23:13:13.2093551Z Receiving objects:  76% (3829/5037)   
2021-09-15T23:13:13.2094663Z Receiving objects:  77% (3879/5037)   
2021-09-15T23:13:13.2095108Z Receiving objects:  78% (3929/5037)   
2021-09-15T23:13:13.2100634Z Receiving objects:  79% (3980/5037)   
2021-09-15T23:13:13.2169313Z Receiving objects:  80% (4030/5037)   
2021-09-15T23:13:13.2177567Z Receiving objects:  81% (4080/5037)   
2021-09-15T23:13:13.2191230Z Receiving objects:  82% (4131/5037)   
2021-09-15T23:13:13.2193616Z Receiving objects:  83% (4181/5037)   
2021-09-15T23:13:13.2195990Z Receiving objects:  84% (4232/5037)   
2021-09-15T23:13:13.2275380Z Receiving objects:  85% (4282/5037)   
2021-09-15T23:13:13.2282084Z Receiving objects:  86% (4332/5037)   
2021-09-15T23:13:13.2301568Z Receiving objects:  87% (4383/5037)   
2021-09-15T23:13:13.2309070Z Receiving objects:  88% (4433/5037)   
2021-09-15T23:13:13.2319132Z Receiving objects:  89% (4483/5037)   
2021-09-15T23:13:13.2324536Z Receiving objects:  90% (4534/5037)   
2021-09-15T23:13:13.2498387Z Receiving objects:  91% (4584/5037)   
2021-09-15T23:13:13.2503033Z Receiving objects:  92% (4635/5037)   
2021-09-15T23:13:13.2507991Z Receiving objects:  93% (4685/5037)   
2021-09-15T23:13:13.2512728Z Receiving objects:  94% (4735/5037)   
2021-09-15T23:13:13.2546081Z Receiving objects:  95% (4786/5037)   
2021-09-15T23:13:13.2551304Z Receiving objects:  96% (4836/5037)   
2021-09-15T23:13:13.2565759Z Receiving objects:  97% (4886/5037)   
2021-09-15T23:13:13.2570231Z Receiving objects:  98% (4937/5037)   
2021-09-15T23:13:13.2571731Z remote: Total 5037 (delta 3613), reused 2717 (delta 1748), pack-reused 0        
2021-09-15T23:13:13.2593860Z Receiving objects:  99% (4987/5037)   
2021-09-15T23:13:13.2594380Z Receiving objects: 100% (5037/5037)   
2021-09-15T23:13:13.2594979Z Receiving objects: 100% (5037/5037), 7.28 MiB | 21.17 MiB/s, done.
2021-09-15T23:13:13.2602604Z Resolving deltas:   0% (0/3613)   
2021-09-15T23:13:13.2603338Z Resolving deltas:   1% (56/3613)   
2021-09-15T23:13:13.2607577Z Resolving deltas:   2% (75/3613)   
2021-09-15T23:13:13.2608103Z Resolving deltas:   3% (124/3613)   
2021-09-15T23:13:13.2609398Z Resolving deltas:   4% (155/3613)   
2021-09-15T23:13:13.2611956Z Resolving deltas:   5% (181/3613)   
2021-09-15T23:13:13.2614112Z Resolving deltas:   6% (222/3613)   
2021-09-15T23:13:13.2617232Z Resolving deltas:   7% (253/3613)   
2021-09-15T23:13:13.2619356Z Resolving deltas:   8% (296/3613)   
2021-09-15T23:13:13.2621554Z Resolving deltas:   9% (333/3613)   
2021-09-15T23:13:13.2623435Z Resolving deltas:  10% (369/3613)   
2021-09-15T23:13:13.2625513Z Resolving deltas:  11% (398/3613)   
2021-09-15T23:13:13.2626020Z Resolving deltas:  12% (434/3613)   
2021-09-15T23:13:13.2628139Z Resolving deltas:  13% (470/3613)   
2021-09-15T23:13:13.2629207Z Resolving deltas:  14% (512/3613)   
2021-09-15T23:13:13.2631784Z Resolving deltas:  15% (548/3613)   
2021-09-15T23:13:13.2633611Z Resolving deltas:  16% (588/3613)   
2021-09-15T23:13:13.2636668Z Resolving deltas:  17% (617/3613)   
2021-09-15T23:13:13.2639513Z Resolving deltas:  18% (651/3613)   
2021-09-15T23:13:13.2642363Z Resolving deltas:  19% (690/3613)   
2021-09-15T23:13:13.2646328Z Resolving deltas:  20% (725/3613)   
2021-09-15T23:13:13.2648426Z Resolving deltas:  21% (765/3613)   
2021-09-15T23:13:13.2653803Z Resolving deltas:  22% (796/3613)   
2021-09-15T23:13:13.2667136Z Resolving deltas:  23% (833/3613)   
2021-09-15T23:13:13.2678688Z Resolving deltas:  24% (870/3613)   
2021-09-15T23:13:13.2682545Z Resolving deltas:  25% (906/3613)   
2021-09-15T23:13:13.2780269Z Resolving deltas:  26% (941/3613)   
2021-09-15T23:13:13.2782167Z Resolving deltas:  27% (997/3613)   
2021-09-15T23:13:13.2782629Z Resolving deltas:  28% (1012/3613)   
2021-09-15T23:13:13.2783033Z Resolving deltas:  29% (1068/3613)   
2021-09-15T23:13:13.2783430Z Resolving deltas:  30% (1102/3613)   
2021-09-15T23:13:13.2783830Z Resolving deltas:  31% (1136/3613)   
2021-09-15T23:13:13.2784222Z Resolving deltas:  32% (1157/3613)   
2021-09-15T23:13:13.2784626Z Resolving deltas:  33% (1217/3613)   
2021-09-15T23:13:13.2785014Z Resolving deltas:  34% (1230/3613)   
2021-09-15T23:13:13.2785410Z Resolving deltas:  35% (1271/3613)   
2021-09-15T23:13:13.2785805Z Resolving deltas:  36% (1307/3613)   
2021-09-15T23:13:13.2786189Z Resolving deltas:  37% (1340/3613)   
2021-09-15T23:13:13.2786593Z Resolving deltas:  38% (1374/3613)   
2021-09-15T23:13:13.2786983Z Resolving deltas:  39% (1423/3613)   
2021-09-15T23:13:13.2787383Z Resolving deltas:  40% (1447/3613)   
2021-09-15T23:13:13.2787772Z Resolving deltas:  41% (1486/3613)   
2021-09-15T23:13:13.2788724Z Resolving deltas:  42% (1520/3613)   
2021-09-15T23:13:13.2789125Z Resolving deltas:  43% (1568/3613)   
2021-09-15T23:13:13.2793745Z Resolving deltas:  44% (1590/3613)   
2021-09-15T23:13:13.2801714Z Resolving deltas:  45% (1627/3613)   
2021-09-15T23:13:13.2805951Z Resolving deltas:  46% (1679/3613)   
2021-09-15T23:13:13.2817904Z Resolving deltas:  47% (1700/3613)   
2021-09-15T23:13:13.2822204Z Resolving deltas:  48% (1746/3613)   
2021-09-15T23:13:13.2829719Z Resolving deltas:  49% (1773/3613)   
2021-09-15T23:13:13.2835995Z Resolving deltas:  50% (1807/3613)   
2021-09-15T23:13:13.2837222Z Resolving deltas:  51% (1843/3613)   
2021-09-15T23:13:13.2841784Z Resolving deltas:  52% (1883/3613)   
2021-09-15T23:13:13.2861714Z Resolving deltas:  53% (1919/3613)   
2021-09-15T23:13:13.2870441Z Resolving deltas:  54% (1970/3613)   
2021-09-15T23:13:13.2884197Z Resolving deltas:  55% (1992/3613)   
2021-09-15T23:13:13.2890845Z Resolving deltas:  56% (2033/3613)   
2021-09-15T23:13:13.2894699Z Resolving deltas:  57% (2060/3613)   
2021-09-15T23:13:13.2900502Z Resolving deltas:  58% (2099/3613)   
2021-09-15T23:13:13.2905016Z Resolving deltas:  59% (2134/3613)   
2021-09-15T23:13:13.2915634Z Resolving deltas:  60% (2171/3613)   
2021-09-15T23:13:13.2939918Z Resolving deltas:  61% (2217/3613)   
2021-09-15T23:13:13.2946163Z Resolving deltas:  62% (2256/3613)   
2021-09-15T23:13:13.2971625Z Resolving deltas:  63% (2279/3613)   
2021-09-15T23:13:13.3001836Z Resolving deltas:  64% (2337/3613)   
2021-09-15T23:13:13.3002282Z Resolving deltas:  65% (2372/3613)   
2021-09-15T23:13:13.3031712Z Resolving deltas:  66% (2385/3613)   
2021-09-15T23:13:13.3039827Z Resolving deltas:  67% (2422/3613)   
2021-09-15T23:13:13.3059516Z Resolving deltas:  68% (2461/3613)   
2021-09-15T23:13:13.3073211Z Resolving deltas:  69% (2506/3613)   
2021-09-15T23:13:13.3096176Z Resolving deltas:  70% (2531/3613)   
2021-09-15T23:13:13.3105466Z Resolving deltas:  71% (2570/3613)   
2021-09-15T23:13:13.3114306Z Resolving deltas:  72% (2603/3613)   
2021-09-15T23:13:13.3121160Z Resolving deltas:  73% (2650/3613)   
2021-09-15T23:13:13.3123295Z Resolving deltas:  74% (2699/3613)   
2021-09-15T23:13:13.3129298Z Resolving deltas:  75% (2713/3613)   
2021-09-15T23:13:13.3135657Z Resolving deltas:  76% (2755/3613)   
2021-09-15T23:13:13.3145781Z Resolving deltas:  77% (2798/3613)   
2021-09-15T23:13:13.3159410Z Resolving deltas:  78% (2832/3613)   
2021-09-15T23:13:13.3163119Z Resolving deltas:  79% (2880/3613)   
2021-09-15T23:13:13.3172565Z Resolving deltas:  80% (2897/3613)   
2021-09-15T23:13:13.3178661Z Resolving deltas:  81% (2937/3613)   
2021-09-15T23:13:13.3187178Z Resolving deltas:  82% (2964/3613)   
2021-09-15T23:13:13.3210375Z Resolving deltas:  83% (3004/3613)   
2021-09-15T23:13:13.3213065Z Resolving deltas:  84% (3057/3613)   
2021-09-15T23:13:13.3221241Z Resolving deltas:  85% (3080/3613)   
2021-09-15T23:13:13.3229629Z Resolving deltas:  86% (3118/3613)   
2021-09-15T23:13:13.3236278Z Resolving deltas:  87% (3156/3613)   
2021-09-15T23:13:13.3244592Z Resolving deltas:  88% (3180/3613)   
2021-09-15T23:13:13.3250136Z Resolving deltas:  89% (3217/3613)   
2021-09-15T23:13:13.3257568Z Resolving deltas:  90% (3255/3613)   
2021-09-15T23:13:13.3261694Z Resolving deltas:  91% (3293/3613)   
2021-09-15T23:13:13.3266877Z Resolving deltas:  92% (3325/3613)   
2021-09-15T23:13:13.3272958Z Resolving deltas:  93% (3364/3613)   
2021-09-15T23:13:13.3276444Z Resolving deltas:  94% (3398/3613)   
2021-09-15T23:13:13.3281154Z Resolving deltas:  95% (3435/3613)   
2021-09-15T23:13:13.3284783Z Resolving deltas:  96% (3473/3613)   
2021-09-15T23:13:13.3292047Z Resolving deltas:  97% (3509/3613)   
2021-09-15T23:13:13.3297688Z Resolving deltas:  98% (3541/3613)   
2021-09-15T23:13:13.3306262Z Resolving deltas:  99% (3577/3613)   
2021-09-15T23:13:13.3306689Z Resolving deltas: 100% (3613/3613)   
2021-09-15T23:13:13.3307109Z Resolving deltas: 100% (3613/3613), done.
2021-09-15T23:13:13.4165516Z From https://github.com/microsoft/LightGBM
2021-09-15T23:13:13.4167000Z  * [new branch]      ci/qemu-tests        -> origin/ci/qemu-tests
2021-09-15T23:13:13.4201968Z  * [new branch]      ci/static-security   -> origin/ci/static-security
2021-09-15T23:13:13.4205512Z  * [new branch]      feat/faster-r-cmake  -> origin/feat/faster-r-cmake
2021-09-15T23:13:13.4206657Z  * [new branch]      fix/network-setup    -> origin/fix/network-setup
2021-09-15T23:13:13.4207722Z  * [new branch]      fix/r-segfaults      -> origin/fix/r-segfaults
2021-09-15T23:13:13.4208820Z  * [new branch]      fix/split-comparison -> origin/fix/split-comparison
2021-09-15T23:13:13.4209925Z  * [new branch]      function-hints       -> origin/function-hints
2021-09-15T23:13:13.4211123Z  * [new branch]      master               -> origin/master
2021-09-15T23:13:13.4212096Z  * [new branch]      mingw_link           -> origin/mingw_link
2021-09-15T23:13:13.4213181Z  * [new branch]      r/info-setter-getter -> origin/r/info-setter-getter
2021-09-15T23:13:13.4214272Z  * [new ref]         refs/pull/4606/merge -> pull/4606/merge
2021-09-15T23:13:13.4215115Z  * [new tag]         stable               -> stable
2021-09-15T23:13:13.4215907Z  * [new tag]         v1                   -> v1
2021-09-15T23:13:13.4216660Z  * [new tag]         v2.0                 -> v2.0
2021-09-15T23:13:13.4217423Z  * [new tag]         v2.0.10              -> v2.0.10
2021-09-15T23:13:13.4218206Z  * [new tag]         v2.0.11              -> v2.0.11
2021-09-15T23:13:13.4218982Z  * [new tag]         v2.0.12              -> v2.0.12
2021-09-15T23:13:13.4219759Z  * [new tag]         v2.0.3               -> v2.0.3
2021-09-15T23:13:13.4220530Z  * [new tag]         v2.0.4               -> v2.0.4
2021-09-15T23:13:13.4221289Z  * [new tag]         v2.0.5               -> v2.0.5
2021-09-15T23:13:13.4222064Z  * [new tag]         v2.0.6               -> v2.0.6
2021-09-15T23:13:13.4222834Z  * [new tag]         v2.0.7               -> v2.0.7
2021-09-15T23:13:13.4223602Z  * [new tag]         v2.0.8               -> v2.0.8
2021-09-15T23:13:13.4224375Z  * [new tag]         v2.1.0               -> v2.1.0
2021-09-15T23:13:13.4225146Z  * [new tag]         v2.1.1               -> v2.1.1
2021-09-15T23:13:13.4225902Z  * [new tag]         v2.1.2               -> v2.1.2
2021-09-15T23:13:13.4226674Z  * [new tag]         v2.2.0               -> v2.2.0
2021-09-15T23:13:13.4227446Z  * [new tag]         v2.2.1               -> v2.2.1
2021-09-15T23:13:13.4228201Z  * [new tag]         v2.2.2               -> v2.2.2
2021-09-15T23:13:13.4228970Z  * [new tag]         v2.2.3               -> v2.2.3
2021-09-15T23:13:13.4229729Z  * [new tag]         v2.3.0               -> v2.3.0
2021-09-15T23:13:13.4230500Z  * [new tag]         v2.3.1               -> v2.3.1
2021-09-15T23:13:13.4231272Z  * [new tag]         v3.0.0               -> v3.0.0
2021-09-15T23:13:13.4232051Z  * [new tag]         v3.0.0rc1            -> v3.0.0rc1
2021-09-15T23:13:13.4232843Z  * [new tag]         v3.1.0               -> v3.1.0
2021-09-15T23:13:13.4233832Z  * [new tag]         v3.1.1               -> v3.1.1
2021-09-15T23:13:13.4234602Z  * [new tag]         v3.2.0               -> v3.2.0
2021-09-15T23:13:13.4235373Z  * [new tag]         v3.2.1               -> v3.2.1
2021-09-15T23:13:13.4322671Z ##[command]git checkout --progress --force refs/remotes/pull/4606/merge
2021-09-15T23:13:13.5349242Z Note: checking out 'refs/remotes/pull/4606/merge'.
2021-09-15T23:13:13.5349677Z 
2021-09-15T23:13:13.5351286Z You are in 'detached HEAD' state. You can look around, make experimental
2021-09-15T23:13:13.5352012Z changes and commit them, and you can discard any commits you make in this
2021-09-15T23:13:13.5352709Z state without impacting any branches by performing another checkout.
2021-09-15T23:13:13.5353130Z 
2021-09-15T23:13:13.5353619Z If you want to create a new branch to retain commits you create, you may
2021-09-15T23:13:13.5354542Z do so (now or later) by using -b with the checkout command again. Example:
2021-09-15T23:13:13.5354953Z 
2021-09-15T23:13:13.5355504Z   git checkout -b <new-branch-name>
2021-09-15T23:13:13.5355822Z 
2021-09-15T23:13:13.5356540Z HEAD is now at 198bff4 Merge 4cf98afaf4583a39b311d495e48edf66230ee132 into 54facc4d727812075b0c90e5506e521938953b25
2021-09-15T23:13:13.5364228Z ##[command]git submodule sync
2021-09-15T23:13:13.5585117Z ##[command]git -c http.https://github.saobby.my.eu.org.extraheader="AUTHORIZATION: basic ***" submodule update --init --force --depth=5
2021-09-15T23:13:13.5779104Z Submodule 'include/boost/compute' (https://github.com/boostorg/compute) registered for path 'external_libs/compute'
2021-09-15T23:13:13.5780949Z Submodule 'eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'external_libs/eigen'
2021-09-15T23:13:13.5783676Z Submodule 'external_libs/fast_double_parser' (https://github.com/lemire/fast_double_parser.git) registered for path 'external_libs/fast_double_parser'
2021-09-15T23:13:13.5786185Z Submodule 'external_libs/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'external_libs/fmt'
2021-09-15T23:13:13.5825779Z Cloning into '/home/guoke/actions-runner/_work/LightGBM/LightGBM/external_libs/compute'...
2021-09-15T23:13:14.3409109Z Cloning into '/home/guoke/actions-runner/_work/LightGBM/LightGBM/external_libs/eigen'...
2021-09-15T23:13:15.2444503Z Cloning into '/home/guoke/actions-runner/_work/LightGBM/LightGBM/external_libs/fast_double_parser'...
2021-09-15T23:13:15.8402851Z Cloning into '/home/guoke/actions-runner/_work/LightGBM/LightGBM/external_libs/fmt'...
2021-09-15T23:13:16.6976810Z Submodule path 'external_libs/compute': checked out '36c89134d4013b2e5e45bc55656a18bd6141995a'
2021-09-15T23:13:18.9323150Z From https://gitlab.com/libeigen/eigen
2021-09-15T23:13:18.9324592Z  * branch            8ba1b0f41a7950dc3e1d4ed75859e36c73311235 -> FETCH_HEAD
2021-09-15T23:13:19.0821158Z Submodule path 'external_libs/eigen': checked out '8ba1b0f41a7950dc3e1d4ed75859e36c73311235'
2021-09-15T23:13:20.1265187Z From https://github.com/lemire/fast_double_parser
2021-09-15T23:13:20.1267120Z  * branch            ace60646c02dc54c57f19d644e49a61e7e7758ec -> FETCH_HEAD
2021-09-15T23:13:20.1596501Z Submodule path 'external_libs/fast_double_parser': checked out 'ace60646c02dc54c57f19d644e49a61e7e7758ec'
2021-09-15T23:13:21.4918292Z From https://github.com/fmtlib/fmt
2021-09-15T23:13:21.4919724Z  * branch            cc09f1a6798c085c325569ef466bcdcffdc266d4 -> FETCH_HEAD
2021-09-15T23:13:21.5610942Z Submodule path 'external_libs/fmt': checked out 'cc09f1a6798c085c325569ef466bcdcffdc266d4'
2021-09-15T23:13:21.5800916Z ##[group]Run export ROOT_DOCKER_FOLDER=/LightGBM
2021-09-15T23:13:21.5801643Z �[36;1mexport ROOT_DOCKER_FOLDER=/LightGBM�[0m
2021-09-15T23:13:21.5802099Z �[36;1mcat > docker.env <<EOF�[0m
2021-09-15T23:13:21.5802513Z �[36;1mGITHUB_ACTIONS=true�[0m
2021-09-15T23:13:21.5802881Z �[36;1mOS_NAME=linux�[0m
2021-09-15T23:13:21.5803251Z �[36;1mCOMPILER=gcc�[0m
2021-09-15T23:13:21.5803602Z �[36;1mTASK=cuda�[0m
2021-09-15T23:13:21.5803938Z �[36;1mMETHOD=source�[0m
2021-09-15T23:13:21.5804493Z �[36;1mCONDA_ENV=test-env�[0m
2021-09-15T23:13:21.5804877Z �[36;1mPYTHON_VERSION=3.7�[0m
2021-09-15T23:13:21.5805329Z �[36;1mBUILD_DIRECTORY=$ROOT_DOCKER_FOLDER�[0m
2021-09-15T23:13:21.5805798Z �[36;1mLGB_VER=$(head -n 1 VERSION.txt)�[0m
2021-09-15T23:13:21.5806185Z �[36;1mEOF�[0m
2021-09-15T23:13:21.5806587Z �[36;1mcat > docker-script.sh <<EOF�[0m
2021-09-15T23:13:21.5807057Z �[36;1mexport CONDA=\$HOME/miniconda�[0m
2021-09-15T23:13:21.5807526Z �[36;1mexport PATH=\$CONDA/bin:\$PATH�[0m
2021-09-15T23:13:21.5807984Z �[36;1mnvidia-smi�[0m
2021-09-15T23:13:21.5808429Z �[36;1m$ROOT_DOCKER_FOLDER/.ci/setup.sh || exit -1�[0m
2021-09-15T23:13:21.5808944Z �[36;1m$ROOT_DOCKER_FOLDER/.ci/test.sh || exit -1�[0m
2021-09-15T23:13:21.5809347Z �[36;1mEOF�[0m
2021-09-15T23:13:21.5810165Z �[36;1mdocker run --env-file docker.env -v "$GITHUB_WORKSPACE":"$ROOT_DOCKER_FOLDER" --rm --gpus all "nvcr.io/nvidia/cuda:11.4.0-devel" /bin/bash $ROOT_DOCKER_FOLDER/docker-script.sh�[0m
2021-09-15T23:13:21.5829304Z shell: /bin/bash -e {0}
2021-09-15T23:13:21.5829653Z env:
2021-09-15T23:13:21.5829977Z   github_actions: true
2021-09-15T23:13:21.5830337Z   os_name: linux
2021-09-15T23:13:21.5830642Z   task: cuda
2021-09-15T23:13:21.5830991Z   conda_env: test-env
2021-09-15T23:13:21.5831340Z ##[endgroup]
2021-09-15T23:13:26.0322704Z docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown.
2021-09-15T23:13:26.0359071Z ##[error]Process completed with exit code 125.
2021-09-15T23:13:26.0388144Z Cleaning up orphan processes

Neither reinstallation (sudo apt-get install --reinstall --no-install-recommends -y) of all docker-related stuff, nor pining versions of libnvidia-container1 and nvidia-container-toolkit from a suggestion from internet helped.

LightGBM/.github/workflows/cuda.yml

Lines 54 to 58 in 54facc4

sudo apt-get install --no-install-recommends -y \
containerd.io \
docker-ce \
docker-ce-cli \
nvidia-docker2

I guess the first thing to try is to update NVIDIA driver with machine rebooting.
@StrikerRUS StrikerRUS mentioned this issue Sep 17, 2021
Merged
@StrikerRUS
Copy link
Collaborator Author

Fixed via #4621.

@github-actions
Copy link

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@shiyu1994 shiyu1994

Labels
maintenance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@shiyu1994 @StrikerRUS