Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH action does not support Start Stop #8052

Closed
medyagh opened this issue May 9, 2020 · 6 comments
Closed

GH action does not support Start Stop #8052

medyagh opened this issue May 9, 2020 · 6 comments
Labels
area/testing co/docker-driver Issues related to kubernetes in container co/runtime/docker Issues specific to a docker runtime kind/bug Categorizes issue or PR as related to a bug. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.

Comments

@medyagh
Copy link
Member

medyagh commented May 9, 2020

this happens only in github actions when trying to start a stopped kic. as seen in this PR: https://github.com/kubernetes/minikube/pull/8025/checks?check_run_id=658120043

Failed to bump fs.file-max, ignoring: Invalid argument
2020-05-09T01:13:33.3941636Z         	 Failed to attach 1 to compat systemd cgroup /actions_job/7f8719abd6da75f05773821619c61449d06518cd14536021b8f5535573d030c5/init.s

2020-05-09T01:09:41.2419720Z ##[section]Starting: Request a runner to run this job
2020-05-09T01:09:41.8238530Z Requesting a self-hosted runner in current repository with labels: 'ubuntu-18.04', require runner match: False
2020-05-09T01:09:42.3813529Z Requesting a hosted runner in current repository's account/organization with labels: 'ubuntu-18.04', require runner match: True
2020-05-09T01:09:42.4804128Z Labels matched hosted runners has been found, waiting for one of them get assigned for this job.
2020-05-09T01:09:42.5645959Z ##[section]Finishing: Request a runner to run this job
2020-05-09T01:09:50.5181280Z Current runner version: '2.169.1'
2020-05-09T01:09:50.5208181Z ##[group]Operating System
2020-05-09T01:09:50.5208851Z Ubuntu
2020-05-09T01:09:50.5209007Z 18.04.4
2020-05-09T01:09:50.5209207Z LTS
2020-05-09T01:09:50.5209409Z ##[endgroup]
2020-05-09T01:09:50.5209613Z ##[group]Virtual Environment
2020-05-09T01:09:50.5209788Z Environment: ubuntu-18.04
2020-05-09T01:09:50.5210009Z Version: 20200430.1
2020-05-09T01:09:50.5210300Z Included Software: https://github.com/actions/virtual-environments/blob/ubuntu18/20200430.1/images/linux/Ubuntu1804-README.md
2020-05-09T01:09:50.5210550Z ##[endgroup]
2020-05-09T01:09:50.5211462Z Prepare workflow directory
2020-05-09T01:09:50.5583846Z Prepare all required actions
2020-05-09T01:09:50.5595013Z Download action repository 'actions/download-artifact@v1'
2020-05-09T01:09:51.8152969Z Download action repository 'actions/upload-artifact@v1'
2020-05-09T01:09:52.8379833Z ##[group]Run curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.18.0/bin/linux/amd64/kubectl
2020-05-09T01:09:52.8380188Z �[36;1mcurl -LO https://storage.googleapis.com/kubernetes-release/release/v1.18.0/bin/linux/amd64/kubectl�[0m
2020-05-09T01:09:52.8380407Z �[36;1msudo install kubectl /usr/local/bin/kubectl�[0m
2020-05-09T01:09:52.8380587Z �[36;1mkubectl version --client=true�[0m
2020-05-09T01:09:52.8419432Z shell: /bin/bash --noprofile --norc -e -o pipefail {0}
2020-05-09T01:09:52.8419761Z env:
2020-05-09T01:09:52.8419960Z   GOPROXY: https://proxy.golang.org
2020-05-09T01:09:52.8420135Z   TIME_ELAPSED: time
2020-05-09T01:09:52.8420274Z   JOB_NAME: start_stop_embed_cert_docker_ubuntu
2020-05-09T01:09:52.8420428Z   GOPOGH_RESULT: 
2020-05-09T01:09:52.8420580Z   SHELL: /bin/bash
2020-05-09T01:09:52.8420729Z ##[endgroup]
2020-05-09T01:09:53.7999825Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-05-09T01:09:53.8000376Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-05-09T01:09:53.8000489Z 
2020-05-09T01:09:53.8008822Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-05-09T01:09:53.8009396Z 100 41.9M  100 41.9M    0     0  80.4M      0 --:--:-- --:--:-- --:--:-- 80.4M
2020-05-09T01:09:53.8010102Z Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
2020-05-09T01:09:53.8080538Z ##[group]Run sudo apt-get update -qq
2020-05-09T01:09:53.8080751Z �[36;1msudo apt-get update -qq�[0m
2020-05-09T01:09:53.8080914Z �[36;1msudo apt-get -qq -y install liblz4-tool�[0m
2020-05-09T01:09:53.8118417Z shell: /bin/bash --noprofile --norc -e -o pipefail {0}
2020-05-09T01:09:53.8118578Z env:
2020-05-09T01:09:53.8118787Z   GOPROXY: https://proxy.golang.org
2020-05-09T01:09:53.8118965Z   TIME_ELAPSED: time
2020-05-09T01:09:53.8119122Z   JOB_NAME: start_stop_embed_cert_docker_ubuntu
2020-05-09T01:09:53.8119268Z   GOPOGH_RESULT: 
2020-05-09T01:09:53.8119436Z   SHELL: /bin/bash
2020-05-09T01:09:53.8119586Z ##[endgroup]
2020-05-09T01:10:03.3001816Z Selecting previously unselected package liblz4-tool.
2020-05-09T01:10:03.3005997Z (Reading database ... 
2020-05-09T01:10:03.3007836Z (Reading database ... 5%
2020-05-09T01:10:03.3009752Z (Reading database ... 10%
2020-05-09T01:10:03.3127546Z (Reading database ... 15%
2020-05-09T01:10:03.3131063Z (Reading database ... 20%
2020-05-09T01:10:03.3132341Z (Reading database ... 25%
2020-05-09T01:10:03.3132477Z (Reading database ... 30%
2020-05-09T01:10:03.3132664Z (Reading database ... 35%
2020-05-09T01:10:03.3132781Z (Reading database ... 40%
2020-05-09T01:10:03.3132877Z (Reading database ... 45%
2020-05-09T01:10:03.3132993Z (Reading database ... 50%
2020-05-09T01:10:03.3133104Z (Reading database ... 55%
2020-05-09T01:10:03.3133215Z (Reading database ... 60%
2020-05-09T01:10:03.3133449Z (Reading database ... 65%
2020-05-09T01:10:03.3133564Z (Reading database ... 70%
2020-05-09T01:10:03.3133677Z (Reading database ... 75%
2020-05-09T01:10:03.3134114Z (Reading database ... 80%
2020-05-09T01:10:03.3568519Z (Reading database ... 85%
2020-05-09T01:10:03.4460480Z (Reading database ... 90%
2020-05-09T01:10:03.5290035Z (Reading database ... 95%
2020-05-09T01:10:03.5290869Z (Reading database ... 100%
2020-05-09T01:10:03.5349805Z (Reading database ... 277214 files and directories currently installed.)
2020-05-09T01:10:03.5423649Z Preparing to unpack .../liblz4-tool_0.0~r131-2ubuntu3_amd64.deb ...
2020-05-09T01:10:03.5455747Z Unpacking liblz4-tool (0.0~r131-2ubuntu3) ...
2020-05-09T01:10:03.6286496Z Setting up liblz4-tool (0.0~r131-2ubuntu3) ...
2020-05-09T01:10:03.6335448Z Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
2020-05-09T01:10:05.9735022Z ##[group]Run echo "--------------------------"
2020-05-09T01:10:05.9735414Z �[36;1mecho "--------------------------"�[0m
2020-05-09T01:10:05.9735574Z �[36;1mdocker version || true�[0m
2020-05-09T01:10:05.9735704Z �[36;1mecho "--------------------------"�[0m
2020-05-09T01:10:05.9736025Z �[36;1mdocker info || true�[0m
2020-05-09T01:10:05.9737258Z �[36;1mecho "--------------------------"�[0m
2020-05-09T01:10:05.9739547Z �[36;1mdocker system df || true�[0m
2020-05-09T01:10:05.9739930Z �[36;1mecho "--------------------------"�[0m
2020-05-09T01:10:05.9740225Z �[36;1mdocker system info || true�[0m
2020-05-09T01:10:05.9740373Z �[36;1mecho "--------------------------"�[0m
2020-05-09T01:10:05.9740514Z �[36;1mdocker ps || true�[0m
2020-05-09T01:10:05.9740654Z �[36;1mecho "--------------------------"�[0m
2020-05-09T01:10:05.9811532Z shell: /bin/bash --noprofile --norc -e -o pipefail {0}
2020-05-09T01:10:05.9811725Z env:
2020-05-09T01:10:05.9812054Z   GOPROXY: https://proxy.golang.org
2020-05-09T01:10:05.9812351Z   TIME_ELAPSED: time
2020-05-09T01:10:05.9812502Z   JOB_NAME: start_stop_embed_cert_docker_ubuntu
2020-05-09T01:10:05.9812625Z   GOPOGH_RESULT: 
2020-05-09T01:10:05.9812742Z   SHELL: /bin/bash
2020-05-09T01:10:05.9812841Z ##[endgroup]
2020-05-09T01:10:05.9888910Z --------------------------
2020-05-09T01:10:06.4887879Z Client:
2020-05-09T01:10:06.4888285Z  Version:           3.0.11+azure
2020-05-09T01:10:06.4888413Z  API version:       1.40
2020-05-09T01:10:06.4888529Z  Go version:        go1.12.17
2020-05-09T01:10:06.4888646Z  Git commit:        eb310fca49568dccd87c6136f774ef6fff2a1b51
2020-05-09T01:10:06.4888759Z  Built:             Tue Mar  3 21:59:52 2020
2020-05-09T01:10:06.4888874Z  OS/Arch:           linux/amd64
2020-05-09T01:10:06.4888984Z  Experimental:      false
2020-05-09T01:10:06.4889040Z 
2020-05-09T01:10:06.4889143Z Server:
2020-05-09T01:10:06.4889249Z  Engine:
2020-05-09T01:10:06.4889340Z   Version:          3.0.11+azure
2020-05-09T01:10:06.4889480Z   API version:      1.40 (minimum version 1.12)
2020-05-09T01:10:06.4889595Z   Go version:       go1.12.17
2020-05-09T01:10:06.4889709Z   Git commit:       aa6a9891b0
2020-05-09T01:10:06.4889822Z   Built:            Tue Mar 10 18:53:36 2020
2020-05-09T01:10:06.4889921Z   OS/Arch:          linux/amd64
2020-05-09T01:10:06.4890038Z   Experimental:     false
2020-05-09T01:10:06.4890146Z  containerd:
2020-05-09T01:10:06.4890254Z   Version:          v1.2.13
2020-05-09T01:10:06.4890355Z   GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
2020-05-09T01:10:06.4890471Z  runc:
2020-05-09T01:10:06.4891174Z   Version:          spec: 1.0.1-dev
2020-05-09T01:10:06.4891288Z   GitCommit:        
2020-05-09T01:10:06.4891513Z  docker-init:
2020-05-09T01:10:06.4891621Z   Version:          0.18.0
2020-05-09T01:10:06.4891730Z   GitCommit:        fec3683
2020-05-09T01:10:06.4906226Z --------------------------
2020-05-09T01:10:09.0581860Z Client:
2020-05-09T01:10:09.0594510Z  Debug Mode: false
2020-05-09T01:10:09.0594644Z  Plugins:
2020-05-09T01:10:09.0594815Z   buildx: Build with BuildKit (Docker Inc., 0.3.1+azure)
2020-05-09T01:10:09.0594893Z 
2020-05-09T01:10:09.0595013Z Server:
2020-05-09T01:10:09.0595130Z  Containers: 0
2020-05-09T01:10:09.0595248Z   Running: 0
2020-05-09T01:10:09.0595348Z   Paused: 0
2020-05-09T01:10:09.0595466Z   Stopped: 0
2020-05-09T01:10:09.0595859Z  Images: 14
2020-05-09T01:10:09.0595981Z  Server Version: 3.0.11+azure
2020-05-09T01:10:09.0596088Z  Storage Driver: overlay2
2020-05-09T01:10:09.0596210Z   Backing Filesystem: <unknown>
2020-05-09T01:10:09.0596332Z   Supports d_type: true
2020-05-09T01:10:09.0596455Z   Native Overlay Diff: false
2020-05-09T01:10:09.0597172Z  Logging Driver: json-file
2020-05-09T01:10:09.0597301Z  Cgroup Driver: cgroupfs
2020-05-09T01:10:09.0597421Z  Plugins:
2020-05-09T01:10:09.0597540Z   Volume: local
2020-05-09T01:10:09.0597647Z   Network: bridge host ipvlan macvlan null overlay
2020-05-09T01:10:09.0598023Z   Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
2020-05-09T01:10:09.0598307Z  Swarm: inactive
2020-05-09T01:10:09.0598438Z  Runtimes: runc
2020-05-09T01:10:09.0598542Z  Default Runtime: runc
2020-05-09T01:10:09.0598834Z  Init Binary: docker-init
2020-05-09T01:10:09.0600005Z  containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
2020-05-09T01:10:09.0600152Z  runc version: 
2020-05-09T01:10:09.0600253Z  init version: fec3683
2020-05-09T01:10:09.0600374Z  Security Options:
2020-05-09T01:10:09.0600491Z   apparmor
2020-05-09T01:10:09.0600607Z   seccomp
2020-05-09T01:10:09.0600710Z    Profile: default
2020-05-09T01:10:09.0601076Z  Kernel Version: 5.3.0-1020-azure
2020-05-09T01:10:09.0601205Z  Operating System: Ubuntu 18.04.4 LTS
2020-05-09T01:10:09.0601325Z  OSType: linux
2020-05-09T01:10:09.0601444Z  Architecture: x86_64
2020-05-09T01:10:09.0601547Z  CPUs: 2
2020-05-09T01:10:09.0601666Z  Total Memory: 6.765GiB
2020-05-09T01:10:09.0601931Z  Name: fv-az74
2020-05-09T01:10:09.0602066Z  ID: 45QP:JIYB:YQTS:MLSJ:7GWE:V2RU:KUIX:RNZZ:LTLJ:CMSZ:M473:W3HR
2020-05-09T01:10:09.0602193Z  Docker Root Dir: /var/lib/docker
2020-05-09T01:10:09.0602317Z  Debug Mode: false
2020-05-09T01:10:09.0602708Z  Registry: https://index.docker.io/v1/
2020-05-09T01:10:09.0602858Z  Labels:
2020-05-09T01:10:09.0602962Z  Experimental: false
2020-05-09T01:10:09.0603081Z  Insecure Registries:
2020-05-09T01:10:09.0603207Z   127.0.0.0/8
2020-05-09T01:10:09.0603323Z  Live Restore Enabled: false
2020-05-09T01:10:09.0603390Z 
2020-05-09T01:10:09.0625509Z --------------------------
2020-05-09T01:10:09.1014720Z TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
2020-05-09T01:10:09.1016214Z Images              14                  0                   2.761GB             2.761GB (100%)
2020-05-09T01:10:09.1016415Z Containers          0                   0                   0B                  0B
2020-05-09T01:10:09.1016565Z Local Volumes       0                   0                   0B                  0B
2020-05-09T01:10:09.1016722Z Build Cache         0                   0                   0B                  0B
2020-05-09T01:10:09.1030571Z --------------------------
2020-05-09T01:10:09.1535433Z Client:
2020-05-09T01:10:09.1546272Z  Debug Mode: false
2020-05-09T01:10:09.1549575Z  Plugins:
2020-05-09T01:10:09.1549699Z   buildx: Build with BuildKit (Docker Inc., 0.3.1+azure)
2020-05-09T01:10:09.1549784Z 
2020-05-09T01:10:09.1549891Z Server:
2020-05-09T01:10:09.1549995Z  Containers: 0
2020-05-09T01:10:09.1550101Z   Running: 0
2020-05-09T01:10:09.1550189Z   Paused: 0
2020-05-09T01:10:09.1550293Z   Stopped: 0
2020-05-09T01:10:09.1550398Z  Images: 14
2020-05-09T01:10:09.1550489Z  Server Version: 3.0.11+azure
2020-05-09T01:10:09.1550600Z  Storage Driver: overlay2
2020-05-09T01:10:09.1550707Z   Backing Filesystem: <unknown>
2020-05-09T01:10:09.1550814Z   Supports d_type: true
2020-05-09T01:10:09.1550907Z   Native Overlay Diff: false
2020-05-09T01:10:09.1551576Z  Logging Driver: json-file
2020-05-09T01:10:09.1551687Z  Cgroup Driver: cgroupfs
2020-05-09T01:10:09.1551805Z  Plugins:
2020-05-09T01:10:09.1551894Z   Volume: local
2020-05-09T01:10:09.1552005Z   Network: bridge host ipvlan macvlan null overlay
2020-05-09T01:10:09.1552321Z   Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
2020-05-09T01:10:09.1552447Z  Swarm: inactive
2020-05-09T01:10:09.1552783Z  Runtimes: runc
2020-05-09T01:10:09.1552872Z  Default Runtime: runc
2020-05-09T01:10:09.1553121Z  Init Binary: docker-init
2020-05-09T01:10:09.1553238Z  containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
2020-05-09T01:10:09.1553352Z  runc version: 
2020-05-09T01:10:09.1553443Z  init version: fec3683
2020-05-09T01:10:09.1553547Z  Security Options:
2020-05-09T01:10:09.1553650Z   apparmor
2020-05-09T01:10:09.1553740Z   seccomp
2020-05-09T01:10:09.1553844Z    Profile: default
2020-05-09T01:10:09.1554096Z  Kernel Version: 5.3.0-1020-azure
2020-05-09T01:10:09.1554209Z  Operating System: Ubuntu 18.04.4 LTS
2020-05-09T01:10:09.1554301Z  OSType: linux
2020-05-09T01:10:09.1554903Z  Architecture: x86_64
2020-05-09T01:10:09.1555027Z  CPUs: 2
2020-05-09T01:10:09.1555131Z  Total Memory: 6.765GiB
2020-05-09T01:10:09.1555375Z  Name: fv-az74
2020-05-09T01:10:09.1555531Z  ID: 45QP:JIYB:YQTS:MLSJ:7GWE:V2RU:KUIX:RNZZ:LTLJ:CMSZ:M473:W3HR
2020-05-09T01:10:09.1555649Z  Docker Root Dir: /var/lib/docker
2020-05-09T01:10:09.1555763Z  Debug Mode: false
2020-05-09T01:10:09.1555947Z  Registry: https://index.docker.io/v1/
2020-05-09T01:10:09.1556062Z  Labels:
2020-05-09T01:10:09.1556166Z  Experimental: false
2020-05-09T01:10:09.1556272Z  Insecure Registries:
2020-05-09T01:10:09.1556376Z   127.0.0.0/8
2020-05-09T01:10:09.1556466Z  Live Restore Enabled: false
2020-05-09T01:10:09.1556541Z 
2020-05-09T01:10:09.1557204Z --------------------------
2020-05-09T01:10:09.1818857Z CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
2020-05-09T01:10:09.1832195Z --------------------------
2020-05-09T01:10:09.1858464Z ##[group]Run curl -LO https://github.com/medyagh/gopogh/releases/download/v0.1.19/gopogh-linux-amd64
2020-05-09T01:10:09.1858738Z �[36;1mcurl -LO https://github.com/medyagh/gopogh/releases/download/v0.1.19/gopogh-linux-amd64�[0m
2020-05-09T01:10:09.1858884Z �[36;1msudo install gopogh-linux-amd64 /usr/local/bin/gopogh�[0m
2020-05-09T01:10:09.1895520Z shell: /bin/bash --noprofile --norc -e -o pipefail {0}
2020-05-09T01:10:09.1895627Z env:
2020-05-09T01:10:09.1895780Z   GOPROXY: https://proxy.golang.org
2020-05-09T01:10:09.1895908Z   TIME_ELAPSED: time
2020-05-09T01:10:09.1896016Z   JOB_NAME: start_stop_embed_cert_docker_ubuntu
2020-05-09T01:10:09.1896110Z   GOPOGH_RESULT: 
2020-05-09T01:10:09.1896211Z   SHELL: /bin/bash
2020-05-09T01:10:09.1896314Z ##[endgroup]
2020-05-09T01:10:09.2029843Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-05-09T01:10:09.2032758Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-05-09T01:10:09.2033228Z 
2020-05-09T01:10:09.4071927Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-05-09T01:10:09.4072446Z 100   631  100   631    0     0   3093      0 --:--:-- --:--:-- --:--:--  3093
2020-05-09T01:10:09.5016943Z 
2020-05-09T01:10:10.4677303Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-05-09T01:10:10.4690582Z  97 10.0M   97  9.7M    0     0  7929k      0  0:00:01  0:00:01 --:--:-- 10.1M
2020-05-09T01:10:10.4692086Z 100 10.0M  100 10.0M    0     0  8091k      0  0:00:01  0:00:01 --:--:-- 10.3M
2020-05-09T01:10:10.5238543Z ##[group]Run actions/download-artifact@v1
2020-05-09T01:10:10.5238679Z with:
2020-05-09T01:10:10.5238797Z   name: minikube_binaries
2020-05-09T01:10:10.5238908Z env:
2020-05-09T01:10:10.5239074Z   GOPROXY: https://proxy.golang.org
2020-05-09T01:10:10.5239211Z   TIME_ELAPSED: time
2020-05-09T01:10:10.5239322Z   JOB_NAME: start_stop_embed_cert_docker_ubuntu
2020-05-09T01:10:10.5239434Z   GOPOGH_RESULT: 
2020-05-09T01:10:10.5239527Z   SHELL: /bin/bash
2020-05-09T01:10:10.5239632Z ##[endgroup]
2020-05-09T01:10:10.7578842Z Downloading artifact 'minikube_binaries' to: '/home/runner/work/minikube/minikube/minikube_binaries'
2020-05-09T01:10:11.4392574Z 6 folders created.
2020-05-09T01:10:16.4468096Z Total file: 18 ---- Downloaded file: 5 (27%)
2020-05-09T01:10:20.4629581Z 18 files download succeed.
2020-05-09T01:10:20.4699607Z Artifact download finished.
2020-05-09T01:10:20.4735720Z ##[group]Run cd minikube_binaries
2020-05-09T01:10:20.4735958Z �[36;1mcd minikube_binaries�[0m
2020-05-09T01:10:20.4736068Z �[36;1mmkdir -p report�[0m
2020-05-09T01:10:20.4736162Z �[36;1mmkdir -p testhome�[0m
2020-05-09T01:10:20.4736268Z �[36;1mchmod a+x e2e-*�[0m
2020-05-09T01:10:20.4736372Z �[36;1mchmod a+x minikube-*�[0m
2020-05-09T01:10:20.4736486Z �[36;1msudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/�[0m
2020-05-09T01:10:20.4736600Z �[36;1msudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld�[0m
2020-05-09T01:10:20.4736715Z �[36;1mSTART_TIME=$(date -u +%s)�[0m
2020-05-09T01:10:20.4736876Z �[36;1mKUBECONFIG=$(pwd)/testhome/kubeconfig MINIKUBE_HOME=$(pwd)/testhome ./e2e-linux-amd64 -minikube-start-args=--driver=docker  -test.run "TestStartStop/group/embed-certs" -test.timeout=10m -test.v -timeout-multiplier=1 -binary=./minikube-linux-amd64 2>&1 | tee ./report/testout.txt�[0m
2020-05-09T01:10:20.4737038Z �[36;1mEND_TIME=$(date -u +%s)�[0m
2020-05-09T01:10:20.4737159Z �[36;1mTIME_ELAPSED=$(($END_TIME-$START_TIME))�[0m
2020-05-09T01:10:20.4737276Z �[36;1mmin=$((${TIME_ELAPSED}/60))�[0m
2020-05-09T01:10:20.4737385Z �[36;1msec=$((${TIME_ELAPSED}%60))�[0m
2020-05-09T01:10:20.4737490Z �[36;1mTIME_ELAPSED="${min} min $sec seconds "�[0m
2020-05-09T01:10:20.4737590Z �[36;1mecho ::set-env name=TIME_ELAPSED::${TIME_ELAPSED}�[0m
2020-05-09T01:10:20.4774338Z shell: /bin/bash {0}
2020-05-09T01:10:20.4774441Z env:
2020-05-09T01:10:20.4774610Z   GOPROXY: https://proxy.golang.org
2020-05-09T01:10:20.4774741Z   TIME_ELAPSED: time
2020-05-09T01:10:20.4774850Z   JOB_NAME: start_stop_embed_cert_docker_ubuntu
2020-05-09T01:10:20.4774948Z   GOPOGH_RESULT: 
2020-05-09T01:10:20.4775051Z   SHELL: /bin/bash
2020-05-09T01:10:20.4775226Z ##[endgroup]
2020-05-09T01:10:20.6447225Z === RUN   TestStartStop
2020-05-09T01:10:20.6448472Z === PAUSE TestStartStop
2020-05-09T01:10:20.6449150Z === CONT  TestStartStop
2020-05-09T01:10:20.6449318Z === RUN   TestStartStop/group
2020-05-09T01:10:20.6450080Z === RUN   TestStartStop/group/embed-certs
2020-05-09T01:10:20.6450379Z === PAUSE TestStartStop/group/embed-certs
2020-05-09T01:10:20.6450729Z === CONT  TestStartStop/group/embed-certs
2020-05-09T01:10:20.6453268Z     TestStartStop/group/embed-certs: start_stop_delete_test.go:101: (dbg) Run:  ./minikube-linux-amd64 start -p embed-certs-20200509011020-5179 --memory=2200 --alsologtostderr --wait=true --embed-certs --driver=docker --kubernetes-version=v1.18.1
2020-05-09T01:11:52.3788620Z     TestStartStop/group/embed-certs: start_stop_delete_test.go:101: (dbg) Done: ./minikube-linux-amd64 start -p embed-certs-20200509011020-5179 --memory=2200 --alsologtostderr --wait=true --embed-certs --driver=docker --kubernetes-version=v1.18.1: (1m31.730898699s)
2020-05-09T01:11:52.3794092Z     TestStartStop/group/embed-certs: start_stop_delete_test.go:107: (dbg) Run:  kubectl --context embed-certs-20200509011020-5179 create -f testdata/busybox.yaml
2020-05-09T01:11:52.6517676Z     TestStartStop/group/embed-certs: start_stop_delete_test.go:107: (dbg) TestStartStop/group/embed-certs: waiting 8m0s for pods matching "integration-test=busybox" in namespace "default" ...
2020-05-09T01:11:52.6706045Z     TestStartStop/group/embed-certs: helpers.go:320: "busybox" [89238be3-8e3c-4640-bb30-6eabf061a984] Pending
2020-05-09T01:11:53.6733767Z     TestStartStop/group/embed-certs: helpers.go:320: "busybox" [89238be3-8e3c-4640-bb30-6eabf061a984] Pending / Ready:ContainersNotReady (containers with unready status: [busybox]) / ContainersReady:ContainersNotReady (containers with unready status: [busybox])
2020-05-09T01:11:55.6743757Z     TestStartStop/group/embed-certs: helpers.go:320: "busybox" [89238be3-8e3c-4640-bb30-6eabf061a984] Running
2020-05-09T01:12:01.6788148Z     TestStartStop/group/embed-certs: start_stop_delete_test.go:107: (dbg) TestStartStop/group/embed-certs: integration-test=busybox healthy within 9.026066282s
2020-05-09T01:12:01.6789151Z     TestStartStop/group/embed-certs: start_stop_delete_test.go:107: (dbg) Run:  kubectl --context embed-certs-20200509011020-5179 exec busybox -- /bin/sh -c "ulimit -n"
2020-05-09T01:12:01.8697103Z     TestStartStop/group/embed-certs: start_stop_delete_test.go:110: (dbg) Run:  ./minikube-linux-amd64 stop -p embed-certs-20200509011020-5179 --alsologtostderr -v=3
2020-05-09T01:12:12.8242734Z     TestStartStop/group/embed-certs: start_stop_delete_test.go:110: (dbg) Done: ./minikube-linux-amd64 stop -p embed-certs-20200509011020-5179 --alsologtostderr -v=3: (10.95445142s)
2020-05-09T01:12:12.8243704Z     TestStartStop/group/embed-certs: start_stop_delete_test.go:117: (dbg) Run:  ./minikube-linux-amd64 status --format={{.Host}} -p embed-certs-20200509011020-5179
2020-05-09T01:12:12.9261438Z     TestStartStop/group/embed-certs: start_stop_delete_test.go:117: (dbg) Non-zero exit: ./minikube-linux-amd64 status --format={{.Host}} -p embed-certs-20200509011020-5179: exit status 7 (101.768771ms)
2020-05-09T01:12:12.9263128Z         
2020-05-09T01:12:12.9264076Z         -- stdout --
2020-05-09T01:12:12.9264631Z         	Stopped
2020-05-09T01:12:12.9264929Z         
2020-05-09T01:12:12.9265823Z         -- /stdout --
2020-05-09T01:12:12.9266163Z     TestStartStop/group/embed-certs: start_stop_delete_test.go:117: status error: exit status 7 (may be ok)
2020-05-09T01:12:12.9266618Z     TestStartStop/group/embed-certs: start_stop_delete_test.go:124: (dbg) Run:  ./minikube-linux-amd64 addons enable dashboard -p embed-certs-20200509011020-5179
2020-05-09T01:12:13.0317877Z     TestStartStop/group/embed-certs: start_stop_delete_test.go:129: (dbg) Run:  ./minikube-linux-amd64 start -p embed-certs-20200509011020-5179 --memory=2200 --alsologtostderr --wait=true --embed-certs --driver=docker --kubernetes-version=v1.18.1
2020-05-09T01:13:33.3451305Z     TestStartStop/group/embed-certs: start_stop_delete_test.go:129: (dbg) Non-zero exit: ./minikube-linux-amd64 start -p embed-certs-20200509011020-5179 --memory=2200 --alsologtostderr --wait=true --embed-certs --driver=docker --kubernetes-version=v1.18.1: exit status 70 (1m20.310175186s)
2020-05-09T01:13:33.3451975Z         
2020-05-09T01:13:33.3452530Z         -- stdout --
2020-05-09T01:13:33.3453473Z         	* [embed-certs-20200509011020-5179] minikube v1.10.0-beta.2 on Ubuntu 18.04
2020-05-09T01:13:33.3455376Z         	  - MINIKUBE_HOME=/home/runner/work/minikube/minikube/minikube_binaries/testhome
2020-05-09T01:13:33.3455995Z         	  - KUBECONFIG=/home/runner/work/minikube/minikube/minikube_binaries/testhome/kubeconfig
2020-05-09T01:13:33.3456860Z         	* Using the docker driver based on existing profile
2020-05-09T01:13:33.3457411Z         	* Starting control plane node embed-certs-20200509011020-5179 in cluster embed-certs-20200509011020-5179
2020-05-09T01:13:33.3457975Z         	* Restarting existing docker container for "embed-certs-20200509011020-5179" ...
2020-05-09T01:13:33.3458509Z         	* Restarting existing docker container for "embed-certs-20200509011020-5179" ...
2020-05-09T01:13:33.3458879Z         
2020-05-09T01:13:33.3463170Z         -- /stdout --
2020-05-09T01:13:33.3463592Z         ** stderr ** 
2020-05-09T01:13:33.3463966Z         	W0509 01:12:13.073471   10966 root.go:248] Error reading config file at /home/runner/work/minikube/minikube/minikube_binaries/testhome/.minikube/config/config.json: open /home/runner/work/minikube/minikube/minikube_binaries/testhome/.minikube/config/config.json: no such file or directory
2020-05-09T01:13:33.3464996Z         	I0509 01:12:13.075221   10966 start.go:99] hostinfo: {"hostname":"fv-az74","uptime":191,"bootTime":1588986542,"procs":217,"os":"linux","platform":"ubuntu","platformFamily":"debian","platformVersion":"18.04","kernelVersion":"5.3.0-1020-azure","virtualizationSystem":"","virtualizationRole":"","hostid":"5803e24b-ce86-1d46-b9d7-13b3951f7343"}
2020-05-09T01:13:33.3465632Z         	I0509 01:12:13.075548   10966 start.go:109] virtualization:  
2020-05-09T01:13:33.3465930Z         	I0509 01:12:13.077235   10966 notify.go:125] Checking for updates...
2020-05-09T01:13:33.3466564Z         	I0509 01:12:13.077840   10966 driver.go:253] Setting default libvirt URI to qemu:///system
2020-05-09T01:13:33.3467159Z         	I0509 01:12:13.121927   10966 docker.go:91] docker version: linux-3.0.11+azure
2020-05-09T01:13:33.3467500Z         	I0509 01:12:13.122864   10966 start.go:215] selected driver: docker
2020-05-09T01:13:33.3469684Z         	I0509 01:12:13.122870   10966 start.go:594] validating driver "docker" against &{Name:embed-certs-20200509011020-5179 KeepContext:false EmbedCerts:true MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.10@sha256:f58e0c4662bac8a9b5dda7984b185bad8502ade5d9fa364bf2755d636ab51438 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio KubernetesConfig:{KubernetesVersion:v1.18.1 ClusterName:embed-certs-20200509011020-5179 APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: ExtraOptions:[{Component:kubeadm Key:pod-network-cidr Value:10.244.0.0/16}] ShouldLoadCachedImages:true EnableDefaultCNI:false NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:172.17.0.3 Port:8443 KubernetesVersion:v1.18.1 ControlPlane:true Worker:true}] Addons:map[dashboard:true default-storageclass:true storage-provisioner:true] VerifyComponents:map[apiserver:true apps_running:true default_sa:true system_pods:true]}
2020-05-09T01:13:33.3470387Z         	I0509 01:12:13.122935   10966 start.go:600] status for docker: {Installed:true Healthy:true Error:<nil> Fix: Doc:}
2020-05-09T01:13:33.3470966Z         	I0509 01:12:13.122948   10966 start.go:917] auto setting extra-config to "kubeadm.pod-network-cidr=10.244.0.0/16".
2020-05-09T01:13:33.3471293Z         	I0509 01:12:13.123020   10966 start_flags.go:576] Waiting for all components: map[apiserver:true apps_running:true default_sa:true system_pods:true]
2020-05-09T01:13:33.3475864Z         	I0509 01:12:13.123611   10966 cache.go:104] Beginning downloading kic artifacts for docker with docker
2020-05-09T01:13:33.3476704Z         	I0509 01:12:13.176115   10966 image.go:88] Found gcr.io/k8s-minikube/kicbase:v0.0.10@sha256:f58e0c4662bac8a9b5dda7984b185bad8502ade5d9fa364bf2755d636ab51438 in local docker daemon, skipping pull
2020-05-09T01:13:33.3477124Z         	I0509 01:12:13.176144   10966 preload.go:81] Checking if preload exists for k8s version v1.18.1 and runtime docker
2020-05-09T01:13:33.3482885Z         	I0509 01:12:13.176173   10966 preload.go:96] Found local preload: /home/runner/work/minikube/minikube/minikube_binaries/testhome/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v3-v1.18.1-docker-overlay2-amd64.tar.lz4
2020-05-09T01:13:33.3483425Z         	I0509 01:12:13.176181   10966 cache.go:48] Caching tarball of preloaded images
2020-05-09T01:13:33.3484074Z         	I0509 01:12:13.176193   10966 preload.go:122] Found /home/runner/work/minikube/minikube/minikube_binaries/testhome/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v3-v1.18.1-docker-overlay2-amd64.tar.lz4 in cache, skipping download
2020-05-09T01:13:33.3484477Z         	I0509 01:12:13.176198   10966 cache.go:51] Finished verifying existence of preloaded tar for  v1.18.1 on docker
2020-05-09T01:13:33.3485137Z         	I0509 01:12:13.176282   10966 profile.go:156] Saving config to /home/runner/work/minikube/minikube/minikube_binaries/testhome/.minikube/profiles/embed-certs-20200509011020-5179/config.json ...
2020-05-09T01:13:33.3485551Z         	I0509 01:12:13.176463   10966 cache.go:132] Successfully downloaded all kic artifacts
2020-05-09T01:13:33.3486467Z         	I0509 01:12:13.176488   10966 start.go:223] acquiring machines lock for embed-certs-20200509011020-5179: {Name:mk73c44599bf574013c1eab5744c5f085a8b4870 Clock:{} Delay:500ms Timeout:15m0s Cancel:<nil>}
2020-05-09T01:13:33.3487063Z         	I0509 01:12:13.176591   10966 start.go:227] acquired machines lock for "embed-certs-20200509011020-5179" in 81.602µs
2020-05-09T01:13:33.3487381Z         	I0509 01:12:13.176607   10966 start.go:87] Skipping create...Using existing machine configuration
2020-05-09T01:13:33.3487715Z         	I0509 01:12:13.176612   10966 fix.go:53] fixHost starting: 
2020-05-09T01:13:33.3488284Z         	I0509 01:12:13.176873   10966 cli_runner.go:108] Run: docker inspect embed-certs-20200509011020-5179 --format={{.State.Status}}
2020-05-09T01:13:33.3488794Z         	I0509 01:12:13.204840   10966 fix.go:105] recreateIfNeeded on embed-certs-20200509011020-5179: state=Stopped err=<nil>
2020-05-09T01:13:33.3489179Z         	W0509 01:12:13.204862   10966 fix.go:131] unexpected machine state, will restart: <nil>
2020-05-09T01:13:33.3490566Z         	I0509 01:12:13.206801   10966 exec_runner.go:49] Run: docker start embed-certs-20200509011020-5179
2020-05-09T01:13:33.3491110Z         	I0509 01:12:13.483543   10966 cli_runner.go:108] Run: docker inspect embed-certs-20200509011020-5179 --format={{.State.Status}}
2020-05-09T01:13:33.3491734Z         	I0509 01:12:13.527139   10966 kic.go:316] container "embed-certs-20200509011020-5179" state is running.
2020-05-09T01:13:33.3492187Z         	I0509 01:12:13.527230   10966 kic_runner.go:91] Run: systemctl --version
2020-05-09T01:13:33.3492698Z         	I0509 01:12:13.527243   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 systemctl --version]
2020-05-09T01:13:33.3493339Z         	I0509 01:12:13.605067   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3493896Z         	I0509 01:12:13.605089   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3494472Z         	I0509 01:12:13.679987   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3502687Z         	I0509 01:12:13.680009   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3503271Z         	I0509 01:12:13.868509   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3503894Z         	I0509 01:12:13.868528   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3504502Z         	I0509 01:12:13.898632   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3505057Z         	I0509 01:12:13.898653   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3505891Z         	I0509 01:12:13.926004   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3506861Z         	I0509 01:12:13.926022   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3507348Z         	I0509 01:12:13.960919   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3508000Z         	I0509 01:12:13.960938   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3508481Z         	I0509 01:12:13.994183   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3509136Z         	I0509 01:12:13.994205   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3509859Z         	I0509 01:12:14.025660   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3510470Z         	I0509 01:12:14.025680   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3510952Z         	I0509 01:12:14.059181   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3511654Z         	I0509 01:12:14.059203   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3512141Z         	I0509 01:12:14.095716   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3512740Z         	I0509 01:12:14.095739   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3513338Z         	I0509 01:12:14.139590   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3513932Z         	I0509 01:12:14.139613   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3514421Z         	I0509 01:12:14.194975   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3515014Z         	I0509 01:12:14.194998   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3515488Z         	I0509 01:12:14.285328   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3516149Z         	I0509 01:12:14.285360   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3516634Z         	I0509 01:12:14.359906   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3517301Z         	I0509 01:12:14.359950   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3517775Z         	I0509 01:12:14.477532   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3518365Z         	I0509 01:12:14.477559   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3518950Z         	I0509 01:12:14.627426   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3519516Z         	I0509 01:12:14.627471   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3520141Z         	I0509 01:12:14.942970   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3520734Z         	I0509 01:12:14.943001   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3521221Z         	I0509 01:12:15.230775   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3521905Z         	I0509 01:12:15.230796   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3524433Z         	I0509 01:12:15.649742   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3524889Z         	I0509 01:12:15.649768   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3525464Z         	I0509 01:12:16.550438   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3525871Z         	I0509 01:12:16.550471   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3526255Z         	I0509 01:12:17.375324   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3526713Z         	I0509 01:12:17.375357   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3527092Z         	I0509 01:12:18.572514   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3527531Z         	I0509 01:12:18.572541   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3527921Z         	I0509 01:12:20.751567   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3528315Z         	I0509 01:12:20.751593   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3528723Z         	I0509 01:12:24.786444   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3529163Z         	I0509 01:12:24.786468   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3529543Z         	I0509 01:12:32.460882   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3529996Z         	I0509 01:12:32.460914   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3530333Z         	I0509 01:12:39.165528   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3530781Z         	I0509 01:12:39.165560   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3531164Z         	I0509 01:12:49.258870   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3531599Z         	I0509 01:12:49.258905   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3531779Z         	I0509 01:12:49.288636   10966 fix.go:55] fixHost completed within 36.112016281s
2020-05-09T01:13:33.3532211Z         	I0509 01:12:49.288655   10966 start.go:74] releasing machines lock for "embed-certs-20200509011020-5179", held for 36.112054081s
2020-05-09T01:13:33.3532341Z         	! StartHost failed, but will try again: driver start: ssh service: SSHD service is not up <nil>
2020-05-09T01:13:33.3532851Z         	I0509 01:12:54.288966   10966 start.go:223] acquiring machines lock for embed-certs-20200509011020-5179: {Name:mk73c44599bf574013c1eab5744c5f085a8b4870 Clock:{} Delay:500ms Timeout:15m0s Cancel:<nil>}
2020-05-09T01:13:33.3533270Z         	I0509 01:12:54.289112   10966 start.go:227] acquired machines lock for "embed-certs-20200509011020-5179" in 91.902µs
2020-05-09T01:13:33.3533463Z         	I0509 01:12:54.289140   10966 start.go:87] Skipping create...Using existing machine configuration
2020-05-09T01:13:33.3533625Z         	I0509 01:12:54.289146   10966 fix.go:53] fixHost starting: 
2020-05-09T01:13:33.3534033Z         	I0509 01:12:54.289439   10966 cli_runner.go:108] Run: docker inspect embed-certs-20200509011020-5179 --format={{.State.Status}}
2020-05-09T01:13:33.3534387Z         	I0509 01:12:54.317177   10966 fix.go:105] recreateIfNeeded on embed-certs-20200509011020-5179: state=Stopped err=<nil>
2020-05-09T01:13:33.3534562Z         	W0509 01:12:54.317196   10966 fix.go:131] unexpected machine state, will restart: <nil>
2020-05-09T01:13:33.3535116Z         	I0509 01:12:54.318647   10966 exec_runner.go:49] Run: docker start embed-certs-20200509011020-5179
2020-05-09T01:13:33.3535558Z         	I0509 01:12:54.599615   10966 cli_runner.go:108] Run: docker inspect embed-certs-20200509011020-5179 --format={{.State.Status}}
2020-05-09T01:13:33.3535966Z         	I0509 01:12:54.650832   10966 kic.go:316] container "embed-certs-20200509011020-5179" state is running.
2020-05-09T01:13:33.3536293Z         	I0509 01:12:54.650950   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3536732Z         	I0509 01:12:54.650960   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3537160Z         	I0509 01:12:54.700176   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3537602Z         	I0509 01:12:54.700197   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3537988Z         	I0509 01:12:54.737553   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3538424Z         	I0509 01:12:54.737572   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3538819Z         	I0509 01:12:54.905940   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3539263Z         	I0509 01:12:54.905962   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3539665Z         	I0509 01:12:54.936043   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3540127Z         	I0509 01:12:54.936064   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3540516Z         	I0509 01:12:54.966792   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3540953Z         	I0509 01:12:54.966812   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3541285Z         	I0509 01:12:54.998751   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3541722Z         	I0509 01:12:54.998775   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3669966Z         	I0509 01:12:55.040994   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3670600Z         	I0509 01:12:55.041015   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3670998Z         	I0509 01:12:55.092796   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3682076Z         	I0509 01:12:55.092817   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3682491Z         	I0509 01:12:55.139252   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3683088Z         	I0509 01:12:55.139274   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3683574Z         	I0509 01:12:55.178977   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3684406Z         	I0509 01:12:55.178997   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3684884Z         	I0509 01:12:55.239080   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3685462Z         	I0509 01:12:55.239124   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3685962Z         	I0509 01:12:55.291715   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3686390Z         	I0509 01:12:55.291736   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3686892Z         	I0509 01:12:55.395647   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3687445Z         	I0509 01:12:55.395679   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3687826Z         	I0509 01:12:55.503153   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3688392Z         	I0509 01:12:55.503174   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3688746Z         	I0509 01:12:55.635938   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3689317Z         	I0509 01:12:55.635963   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3689792Z         	I0509 01:12:55.900950   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3690215Z         	I0509 01:12:55.900978   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3690728Z         	I0509 01:12:56.274498   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3691539Z         	I0509 01:12:56.274542   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3692064Z         	I0509 01:12:56.688878   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3692511Z         	I0509 01:12:56.688903   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3692999Z         	I0509 01:12:57.400687   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3693412Z         	I0509 01:12:57.400713   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3693933Z         	I0509 01:12:58.571641   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3694492Z         	I0509 01:12:58.571681   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3694857Z         	I0509 01:12:59.855919   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3695426Z         	I0509 01:12:59.855945   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3695761Z         	I0509 01:13:01.835679   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3696324Z         	I0509 01:13:01.835707   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3696993Z         	I0509 01:13:06.684072   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3697559Z         	I0509 01:13:06.684101   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3697936Z         	I0509 01:13:11.549393   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3698479Z         	I0509 01:13:11.549423   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3698834Z         	I0509 01:13:23.216916   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3699397Z         	I0509 01:13:23.216939   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3699891Z         	I0509 01:13:33.309048   10966 kic_runner.go:91] Run: sudo systemctl is-active --quiet service sshd
2020-05-09T01:13:33.3700319Z         	I0509 01:13:33.309076   10966 kic_runner.go:112] Args: [docker exec --privileged embed-certs-20200509011020-5179 sudo systemctl is-active --quiet service sshd]
2020-05-09T01:13:33.3700455Z         	I0509 01:13:33.339773   10966 fix.go:55] fixHost completed within 39.050613325s
2020-05-09T01:13:33.3700979Z         	I0509 01:13:33.339793   10966 start.go:74] releasing machines lock for "embed-certs-20200509011020-5179", held for 39.050665626s
2020-05-09T01:13:33.3701524Z         	* Failed to start docker container. "minikube start -p embed-certs-20200509011020-5179" may fix it: driver start: ssh service: SSHD service is not up <nil>
2020-05-09T01:13:33.3701696Z         	I0509 01:13:33.339940   10966 exit.go:58] WithError(error provisioning host)=Failed to start host: driver start: ssh service: SSHD service is not up <nil> called from:
2020-05-09T01:13:33.3701831Z         	goroutine 1 [running]:
2020-05-09T01:13:33.3701945Z         	runtime/debug.Stack(0x40c62a, 0x18203c0, 0x1805960)
2020-05-09T01:13:33.3702176Z         		/usr/local/go1.14/src/runtime/debug/stack.go:24 +0x9d
2020-05-09T01:13:33.3702311Z         	k8s.io/minikube/pkg/minikube/exit.WithError(0x1a756e3, 0x17, 0x1d35fc0, 0xc00071eaa0)
2020-05-09T01:13:33.3702433Z         		/home/runner/work/minikube/minikube/pkg/minikube/exit/exit.go:58 +0x34
2020-05-09T01:13:33.3702550Z         	k8s.io/minikube/cmd/minikube/cmd.runStart(0x2a60b00, 0xc000186e00, 0x0, 0x8)
2020-05-09T01:13:33.3702669Z         		/home/runner/work/minikube/minikube/cmd/minikube/cmd/start.go:170 +0xac2
2020-05-09T01:13:33.3702787Z         	github.com/spf13/cobra.(*Command).execute(0x2a60b00, 0xc000186d80, 0x8, 0x8, 0x2a60b00, 0xc000186d80)
2020-05-09T01:13:33.3703051Z         		/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:846 +0x29d
2020-05-09T01:13:33.3703180Z         	github.com/spf13/cobra.(*Command).ExecuteC(0x2a5fb40, 0x0, 0x1, 0xc000040be0)
2020-05-09T01:13:33.3703311Z         		/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x349
2020-05-09T01:13:33.3703426Z         	github.com/spf13/cobra.(*Command).Execute(...)
2020-05-09T01:13:33.3703530Z         		/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
2020-05-09T01:13:33.3703771Z         	k8s.io/minikube/cmd/minikube/cmd.Execute()
2020-05-09T01:13:33.3703891Z         		/home/runner/work/minikube/minikube/cmd/minikube/cmd/root.go:108 +0x691
2020-05-09T01:13:33.3704006Z         	main.main()
2020-05-09T01:13:33.3704102Z         		/home/runner/work/minikube/minikube/cmd/minikube/main.go:66 +0xe6
2020-05-09T01:13:33.3705011Z         	W0509 01:13:33.340077   10966 out.go:201] error provisioning host: Failed to start host: driver start: ssh service: SSHD service is not up <nil>
2020-05-09T01:13:33.3705141Z         	* 
2020-05-09T01:13:33.3705517Z         	X error provisioning host: Failed to start host: driver start: ssh service: SSHD service is not up <nil>
2020-05-09T01:13:33.3705809Z         	* 
2020-05-09T01:13:33.3705912Z         	* minikube is exiting due to an error. If the above message is not useful, open an issue:
2020-05-09T01:13:33.3706517Z         	  - https://github.com/kubernetes/minikube/issues/new/choose
2020-05-09T01:13:33.3706656Z         
2020-05-09T01:13:33.3706754Z         ** /stderr **
2020-05-09T01:13:33.3707437Z     TestStartStop/group/embed-certs: start_stop_delete_test.go:132: failed to start minikube post-stop. args "./minikube-linux-amd64 start -p embed-certs-20200509011020-5179 --memory=2200 --alsologtostderr --wait=true --embed-certs --driver=docker --kubernetes-version=v1.18.1": exit status 70
2020-05-09T01:13:33.3708039Z     TestStartStop/group/embed-certs: panic.go:617: *** TestStartStop/group/embed-certs FAILED at 2020-05-09 01:13:33.344870136 +0000 UTC m=+192.711264576
2020-05-09T01:13:33.3708413Z     TestStartStop/group/embed-certs: helpers.go:198: -----------------------post-mortem--------------------------------
2020-05-09T01:13:33.3709623Z     TestStartStop/group/embed-certs: helpers.go:201: ======>  post-mortem[TestStartStop/group/embed-certs]: docker logs <======
2020-05-09T01:13:33.3710018Z     TestStartStop/group/embed-certs: helpers.go:202: (dbg) Run:  docker logs --details embed-certs-20200509011020-5179
2020-05-09T01:13:33.3859993Z     TestStartStop/group/embed-certs: helpers.go:206: (dbg) docker logs --details embed-certs-20200509011020-5179:
2020-05-09T01:13:33.3860310Z         
2020-05-09T01:13:33.3861366Z         -- stdout --
2020-05-09T01:13:33.3861673Z         	 INFO: ensuring we can execute /bin/mount even with userns-remap
2020-05-09T01:13:33.3862107Z         	 INFO: remounting /sys read-only
2020-05-09T01:13:33.3862207Z         	 INFO: making mounts shared
2020-05-09T01:13:33.3862320Z         	 INFO: fix cgroup mounts for all subsystems
2020-05-09T01:13:33.3862735Z         	 INFO: clearing and regenerating /etc/machine-id
2020-05-09T01:13:33.3862888Z         	 Initializing machine ID from random generator.
2020-05-09T01:13:33.3862998Z         	 INFO: faking /sys/class/dmi/id/product_name to be "kind"
2020-05-09T01:13:33.3863120Z         	 INFO: faking /sys/class/dmi/id/product_uuid to be random
2020-05-09T01:13:33.3863240Z         	 INFO: faking /sys/devices/virtual/dmi/id/product_uuid as well
2020-05-09T01:13:33.3863483Z         	 INFO: setting iptables to detected mode: legacy
2020-05-09T01:13:33.3864026Z         	 systemd 242 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
2020-05-09T01:13:33.3864292Z         	 Detected virtualization docker.
2020-05-09T01:13:33.3864583Z         	 Detected architecture x86-64.
2020-05-09T01:13:33.3864690Z         	 
2020-05-09T01:13:33.3865058Z         	 Welcome to �[1mUbuntu 19.10�[0m!
2020-05-09T01:13:33.3865159Z         	 
2020-05-09T01:13:33.3865628Z         	 Set hostname to <embed-certs-20200509011020-5179>.
2020-05-09T01:13:33.3865927Z         	 Failed to bump fs.file-max, ignoring: Invalid argument
2020-05-09T01:13:33.3866545Z         	 /lib/systemd/system/docker.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/docker.sock → /run/docker.sock; please update the unit file accordingly.
2020-05-09T01:13:33.3867018Z         	 [�[0;32m  OK  �[0m] Reached target �[0;1;39mSlices�[0m.
2020-05-09T01:13:33.3867312Z         	 [�[0;32m  OK  �[0m] Listening on �[0;1;39mJournal Socket�[0m.
2020-05-09T01:13:33.3867761Z         	          Starting �[0;1;39mRemount Root and Kernel File Systems�[0m...
2020-05-09T01:13:33.3868077Z         	          Mounting �[0;1;39mKernel Debug File System�[0m...
2020-05-09T01:13:33.3868489Z         	          Mounting �[0;1;39mHuge Pages File System�[0m...
2020-05-09T01:13:33.3868861Z         	 [�[0;32m  OK  �[0m] Started �[0;1;39mDispatch Password …ts to Console Directory Watch�[0m.
2020-05-09T01:13:33.3869574Z         	          Starting �[0;1;39mApply Kernel Variables�[0m...
2020-05-09T01:13:33.3870128Z         	 [�[0;32m  OK  �[0m] Listening on �[0;1;39mJournal Socket (/dev/log)�[0m.
2020-05-09T01:13:33.3870493Z         	 [�[0;32m  OK  �[0m] Listening on �[0;1;39mJournal Audit Socket�[0m.
2020-05-09T01:13:33.3870929Z         	          Starting �[0;1;39mJournal Service�[0m...
2020-05-09T01:13:33.3871221Z         	          Mounting �[0;1;39mFUSE Control File System�[0m...
2020-05-09T01:13:33.3871699Z         	 [�[0;32m  OK  �[0m] Set up automount �[0;1;39mArbitrary…s File System Automount Point�[0m.
2020-05-09T01:13:33.3872022Z         	 [�[0;32m  OK  �[0m] Reached target �[0;1;39mPaths�[0m.
2020-05-09T01:13:33.3872478Z         	          Starting �[0;1;39mCreate list of re…odes for the current kernel�[0m...
2020-05-09T01:13:33.3872810Z         	 [�[0;32m  OK  �[0m] Reached target �[0;1;39mNetwork is Online�[0m.
2020-05-09T01:13:33.3873229Z         	 [�[0;32m  OK  �[0m] Reached target �[0;1;39mSwap�[0m.
2020-05-09T01:13:33.3873580Z         	 [�[0;32m  OK  �[0m] Reached target �[0;1;39mLocal Encrypted Volumes�[0m.
2020-05-09T01:13:33.3874039Z         	 [�[0;32m  OK  �[0m] Started �[0;1;39mRemount Root and Kernel File Systems�[0m.
2020-05-09T01:13:33.3874371Z         	 [�[0;32m  OK  �[0m] Mounted �[0;1;39mKernel Debug File System�[0m.
2020-05-09T01:13:33.3874811Z         	 [�[0;32m  OK  �[0m] Mounted �[0;1;39mHuge Pages File System�[0m.
2020-05-09T01:13:33.3875120Z         	          Starting �[0;1;39mUpdate UTMP about System Boot/Shutdown�[0m...
2020-05-09T01:13:33.3875534Z         	          Starting �[0;1;39mCreate System Users�[0m...
2020-05-09T01:13:33.3875860Z         	 [�[0;32m  OK  �[0m] Started �[0;1;39mApply Kernel Variables�[0m.
2020-05-09T01:13:33.3876284Z         	 [�[0;32m  OK  �[0m] Mounted �[0;1;39mFUSE Control File System�[0m.
2020-05-09T01:13:33.3876636Z         	 [�[0;32m  OK  �[0m] Started �[0;1;39mCreate list of req… nodes for the current kernel�[0m.
2020-05-09T01:13:33.3877083Z         	 [�[0;32m  OK  �[0m] Started �[0;1;39mUpdate UTMP about System Boot/Shutdown�[0m.
2020-05-09T01:13:33.3877433Z         	 [�[0;32m  OK  �[0m] Started �[0;1;39mCreate System Users�[0m.
2020-05-09T01:13:33.3877861Z         	          Starting �[0;1;39mCreate Static Device Nodes in /dev�[0m...
2020-05-09T01:13:33.3878209Z         	 [�[0;32m  OK  �[0m] Started �[0;1;39mCreate Static Device Nodes in /dev�[0m.
2020-05-09T01:13:33.3878634Z         	 [�[0;32m  OK  �[0m] Reached target �[0;1;39mLocal File Systems (Pre)�[0m.
2020-05-09T01:13:33.3878975Z         	 [�[0;32m  OK  �[0m] Reached target �[0;1;39mLocal File Systems�[0m.
2020-05-09T01:13:33.3879390Z         	 [�[0;32m  OK  �[0m] Started �[0;1;39mJournal Service�[0m.
2020-05-09T01:13:33.3879729Z         	          Starting �[0;1;39mFlush Journal to Persistent Storage�[0m...
2020-05-09T01:13:33.3880165Z         	 [�[0;32m  OK  �[0m] Reached target �[0;1;39mSystem Initialization�[0m.
2020-05-09T01:13:33.3880476Z         	          Starting �[0;1;39mDocker Socket for the API�[0m.
2020-05-09T01:13:33.3880924Z         	 [�[0;32m  OK  �[0m] Started �[0;1;39mDaily Cleanup of Temporary Directories�[0m.
2020-05-09T01:13:33.3881267Z         	 [�[0;32m  OK  �[0m] Reached target �[0;1;39mTimers�[0m.
2020-05-09T01:13:33.3881566Z         	 [�[0;32m  OK  �[0m] Listening on �[0;1;39mDocker Socket for the API�[0m.
2020-05-09T01:13:33.3882024Z         	 [�[0;32m  OK  �[0m] Reached target �[0;1;39mSockets�[0m.
2020-05-09T01:13:33.3882302Z         	 [�[0;32m  OK  �[0m] Reached target �[0;1;39mBasic System�[0m.
2020-05-09T01:13:33.3882754Z         	          Starting �[0;1;39mOpenBSD Secure Shell server�[0m...
2020-05-09T01:13:33.3883047Z         	          Starting �[0;1;39mcontainerd container runtime�[0m...
2020-05-09T01:13:33.3883528Z         	 [�[0;32m  OK  �[0m] Started �[0;1;39mFlush Journal to Persistent Storage�[0m.
2020-05-09T01:13:33.3883818Z         	 [�[0;32m  OK  �[0m] Started �[0;1;39mcontainerd container runtime�[0m.
2020-05-09T01:13:33.3884546Z         	          Starting �[0;1;39mDocker Application Container Engine�[0m...
2020-05-09T01:13:33.3885507Z         	 [�[0;32m  OK  �[0m] Started �[0;1;39mOpenBSD Secure Shell server�[0m.
2020-05-09T01:13:33.3886136Z         	 [�[0m�[0;31m*     �[0m] A start job is running for Docker A…n Container Engine (5s / no limit)
2020-05-09T01:13:33.3886402Z         	 �M
2020-05-09T01:13:33.3887002Z �[K[�[0;1;31m*�[0m�[0;31m*    �[0m] A start job is running for Docker A…n Container Engine (6s / no limit)
2020-05-09T01:13:33.3887250Z         	 �M
2020-05-09T01:13:33.3887738Z �[K[�[0;31m*�[0;1;31m*�[0m�[0;31m*   �[0m] A start job is running for Docker A…n Container Engine (6s / no limit)
2020-05-09T01:13:33.3887983Z         	 �M
2020-05-09T01:13:33.3888436Z �[K[ �[0;31m*�[0;1;31m*�[0m�[0;31m*  �[0m] A start job is running for Docker A…n Container Engine (7s / no limit)
2020-05-09T01:13:33.3888700Z         	 �M
2020-05-09T01:13:33.3889158Z �[K[  �[0;31m*�[0;1;31m*�[0m�[0;31m* �[0m] A start job is running for Docker A…n Container Engine (7s / no limit)
2020-05-09T01:13:33.3889437Z         	 �M
2020-05-09T01:13:33.3889901Z �[K[   �[0;31m*�[0;1;31m*�[0m�[0;31m*�[0m] A start job is running for Docker A…n Container Engine (8s / no limit)
2020-05-09T01:13:33.3890162Z         	 �M
2020-05-09T01:13:33.3890477Z �[K[    �[0;31m*�[0;1;31m*�[0m] A start job is running for Docker A…n Container Engine (8s / no limit)
2020-05-09T01:13:33.3890881Z         	 �M
2020-05-09T01:13:33.3891186Z �[K[     �[0;31m*�[0m] A start job is running for Docker A…n Container Engine (9s / no limit)
2020-05-09T01:13:33.3891403Z         	 �M
2020-05-09T01:13:33.3891878Z �[K[    �[0;31m*�[0;1;31m*�[0m] A start job is running for Docker A…n Container Engine (9s / no limit)
2020-05-09T01:13:33.3892119Z         	 �M
2020-05-09T01:13:33.3892444Z �[K[   �[0;31m*�[0;1;31m*�[0m�[0;31m*�[0m] A start job is running for Docker A… Container Engine (10s / no limit)
2020-05-09T01:13:33.3892661Z         	 �M
2020-05-09T01:13:33.3893135Z �[K[  �[0;31m*�[0;1;31m*�[0m�[0;31m* �[0m] A start job is running for Docker A… Container Engine (10s / no limit)
2020-05-09T01:13:33.3893413Z         	 �M
2020-05-09T01:13:33.3893742Z �[K[ �[0;31m*�[0;1;31m*�[0m�[0;31m*  �[0m] A start job is running for Docker A… Container Engine (11s / no limit)
2020-05-09T01:13:33.3893976Z         	 �M
2020-05-09T01:13:33.3894283Z �[K[�[0;31m*�[0;1;31m*�[0m�[0;31m*   �[0m] A start job is running for Docker A… Container Engine (11s / no limit)
2020-05-09T01:13:33.3894512Z         	 �M
2020-05-09T01:13:33.3894830Z �[K[�[0;1;31m*�[0m�[0;31m*    �[0m] A start job is running for Docker A… Container Engine (12s / no limit)
2020-05-09T01:13:33.3895063Z         	 �M
2020-05-09T01:13:33.3895528Z �[K[�[0m�[0;31m*     �[0m] A start job is running for Docker A… Container Engine (12s / no limit)
2020-05-09T01:13:33.3895768Z         	 �M
2020-05-09T01:13:33.3896085Z �[K[�[0;1;31m*�[0m�[0;31m*    �[0m] A start job is running for Docker A… Container Engine (13s / no limit)
2020-05-09T01:13:33.3896321Z         	 �M
2020-05-09T01:13:33.3896641Z �[K[�[0;31m*�[0;1;31m*�[0m�[0;31m*   �[0m] A start job is running for Docker A… Container Engine (13s / no limit)
2020-05-09T01:13:33.3896884Z         	 �M
2020-05-09T01:13:33.3897210Z �[K[ �[0;31m*�[0;1;31m*�[0m�[0;31m*  �[0m] A start job is running for Docker A… Container Engine (14s / no limit)
2020-05-09T01:13:33.3897438Z         	 �M
2020-05-09T01:13:33.3897762Z �[K[  �[0;31m*�[0;1;31m*�[0m�[0;31m* �[0m] A start job is running for Docker A… Container Engine (14s / no limit)
2020-05-09T01:13:33.3897976Z         	 �M
2020-05-09T01:13:33.3898303Z �[K[   �[0;31m*�[0;1;31m*�[0m�[0;31m*�[0m] A start job is running for Docker A… Container Engine (15s / no limit)
2020-05-09T01:13:33.3898531Z         	 �M
2020-05-09T01:13:33.3898844Z �[K[    �[0;31m*�[0;1;31m*�[0m] A start job is running for Docker A… Container Engine (15s / no limit)
2020-05-09T01:13:33.3899215Z         	 �M
2020-05-09T01:13:33.3899521Z �[K[     �[0;31m*�[0m] A start job is running for Docker A… Container Engine (16s / no limit)
2020-05-09T01:13:33.3899860Z         	 �M
2020-05-09T01:13:33.3900240Z �[K[    �[0;31m*�[0;1;31m*�[0m] A start job is running for Docker A… Container Engine (16s / no limit)
2020-05-09T01:13:33.3900487Z         	 �M
2020-05-09T01:13:33.3900795Z �[K[   �[0;31m*�[0;1;31m*�[0m�[0;31m*�[0m] A start job is running for Docker A… Container Engine (17s / no limit)
2020-05-09T01:13:33.3901026Z         	 �M
2020-05-09T01:13:33.3901350Z �[K[  �[0;31m*�[0;1;31m*�[0m�[0;31m* �[0m] A start job is running for Docker A… Container Engine (17s / no limit)
2020-05-09T01:13:33.3901583Z         	 �M
2020-05-09T01:13:33.3901892Z �[K[ �[0;31m*�[0;1;31m*�[0m�[0;31m*  �[0m] A start job is running for Docker A… Container Engine (18s / no limit)
2020-05-09T01:13:33.3902119Z         	 �M
2020-05-09T01:13:33.3902444Z �[K[�[0;31m*�[0;1;31m*�[0m�[0;31m*   �[0m] A start job is running for Docker A… Container Engine (18s / no limit)
2020-05-09T01:13:33.3902674Z         	 �M
2020-05-09T01:13:33.3902946Z �[K[�[0;32m  OK  �[0m] Started �[0;1;39mDocker Application Container Engine�[0m.
2020-05-09T01:13:33.3903267Z         	 �[K[�[0;32m  OK  �[0m] Reached target �[0;1;39mMulti-User System�[0m.
2020-05-09T01:13:33.3903567Z         	 [�[0;32m  OK  �[0m] Reached target �[0;1;39mGraphical Interface�[0m.
2020-05-09T01:13:33.3903872Z         	          Starting �[0;1;39mUpdate UTMP about System Runlevel Changes�[0m...
2020-05-09T01:13:33.3904187Z         	 [�[0;32m  OK  �[0m] Started �[0;1;39mUpdate UTMP about System Runlevel Changes�[0m.
2020-05-09T01:13:33.3904475Z         	 [�[0;32m  OK  �[0m] Stopped target �[0;1;39mGraphical Interface�[0m.
2020-05-09T01:13:33.3904772Z         	 [�[0;32m  OK  �[0m] Stopped target �[0;1;39mMulti-User System�[0m.
2020-05-09T01:13:33.3905065Z         	          Stopping �[0;1;39mOpenBSD Secure Shell server�[0m...
2020-05-09T01:13:33.3905539Z         	          Stopping �[0;1;39mDocker Application Container Engine�[0m...
2020-05-09T01:13:33.3905828Z         	 [�[0;32m  OK  �[0m] Stopped target �[0;1;39mTimers�[0m.
2020-05-09T01:13:33.3906331Z         	 [�[0;32m  OK  �[0m] Stopped �[0;1;39mDaily Cleanup of Temporary Directories�[0m.
2020-05-09T01:13:33.3906643Z         	          Stopping �[0;1;39mkubelet: The Kubernetes Node Agent�[0m...
2020-05-09T01:13:33.3906943Z         	 [�[0;32m  OK  �[0m] Stopped �[0;1;39mOpenBSD Secure Shell server�[0m.
2020-05-09T01:13:33.3907254Z         	 [�[0;32m  OK  �[0m] Stopped �[0;1;39mkubelet: The Kubernetes Node Agent�[0m.
2020-05-09T01:13:33.3907567Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/run/docker/netns/aa1b0923dd44�[0m.
2020-05-09T01:13:33.3907889Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…58c78dd97dd8837044/mounts/shm�[0m.
2020-05-09T01:13:33.3908232Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…53f1b10cb248f136a79d62/merged�[0m.
2020-05-09T01:13:33.3908608Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…064d6fdfc70555c1dfa9ae/merged�[0m.
2020-05-09T01:13:33.3908944Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…80bc9e6e23b9b11069/mounts/shm�[0m.
2020-05-09T01:13:33.3909293Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…0bd669f2107501fa439ddc/merged�[0m.
2020-05-09T01:13:33.3909616Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…8988b71a56764918b219eb/merged�[0m.
2020-05-09T01:13:33.3909951Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…dfd7aed2513d7898bb/mounts/shm�[0m.
2020-05-09T01:13:33.3910282Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…86a66f77e5b8db4730e54c/merged�[0m.
2020-05-09T01:13:33.3910613Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…90c26ab4c643a311e3eadf/merged�[0m.
2020-05-09T01:13:33.3910945Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…2d09bb78a9175e5e68/mounts/shm�[0m.
2020-05-09T01:13:33.3911261Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…8d5c06fb3b450ba343dadd/merged�[0m.
2020-05-09T01:13:33.3911591Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…8c3b11266a26c52970/mounts/shm�[0m.
2020-05-09T01:13:33.3912086Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…1f931a99f3c918457299c5/merged�[0m.
2020-05-09T01:13:33.3912428Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…be24cdc7c269e566505aa3/merged�[0m.
2020-05-09T01:13:33.3912759Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…db9bd7b9291c673db8abc9/merged�[0m.
2020-05-09T01:13:33.3913051Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/run/docker/netns/1e283b579cba�[0m.
2020-05-09T01:13:33.3913383Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…dcc8814badd936c3f2/mounts/shm�[0m.
2020-05-09T01:13:33.3913713Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…d033dfa3accf2920bdb690/merged�[0m.
2020-05-09T01:13:33.3914037Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…cff91c6913e058ebdf/mounts/shm�[0m.
2020-05-09T01:13:33.3914369Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…9ff343a4946005ecf3ef30/merged�[0m.
2020-05-09T01:13:33.3914701Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…0b22b2a1b260bc1f6ee1f3/merged�[0m.
2020-05-09T01:13:33.3915031Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…716b8c97f13f4af9e4/mounts/shm�[0m.
2020-05-09T01:13:33.3915362Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…6bddbf27597a52e8175c86/merged�[0m.
2020-05-09T01:13:33.3915669Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/run/docker/netns/6835b76c9e5d�[0m.
2020-05-09T01:13:33.3916001Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…471f07e468959f2c0d/mounts/shm�[0m.
2020-05-09T01:13:33.3916315Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…2dabb750c38ff4b453fbb3/merged�[0m.
2020-05-09T01:13:33.3916649Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…e9d3b534554414b98ed159/merged�[0m.
2020-05-09T01:13:33.3916979Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…d58d02d84d190b40739c4b/merged�[0m.
2020-05-09T01:13:33.3917323Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/docker/…decd3f3639f923da965043/merged�[0m.
2020-05-09T01:13:33.3917635Z         	 [�[0;32m  OK  �[0m] Stopped �[0;1;39mDocker Application Container Engine�[0m.
2020-05-09T01:13:33.3917921Z         	 [�[0;32m  OK  �[0m] Stopped target �[0;1;39mNetwork is Online�[0m.
2020-05-09T01:13:33.3918214Z         	          Stopping �[0;1;39mcontainerd container runtime�[0m...
2020-05-09T01:13:33.3918513Z         	 [�[0;32m  OK  �[0m] Stopped �[0;1;39mcontainerd container runtime�[0m.
2020-05-09T01:13:33.3918801Z         	 [�[0;32m  OK  �[0m] Stopped target �[0;1;39mBasic System�[0m.
2020-05-09T01:13:33.3919074Z         	 [�[0;32m  OK  �[0m] Stopped target �[0;1;39mSockets�[0m.
2020-05-09T01:13:33.3919551Z         	 [�[0;32m  OK  �[0m] Closed �[0;1;39mDocker Socket for the API�[0m.
2020-05-09T01:13:33.3919845Z         	 [�[0;32m  OK  �[0m] Stopped target �[0;1;39mPaths�[0m.
2020-05-09T01:13:33.3920135Z         	 [�[0;32m  OK  �[0m] Stopped target �[0;1;39mSlices�[0m.
2020-05-09T01:13:33.3920441Z         	 [�[0;32m  OK  �[0m] Stopped target �[0;1;39mSystem Initialization�[0m.
2020-05-09T01:13:33.3920723Z         	 [�[0;32m  OK  �[0m] Stopped target �[0;1;39mLocal File Systems�[0m.
2020-05-09T01:13:33.3921004Z         	          Unmounting �[0;1;39m/etc/hosts�[0m...
2020-05-09T01:13:33.3921317Z         	          Unmounting �[0;1;39m/var/lib/kubele…~secret/coredns-token-82jdr�[0m...
2020-05-09T01:13:33.3921633Z         	          Unmounting �[0;1;39m/var/lib/kubele…~secret/default-token-9gkgx�[0m...
2020-05-09T01:13:33.3921902Z         	          Unmounting �[0;1;39m/etc/resolv.conf�[0m...
2020-05-09T01:13:33.3922214Z         	          Unmounting �[0;1;39m/var/lib/kubele…cret/kube-proxy-token-6vd7t�[0m...
2020-05-09T01:13:33.3922531Z         	          Unmounting �[0;1;39m/var/lib/kubele…age-provisioner-token-9xps6�[0m...
2020-05-09T01:13:33.3922846Z         	          Unmounting �[0;1;39m/var/lib/kubele…~secret/coredns-token-82jdr�[0m...
2020-05-09T01:13:33.3923283Z         	          Unmounting �[0;1;39m/run/docker/netns/default�[0m...
2020-05-09T01:13:33.3923611Z         	          Unmounting �[0;1;39m/etc/hostname�[0m...
2020-05-09T01:13:33.3923883Z         	          Unmounting �[0;1;39m/tmp�[0m...
2020-05-09T01:13:33.3924161Z         	          Unmounting �[0;1;39m/usr/lib/modules�[0m...
2020-05-09T01:13:33.3924437Z         	          Unmounting �[0;1;39m/kind/product_uuid�[0m...
2020-05-09T01:13:33.3924735Z         	          Stopping �[0;1;39mUpdate UTMP about System Boot/Shutdown�[0m...
2020-05-09T01:13:33.3925017Z         	 [�[0;32m  OK  �[0m] Stopped �[0;1;39mApply Kernel Variables�[0m.
2020-05-09T01:13:33.3925324Z         	 [�[0;32m  OK  �[0m] Stopped target �[0;1;39mLocal Encrypted Volumes�[0m.
2020-05-09T01:13:33.3925652Z         	 [�[0;32m  OK  �[0m] Stopped �[0;1;39mDispatch Password …ts to Console Directory Watch�[0m.
2020-05-09T01:13:33.3925939Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/etc/hosts�[0m.
2020-05-09T01:13:33.3926257Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/kubelet…io~secret/coredns-token-82jdr�[0m.
2020-05-09T01:13:33.3926588Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/kubelet…io~secret/default-token-9gkgx�[0m.
2020-05-09T01:13:33.3926883Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/etc/resolv.conf�[0m.
2020-05-09T01:13:33.3927201Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/kubelet…secret/kube-proxy-token-6vd7t�[0m.
2020-05-09T01:13:33.3927528Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/kubelet…orage-provisioner-token-9xps6�[0m.
2020-05-09T01:13:33.3927838Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var/lib/kubelet…io~secret/coredns-token-82jdr�[0m.
2020-05-09T01:13:33.3928138Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/run/docker/netns/default�[0m.
2020-05-09T01:13:33.3928426Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/etc/hostname�[0m.
2020-05-09T01:13:33.3928702Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/tmp�[0m.
2020-05-09T01:13:33.3929000Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/usr/lib/modules�[0m.
2020-05-09T01:13:33.3929275Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/kind/product_uuid�[0m.
2020-05-09T01:13:33.3929559Z         	 [�[0;32m  OK  �[0m] Stopped target �[0;1;39mSwap�[0m.
2020-05-09T01:13:33.3929869Z         	 [�[0;32m  OK  �[0m] Stopped �[0;1;39mUpdate UTMP about System Boot/Shutdown�[0m.
2020-05-09T01:13:33.3930140Z         	          Unmounting �[0;1;39m/var�[0m...
2020-05-09T01:13:33.3930396Z         	 [�[0;32m  OK  �[0m] Unmounted �[0;1;39m/var�[0m.
2020-05-09T01:13:33.3930691Z         	 [�[0;32m  OK  �[0m] Reached target �[0;1;39mUnmount All Filesystems�[0m.
2020-05-09T01:13:33.3930995Z         	 [�[0;32m  OK  �[0m] Stopped target �[0;1;39mLocal File Systems (Pre)�[0m.
2020-05-09T01:13:33.3931304Z         	 [�[0;32m  OK  �[0m] Stopped �[0;1;39mCreate Static Device Nodes in /dev�[0m.
2020-05-09T01:13:33.3931596Z         	 [�[0;32m  OK  �[0m] Stopped �[0;1;39mCreate System Users�[0m.
2020-05-09T01:13:33.3931897Z         	 [�[0;32m  OK  �[0m] Stopped �[0;1;39mRemount Root and Kernel File Systems�[0m.
2020-05-09T01:13:33.3932185Z         	 [�[0;32m  OK  �[0m] Reached target �[0;1;39mShutdown�[0m.
2020-05-09T01:13:33.3932468Z         	 [�[0;32m  OK  �[0m] Reached target �[0;1;39mFinal Step�[0m.
2020-05-09T01:13:33.3932742Z         	 [�[0;32m  OK  �[0m] Started �[0;1;39mPower-Off�[0m.
2020-05-09T01:13:33.3933022Z         	 [�[0;32m  OK  �[0m] Reached target �[0;1;39mPower-Off�[0m.
2020-05-09T01:13:33.3933270Z         	          Starting �[0;1;39mHalt�[0m...
2020-05-09T01:13:33.3933554Z         	 INFO: ensuring we can execute /bin/mount even with userns-remap
2020-05-09T01:13:33.3933810Z         	 INFO: remounting /sys read-only
2020-05-09T01:13:33.3933920Z         	 INFO: making mounts shared
2020-05-09T01:13:33.3934015Z         	 INFO: fix cgroup mounts for all subsystems
2020-05-09T01:13:33.3934282Z         	 INFO: clearing and regenerating /etc/machine-id
2020-05-09T01:13:33.3934498Z         	 Initializing machine ID from random generator.
2020-05-09T01:13:33.3934619Z         	 INFO: faking /sys/class/dmi/id/product_name to be "kind"
2020-05-09T01:13:33.3934722Z         	 INFO: faking /sys/class/dmi/id/product_uuid to be random
2020-05-09T01:13:33.3934843Z         	 INFO: faking /sys/devices/virtual/dmi/id/product_uuid as well
2020-05-09T01:13:33.3934964Z         	 INFO: setting iptables to detected mode: legacy
2020-05-09T01:13:33.3935433Z         	 systemd 242 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
2020-05-09T01:13:33.3935575Z         	 Detected virtualization docker.
2020-05-09T01:13:33.3935808Z         	 Detected architecture x86-64.
2020-05-09T01:13:33.3935915Z         	 
2020-05-09T01:13:33.3936161Z         	 Welcome to �[1mUbuntu 19.10�[0m!
2020-05-09T01:13:33.3936271Z         	 
2020-05-09T01:13:33.3936531Z         	 Set hostname to <embed-certs-20200509011020-5179>.
2020-05-09T01:13:33.3936812Z         	 Failed to bump fs.file-max, ignoring: Invalid argument
2020-05-09T01:13:33.3936958Z         	 Failed to attach 1 to compat systemd cgroup /actions_job/7f8719abd6da75f05773821619c61449d06518cd14536021b8f5535573d030c5/init.scope: No such file or directory
2020-05-09T01:13:33.3937272Z         	 �[0;1;31mFailed to open pin file: No such file or directory�[0m
2020-05-09T01:13:33.3937576Z         	 �[0;1;31mFailed to allocate manager object: No such file or directory�[0m
2020-05-09T01:13:33.3937846Z         	 [�[0;1;31m!!!!!!�[0m] Failed to allocate manager object.
2020-05-09T01:13:33.3938097Z         	 �[0;1;31mExiting PID 1...�[0m
2020-05-09T01:13:33.3938376Z         	 INFO: ensuring we can execute /bin/mount even with userns-remap
2020-05-09T01:13:33.3938632Z         	 INFO: remounting /sys read-only
2020-05-09T01:13:33.3938723Z         	 INFO: making mounts shared
2020-05-09T01:13:33.3938838Z         	 INFO: fix cgroup mounts for all subsystems
2020-05-09T01:13:33.3939111Z         	 INFO: clearing and regenerating /etc/machine-id
2020-05-09T01:13:33.3939229Z         	 Initializing machine ID from random generator.
2020-05-09T01:13:33.3939349Z         	 INFO: faking /sys/class/dmi/id/product_name to be "kind"
2020-05-09T01:13:33.3939451Z         	 INFO: faking /sys/class/dmi/id/product_uuid to be random
2020-05-09T01:13:33.3939568Z         	 INFO: faking /sys/devices/virtual/dmi/id/product_uuid as well
2020-05-09T01:13:33.3939685Z         	 INFO: setting iptables to detected mode: legacy
2020-05-09T01:13:33.3940137Z         	 systemd 242 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
2020-05-09T01:13:33.3940270Z         	 Detected virtualization docker.
2020-05-09T01:13:33.3940500Z         	 Detected architecture x86-64.
2020-05-09T01:13:33.3940611Z         	 
2020-05-09T01:13:33.3940859Z         	 Welcome to �[1mUbuntu 19.10�[0m!
2020-05-09T01:13:33.3940965Z         	 
2020-05-09T01:13:33.3941219Z         	 Set hostname to <embed-certs-20200509011020-5179>.
2020-05-09T01:13:33.3941495Z         	 Failed to bump fs.file-max, ignoring: Invalid argument
2020-05-09T01:13:33.3941636Z         	 Failed to attach 1 to compat systemd cgroup /actions_job/7f8719abd6da75f05773821619c61449d06518cd14536021b8f5535573d030c5/init.scope: No such file or directory
2020-05-09T01:13:33.3941947Z         	 �[0;1;31mFailed to open pin file: No such file or directory�[0m
2020-05-09T01:13:33.3942233Z         	 �[0;1;31mFailed to allocate manager object: No such file or directory�[0m
2020-05-09T01:13:33.3942523Z         	 [�[0;1;31m!!!!!!�[0m] Failed to allocate manager object.
2020-05-09T01:13:33.3942776Z         	 �[0;1;31mExiting PID 1...�[0m
2020-05-09T01:13:33.3942881Z         
2020-05-09T01:13:33.3943102Z         -- /stdout --
2020-05-09T01:13:33.3943590Z     TestStartStop/group/embed-certs: helpers.go:209: (dbg) Run:  ./minikube-linux-amd64 status --format={{.Host}} -p embed-certs-20200509011020-5179
2020-05-09T01:13:33.4618946Z     TestStartStop/group/embed-certs: helpers.go:209: (dbg) Non-zero exit: ./minikube-linux-amd64 status --format={{.Host}} -p embed-certs-20200509011020-5179: exit status 7 (76.738765ms)
2020-05-09T01:13:33.4619634Z         
2020-05-09T01:13:33.4620210Z         -- stdout --
2020-05-09T01:13:33.4620622Z         	Stopped
2020-05-09T01:13:33.4620893Z         
2020-05-09T01:13:33.4621418Z         -- /stdout --
2020-05-09T01:13:33.4622069Z     TestStartStop/group/embed-certs: helpers.go:209: status error: exit status 7 (may be ok)
2020-05-09T01:13:33.4622763Z     TestStartStop/group/embed-certs: helpers.go:211: "embed-certs-20200509011020-5179" host is not running, skipping log retrieval (state="Stopped")
2020-05-09T01:13:33.4623489Z     TestStartStop/group/embed-certs: helpers.go:170: (dbg) Run:  ./minikube-linux-amd64 delete -p embed-certs-20200509011020-5179
2020-05-09T01:13:35.2232890Z     TestStartStop/group/embed-certs: helpers.go:170: (dbg) Done: ./minikube-linux-amd64 delete -p embed-certs-20200509011020-5179: (1.760998652s)
2020-05-09T01:13:35.2242879Z --- FAIL: TestStartStop (194.58s)
2020-05-09T01:13:35.2245687Z     --- FAIL: TestStartStop/group (0.00s)
2020-05-09T01:13:35.2246356Z         --- FAIL: TestStartStop/group/embed-certs (194.58s)
2020-05-09T01:13:35.2246534Z FAIL
2020-05-09T01:13:35.2292112Z ##[group]Run cd minikube_binaries
2020-05-09T01:13:35.2292315Z �[36;1mcd minikube_binaries�[0m
2020-05-09T01:13:35.2292448Z �[36;1mexport PATH=${PATH}:`go env GOPATH`/bin�[0m
2020-05-09T01:13:35.2292570Z �[36;1mgo tool test2json -t < ./report/testout.txt > ./report/testout.json || true�[0m
2020-05-09T01:13:35.2292709Z �[36;1mSTAT=$(gopogh -in ./report/testout.json -out ./report/testout.html -name "${JOB_NAME} ${GITHUB_REF}" -repo "${GITHUB_REPOSITORY}"  -details "${GITHUB_SHA}")  || true�[0m
2020-05-09T01:13:35.2292842Z �[36;1mecho status: ${STAT}�[0m
2020-05-09T01:13:35.2293251Z �[36;1mFailNum=$(echo $STAT | jq '.NumberOfFail')�[0m
2020-05-09T01:13:35.2293369Z �[36;1mTestsNum=$(echo $STAT | jq '.NumberOfTests')�[0m
2020-05-09T01:13:35.2293499Z �[36;1mGOPOGH_RESULT="${JOB_NAME} : completed with ${FailNum} / ${TestsNum} failures in ${TIME_ELAPSED}"�[0m
2020-05-09T01:13:35.2293617Z �[36;1mecho ::set-env name=GOPOGH_RESULT::${GOPOGH_RESULT}�[0m
2020-05-09T01:13:35.2293818Z �[36;1mecho ::set-env name=STAT::${STAT}�[0m
2020-05-09T01:13:35.2330047Z shell: /bin/bash --noprofile --norc -e -o pipefail {0}
2020-05-09T01:13:35.2330175Z env:
2020-05-09T01:13:35.2330324Z   GOPROXY: https://proxy.golang.org
2020-05-09T01:13:35.2330457Z   TIME_ELAPSED: 3 min 15 seconds
2020-05-09T01:13:35.2330568Z   JOB_NAME: start_stop_embed_cert_docker_ubuntu
2020-05-09T01:13:35.2330676Z   GOPOGH_RESULT: 
2020-05-09T01:13:35.2330767Z   SHELL: /bin/bash
2020-05-09T01:13:35.2330873Z ##[endgroup]
2020-05-09T01:13:37.8096132Z status: { "NumberOfTests": 1, "NumberOfFail": 1, "NumberOfPass": 0, "NumberOfSkip": 0, "FailedTests": [ "TestStartStop/group/embed-certs" ], "GopoghVersion": "v0.1.19", "GopoghBuild": "88c32f2f9f883a1da5f625dda6849d83d51f130d-dirty", "Detail": { "Name": "start_stop_embed_cert_docker_ubuntu refs/pull/8025/merge", "Details": "dc6a5a658e8146fe1718ef37839c9dfbc3e7c205", "PR": "", "RepoName": "kubernetes/minikube" } }
2020-05-09T01:13:37.9135448Z ##[group]Run actions/upload-artifact@v1
2020-05-09T01:13:37.9135576Z with:
2020-05-09T01:13:37.9135684Z   name: start_stop_embed_cert_docker_ubuntu
2020-05-09T01:13:37.9135795Z   path: minikube_binaries/report
2020-05-09T01:13:37.9135926Z env:
2020-05-09T01:13:37.9136088Z   GOPROXY: https://proxy.golang.org
2020-05-09T01:13:37.9136204Z   TIME_ELAPSED: 3 min 15 seconds
2020-05-09T01:13:37.9136315Z   JOB_NAME: start_stop_embed_cert_docker_ubuntu
2020-05-09T01:13:37.9136446Z   GOPOGH_RESULT: start_stop_embed_cert_docker_ubuntu : completed with 1 / 1 failures in 3 min 15 seconds
2020-05-09T01:13:37.9136573Z   SHELL: /bin/bash
2020-05-09T01:13:37.9136907Z   STAT: { "NumberOfTests": 1, "NumberOfFail": 1, "NumberOfPass": 0, "NumberOfSkip": 0, "FailedTests": [ "TestStartStop/group/embed-certs" ], "GopoghVersion": "v0.1.19", "GopoghBuild": "88c32f2f9f883a1da5f625dda6849d83d51f130d-dirty", "Detail": { "Name": "start_stop_embed_cert_docker_ubuntu refs/pull/8025/merge", "Details": "dc6a5a658e8146fe1718ef37839c9dfbc3e7c205", "PR": "", "RepoName": "kubernetes/minikube" } }
2020-05-09T01:13:37.9137094Z ##[endgroup]
2020-05-09T01:13:38.1333750Z Uploading artifact 'start_stop_embed_cert_docker_ubuntu' from '/home/runner/work/minikube/minikube/minikube_binaries/report' for run #2398
2020-05-09T01:13:38.2184438Z Uploading 3 files
2020-05-09T01:13:38.7602702Z File upload complete.
2020-05-09T01:13:38.7615376Z Uploaded '455028' bytes from '/home/runner/work/minikube/minikube/minikube_binaries/report' to server
2020-05-09T01:13:38.9143825Z Associated artifact start_stop_embed_cert_docker_ubuntu (6093853) with run #2398
2020-05-09T01:13:38.9242110Z ##[group]Run echo ${GOPOGH_RESULT}
2020-05-09T01:13:38.9242308Z �[36;1mecho ${GOPOGH_RESULT}�[0m
2020-05-09T01:13:38.9242422Z �[36;1mnumFail=$(echo $STAT | jq '.NumberOfFail')�[0m
2020-05-09T01:13:38.9242538Z �[36;1mecho "----------------${numFail} Failures----------------------------"�[0m
2020-05-09T01:13:38.9242660Z �[36;1mecho $STAT | jq '.FailedTests' || true�[0m
2020-05-09T01:13:38.9242770Z �[36;1mecho "-------------------------------------------------------"�[0m
2020-05-09T01:13:38.9242873Z �[36;1mnumPass=$(echo $STAT | jq '.NumberOfPass')�[0m
2020-05-09T01:13:38.9242982Z �[36;1mecho "*** $numPass Passed ***"�[0m
2020-05-09T01:13:38.9243092Z �[36;1mif [ "$numFail" -gt 0 ];then echo "*** $numFail Failed ***";exit 2;fi�[0m
2020-05-09T01:13:38.9280888Z shell: /bin/bash --noprofile --norc -e -o pipefail {0}
2020-05-09T01:13:38.9281013Z env:
2020-05-09T01:13:38.9281195Z   GOPROXY: https://proxy.golang.org
2020-05-09T01:13:38.9281321Z   TIME_ELAPSED: 3 min 15 seconds
2020-05-09T01:13:38.9281506Z   JOB_NAME: start_stop_embed_cert_docker_ubuntu
2020-05-09T01:13:38.9281635Z   GOPOGH_RESULT: start_stop_embed_cert_docker_ubuntu : completed with 1 / 1 failures in 3 min 15 seconds
2020-05-09T01:13:38.9281742Z   SHELL: /bin/bash
2020-05-09T01:13:38.9281919Z   STAT: { "NumberOfTests": 1, "NumberOfFail": 1, "NumberOfPass": 0, "NumberOfSkip": 0, "FailedTests": [ "TestStartStop/group/embed-certs" ], "GopoghVersion": "v0.1.19", "GopoghBuild": "88c32f2f9f883a1da5f625dda6849d83d51f130d-dirty", "Detail": { "Name": "start_stop_embed_cert_docker_ubuntu refs/pull/8025/merge", "Details": "dc6a5a658e8146fe1718ef37839c9dfbc3e7c205", "PR": "", "RepoName": "kubernetes/minikube" } }
2020-05-09T01:13:38.9282095Z ##[endgroup]
2020-05-09T01:13:38.9350713Z start_stop_embed_cert_docker_ubuntu : completed with 1 / 1 failures in 3 min 15 seconds
2020-05-09T01:13:38.9385668Z ----------------1 Failures----------------------------
2020-05-09T01:13:38.9416239Z [
2020-05-09T01:13:38.9416863Z   "TestStartStop/group/embed-certs"
2020-05-09T01:13:38.9417001Z ]
2020-05-09T01:13:38.9419707Z -------------------------------------------------------
2020-05-09T01:13:38.9455512Z *** 0 Passed ***
2020-05-09T01:13:38.9455999Z *** 1 Failed ***
2020-05-09T01:13:38.9460603Z ##[error]Process completed with exit code 2.
2020-05-09T01:13:38.9471103Z Cleaning up orphan processes
@medyagh medyagh added priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. co/docker-driver Issues related to kubernetes in container labels May 9, 2020
@afbjorklund
Copy link
Collaborator

This looks like the exact same issue that we were seeing with podman in #8033

The real error isn't actually the re-attaching to the old cgroups (since those are gone).
We see this all the time, even when it starts up correctly again - for both drivers.

The problem is about failing to create some fiiles, possibly due to read-only /sys.

@afbjorklund
Copy link
Collaborator

These lines are supposed to be enough:

INFO: remounting /sys read-only

INFO: fix cgroup mounts for all subsystems

And there doesn't seem to be any errors ?

That should make /sys/fs/cgroup/systemd writable, for the docker/<container> subdirectory.

But here it is still failing, the second time ?

2020-05-09T01:13:33.3937272Z         	 �[0;1;31mFailed to open pin file: No such file or directory�[0m
2020-05-09T01:13:33.3937576Z         	 �[0;1;31mFailed to allocate manager object: No such file or directory�[0m
2020-05-09T01:13:33.3937846Z         	 [�[0;1;31m!!!!!!�[0m] Failed to allocate manager object.
2020-05-09T01:13:33.3938097Z         	 �[0;1;31mExiting PID 1...�[0m

For podman the issue was just that KIND only did the workaround for "docker", not "libpod_parent"

@afbjorklund afbjorklund added kind/bug Categorizes issue or PR as related to a bug. co/runtime/docker Issues specific to a docker runtime labels May 9, 2020
@afbjorklund
Copy link
Collaborator

For debugging this, you can get the ID of the minikube container and then look at the mount:

findmnt /sys/fs/cgroup/systemd/docker/${CONTAINER_ID}

It supposed to be mounted as cgroup rw, even though systemd requires /sys to be ro.

Seems like fix_cgroups (in entrypoint) silently failed on us ?

@afbjorklund
Copy link
Collaborator

Doesn't seem to reproduce locally, can stop/start docker containers multiple times

So it must be something specific to Azure Linux, or maybe just a fluke perhaps...

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 7, 2020
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Sep 6, 2020
@medyagh medyagh changed the title start stop in github actions: SSH service wont come up after second stop GH action does not support Start Stop Sep 23, 2020
@sharifelgamal sharifelgamal added area/testing lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. and removed lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. labels Sep 30, 2020
@medyagh medyagh closed this as completed Feb 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/testing co/docker-driver Issues related to kubernetes in container co/runtime/docker Issues specific to a docker runtime kind/bug Categorizes issue or PR as related to a bug. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@medyagh @afbjorklund @k8s-ci-robot @sharifelgamal @fejta-bot