CapRover: Error response from daemon

[mik-tf]

CapRover: Error response from daemon

Situation

• CapRover weblet on test and dev nets is not working
• When running zinit log caprover I get:

[-] caprover: Unable to find image 'caprover/caprover:latest' locally
[-] caprover: docker: Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:39253->[::1]:53: read: connection refused.
[-] caprover: See 'docker run --help'.

• When running zinit log dockerd I get:

[-] dockerd: time="2023-12-12T18:17:41.056089002Z" level=info msg="Attempting next endpoint for pull after error: Get \"https://registry-1.docker.io/v2/\": dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:45512->[::1]:53: read: connection refused"

There is no tf-images of caprover, so it's hard to troubleshoot.

To reproduce

• Deploy caprover weblet, set a domain, a leader and a worker node
• Set the wildcard DNS for captain.domain.com
• SSH into the VM, run the zinit log commands

Assessment

It seems there is a problem with docker itself on this flist. docker run hello-world return the same error:

Unable to find image 'hello-world:latest' locally
docker: Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:51844->[::1]:53: read: connection refused.

[sabrinasadik]

Looks like DNS resolution is not working properly. What image is it, how is the VM deployed?
@zaelgohary doesn't look like an ops issue to me, someone from development needs to get more information & look into it

[mik-tf]

This is directly from the playground, so the flist is the following: https://hub.grid.tf/tf-official-apps/tf-caprover-latest.flist, as can be seen on the caprover weblet page here.

I tested the caprover weblet of the playground on both dev and test nets, with the same result.
Let me know if you need any other info.

As I wrote, we don't have an image in tf-images of the caprover weblet, so it's hard to troubleshoot.
I already made an issue in tf-images to get a caprover image publicly available.

[AhmedHanafy725]

I tried on qanet and it could download the image successfully but it failed while installing.

not sure if that is as we use the latest tag for the image so should we downgrade or not?

for the caprover image, it's here https://github.com/threefoldtech/freeflow_caprover

[A-Harby]

I tried on testnet and got the same error message but installation was successful.

[-] caprover: Unable to find image 'caprover/caprover:latest' locally
[-] caprover: latest: Pulling from caprover/caprover
[-] caprover: 7264a8db6415: Pulling fs layer
[-] caprover: 751194035c36: Pulling fs layer
[-] caprover: eff5dce73b38: Pulling fs layer
[-] caprover: c8ce5be43019: Pulling fs layer
[-] caprover: c6458455a173: Pulling fs layer
[-] caprover: 40c4b680317f: Pulling fs layer
[-] caprover: 4f4fb700ef54: Pulling fs layer
[-] caprover: aeccc6266749: Pulling fs layer
[-] caprover: 2a38da3852b8: Pulling fs layer
[-] caprover: c8ce5be43019: Waiting
[-] caprover: c6458455a173: Waiting
[-] caprover: 40c4b680317f: Waiting
[-] caprover: 4f4fb700ef54: Waiting
[-] caprover: aeccc6266749: Waiting
[-] caprover: 2a38da3852b8: Waiting
[-] caprover: eff5dce73b38: Verifying Checksum
[-] caprover: eff5dce73b38: Download complete
[-] caprover: 7264a8db6415: Verifying Checksum
[-] caprover: 7264a8db6415: Download complete
[-] caprover: c8ce5be43019: Verifying Checksum
[-] caprover: c8ce5be43019: Download complete
[-] caprover: 40c4b680317f: Verifying Checksum
[-] caprover: 40c4b680317f: Download complete
[-] caprover: 4f4fb700ef54: Verifying Checksum
[-] caprover: 4f4fb700ef54: Download complete
[-] caprover: 7264a8db6415: Pull complete
[-] caprover: aeccc6266749: Verifying Checksum
[-] caprover: aeccc6266749: Download complete
[-] caprover: 751194035c36: Verifying Checksum
[-] caprover: 751194035c36: Download complete
[-] caprover: c6458455a173: Verifying Checksum
[-] caprover: c6458455a173: Download complete
[-] caprover: 2a38da3852b8: Verifying Checksum
[-] caprover: 2a38da3852b8: Download complete
[-] caprover: 751194035c36: Pull complete
[-] caprover: eff5dce73b38: Pull complete
[-] caprover: c8ce5be43019: Pull complete
[-] caprover: c6458455a173: Pull complete
[-] caprover: 40c4b680317f: Pull complete
[-] caprover: 4f4fb700ef54: Pull complete
[-] caprover: aeccc6266749: Pull complete
[-] caprover: 2a38da3852b8: Pull complete
[-] caprover: Digest: sha256:905815f252fda652c99739214361ab1501b525f3063fa68b28bb19767a8b0247
[-] caprover: Status: Downloaded newer image for caprover/caprover:latest
[-] containerd: time="2023-12-18T15:33:48.701426869Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
[-] containerd: time="2023-12-18T15:33:48.701864019Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
[-] containerd: time="2023-12-18T15:33:48.701899685Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
[-] containerd: time="2023-12-18T15:33:48.702869032Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/8fc1f518c550f722cf620a7d907b5e7d005c7ed581e0eb15c8eb7294e1e8bd84 pid=448 runtime=io.containerd.runc.v2
[+] caprover: Captain Starting ...
[+] caprover: Overriding publishedNameOnDockerHub from /captain/data/config-override.json
[+] caprover: Overriding version from /captain/data/config-override.json
[+] caprover: Overriding skipVerifyingDomains from /captain/data/config-override.json
[+] caprover: Installing Captain Service ...
[+] caprover: Installation failed.
[-] caprover: Error: Add the following to the installer line:
[-] caprover:                 -e ACCEPTED_TERMS=true
[-] caprover:                 
[-] caprover:                 Terms of service must be accepted before installation, view them here: 
[-] caprover:                 https://github.com/caprover/caprover/blob/master/TERMS_AND_CONDITIONS.md
[-] caprover:     at /usr/src/app/built/utils/CaptainInstaller.js:167:19
[-] caprover:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[-] dockerd: time="2023-12-18T15:33:51.420234737Z" level=info msg="ignoring event" container=8fc1f518c550f722cf620a7d907b5e7d005c7ed581e0eb15c8eb7294e1e8bd84 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[-] containerd: time="2023-12-18T15:33:51.421840320Z" level=info msg="shim disconnected" id=8fc1f518c550f722cf620a7d907b5e7d005c7ed581e0eb15c8eb7294e1e8bd84
[-] containerd: time="2023-12-18T15:33:51.421935608Z" level=warning msg="cleaning up after shim disconnected" id=8fc1f518c550f722cf620a7d907b5e7d005c7ed581e0eb15c8eb7294e1e8bd84 namespace=moby
[-] containerd: time="2023-12-18T15:33:51.421954346Z" level=info msg="cleaning up dead shim"
[-] containerd: time="2023-12-18T15:33:51.487090732Z" level=warning msg="cleanup warnings time=\"2023-12-18T15:33:51Z\" level=info msg=\"starting signal loop\" namespace=moby pid=506 runtime=io.containerd.runc.v2\n"
[-] dockerd: time="2023-12-18T15:33:51.731062478Z" level=warning msg="Failed to delete conntrack state for 172.17.0.2: invalid argument"
[+] caprover: ===================================
[+] caprover:  **** Installation is done! *****  
[+] caprover: CapRover is available at http://captain.test.img.gridtesting.xyz
[+] caprover: Default password is: captain42
[+] caprover: ===================================

[ramezsaeed]

Test was passed on testnet 3.12 Run:
https://app.testlodge.com/a/26076/projects/40893/runs/725177?tab=2#executed_case_31436913

Test was passed on QAnet 3.12 Run:
https://app.testlodge.com/a/26076/projects/40893/runs/725034?tab=2#executed_case_31420329

[AhmedHanafy725]

Work Completed:

the image version of Caprover is downgraded to the version that used to work. and also created an issue for updating the current image to the latest threefoldtech/tf-images#189

[A-Harby]

Verified,

• Devnet
• e2580a5.
• Caprover test suite https://app.testlodge.com/a/26076/projects/40893/suites/194660

Caprover was deployed successfully, and installation was successful without any errors. I was able to open the Caprover dashboard.

[mik-tf]

Thanks everyone. Closing this issue as the work has been verified and done.

[AhmedHanafy725]

I have changed the image tag to 1.10.1 in the flist to be solved on all networks.

@maxux can you promote this flist? https://hub.grid.tf/samehabouelsaad.3bot/tf-caprover-main.flist -> https://hub.grid.tf/tf-official-apps/tf-caprover-latest.flist

please re-verify the issue on all networks once the image is promoted

[maxux]

Promoted :)

[A-Harby]

Verified,

• Devnet
• e2580a5.
• Caprover test suite https://app.testlodge.com/a/26076/projects/40893/suites/194660

Caprover was deployed successfully, and installation was successful without any errors. I was able to open the Caprover dashboard.

Verified,
I was able to deploy and use Caprover on Qa, Test, and Mainnet. I was also able to enable HTTP, activate the Docker registry, start monitoring, and deploy apps.
The only issue found is reported here