Cloud2Edge: Ditto helm chart moved to Ditto's repository

[sgloutnikov]

I just noticed that the Ditto helm chart used in cloud2edge requirements.yaml is no longer maintained in this repository but was moved to Ditto's repository and is also served from a different Helm repository. Does this mean that the cloud2edge package should be updated to use the new and updated Ditto Helm chart, or are there some incompatibilities that could be preventing the switch over?

I'll test on my end and see how far I can get with updating the cloud2edge Ditto version to latest.

[casafurix]

Hello, I am actually unable to install the cloud2edge package using the steps mentioned on the website (facing many errors), can you please guide me the right steps now to install it? I want to use Ditto+Hono. Thanks

[calohmn]

(Cloud2edge installation issues were resolved with @casafurix in #481. In general, the Gitter channel or Stackoverflow (with eclipse-iot tag) are good places for asking questions, or, in case of a chart or documentation bug, an issue can be raised here.)

[calohmn]

@sgloutnikov yes, the cloud2edge chart should be updated. A PR is welcome here.

[calohmn]

@sgloutnikov The update actually looked quite straightforward. I've integrated it in PR #493 .

[sgloutnikov]

Really strange, I'm unable to bring up the environment in a fresh k3s or minikube single node cluster, any tips are greatly appreciated. This behaviour seems identical for k3s with LoadBalancer or minikube without.

Looks like a bunch of Ditto pods failing with rediness probe and Akka errors in logs, which chains down to nginx and swagger-ui not starting up.

Helm install:

 cloud2edge git:(pr/493) helm install -n $NS --wait --timeout 20m --set hono.useLoadBalancer=true --set ditto.nginx.service.type=LoadBalancer $RELEASE ./

Get pods:

 ➜  ~ k get pods
NAME                                           READY   STATUS                  RESTARTS        AGE
c2e-ditto-nginx-6859974f89-5s5gv               0/1     Init:0/1                0               10m
c2e-ditto-dittoui-649d4b487d-jz7nt             1/1     Running                 0               10m
c2e-ditto-gateway-9fc695954-tmb75              0/1     Running                 0               10m
c2e-ditto-thingssearch-76c969667d-mpmqk        0/1     Running                 0               10m
c2e-ditto-policies-56fc485445-pxmqb            0/1     Running                 0               10m
c2e-ditto-things-6944d6d6fd-rzjn7              0/1     Running                 0               10m
c2e-ditto-connectivity-7d94b56b85-85rzt        0/1     Running                 0               10m
c2e-zookeeper-0                                1/1     Running                 0               10m
ditto-mongodb-778c99d5bf-2lsq5                 1/1     Running                 0               10m
c2e-kafka-0                                    1/1     Running                 0               10m
c2e-service-auth-7d7d8dfdf6-p4d84              1/1     Running                 0               10m
c2e-service-device-registry-86484b7f7b-wkjxg   1/1     Running                 0               10m
c2e-service-command-router-587d5c68fd-kg5fl    1/1     Running                 0               10m
c2e-adapter-mqtt-687c66bbdd-548pc              1/1     Running                 0               10m
c2e-adapter-amqp-77579d66f4-hgzt8              1/1     Running                 0               10m
c2e-adapter-http-d56575cbf-kmlkl               1/1     Running                 0               10m
c2e-ditto-swaggerui-646678bdfd-5lkr8           0/1     Init:CrashLoopBackOff   6 (3m36s ago)   10m

Get services:

 ➜  ~ k get services
NAME                              TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE
c2e-kafka-headless                ClusterIP      None            <none>         9092/TCP,9093/TCP            10m
c2e-zookeeper-headless            ClusterIP      None            <none>         2181/TCP,2888/TCP,3888/TCP   10m
ditto-mongodb                     ClusterIP      10.43.206.8     <none>         27017/TCP                    10m
c2e-service-device-registry       ClusterIP      10.43.89.217    <none>         5671/TCP,8080/TCP,8443/TCP   10m
c2e-service-auth                  ClusterIP      10.43.125.245   <none>         5671/TCP,8088/TCP            10m
c2e-ditto-swaggerui               ClusterIP      10.43.22.151    <none>         8080/TCP                     10m
c2e-service-command-router        ClusterIP      10.43.133.236   <none>         5671/TCP                     10m
c2e-ditto-dittoui                 ClusterIP      10.43.239.86    <none>         8080/TCP                     10m
c2e-ditto-gateway                 ClusterIP      10.43.142.65    <none>         8080/TCP                     10m
c2e-kafka                         ClusterIP      10.43.32.223    <none>         9092/TCP,9094/TCP            10m
c2e-zookeeper                     ClusterIP      10.43.159.167   <none>         2181/TCP,2888/TCP,3888/TCP   10m
c2e-kafka-0-external              LoadBalancer   10.43.151.26    172.XX.XX.XX   9094:32094/TCP               10m
c2e-adapter-amqp                  LoadBalancer   10.43.163.121   172.XX.XX.XX   5671:31200/TCP               10m
c2e-ditto-nginx                   LoadBalancer   10.43.193.12    172.XX.XX.XX   8080:30341/TCP               10m
c2e-service-device-registry-ext   LoadBalancer   10.43.181.30    172.XX.XX.XX   28443:31396/TCP              10m
c2e-adapter-http                  LoadBalancer   10.43.50.24     172.XX.XX.XX   8443:32470/TCP               10m
c2e-adapter-mqtt                  LoadBalancer   10.43.186.5     172.XX.XX.XX   8883:32334/TCP               10m

Describe nginx pod and logs:

...
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  ...
  Normal  Pulling    12m   kubelet            Pulling image "rancher/curlimages-curl:7.73.0"
  Normal  Pulled     12m   kubelet            Successfully pulled image "rancher/curlimages-curl:7.73.0" in 3.785419237s (3.785428409s including waiting)
  Normal  Created    12m   kubelet            Created container wait-for-gateway
  Normal  Started    12m   kubelet            Started container wait-for-gateway
  
  
   ~ k logs c2e-ditto-nginx-6859974f89-5s5gv
Defaulted container "ditto-nginx" out of: ditto-nginx, wait-for-gateway (init)
Error from server (BadRequest): container "ditto-nginx" in pod "c2e-ditto-nginx-6859974f89-5s5gv" is waiting to start: PodInitializing

Describe gateway pod:

Containers:
  ditto-gateway:
    Container ID:   containerd://bcc2029e8d57fa99a7cc8514f6ef97eb9cc743131ad7bb9feedd0db9bd5d57db
    Image:          docker.io/eclipse/ditto-gateway:3.3.6
    ...
    Liveness:   http-get http://:management/alive delay=160s timeout=5s period=10s #success=1 #failure=4
    Readiness:  http-get http://:management/ready delay=30s timeout=3s period=10s #success=1 #failure=3

Things Logs:

k logs c2e-ditto-things-6944d6d6fd-rzjn7
...
{"@timestamp":"2023-09-07T22:19:03.654+02:00","@version":"1","message":"Probing [http://10-42-0-21.spg-dev.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Exceeded configured max-open-requests value of [32]. This means that the request queue of this pool (HostConnectionPoolSetup(10-42-0-21.spg-dev.pod.cluster.local,8558,ConnectionPoolSetup(ConnectionPoolSettings(4,0,0,32,1,Duration.Inf,100 milliseconds,2 minutes,30 seconds,Duration.Inf,ClientConnectionSettings(Some(User-Agent: akka-http/10.2.10),10 seconds,60 seconds,512,None,WebSocketSettings(<function0>,ping,Duration.Inf,akka.http.impl.settings.WebSocketSettingsImpl$$$Lambda$1168/0x00007f1aeb6e0738@4c674487,false),List(),ParserSettings(2048,16,64,64,8192,64,Some(9223372036854775807),8388608,256,1048576,5,Strict,RFC6265,true,Set(),Full,Error,Error,Error,HashMap(If-Range -> 0, If-Modified-Since -> 0, If-Unmodified-Since -> 0, default -> 12, If-None-Match -> 0, User-Agent -> 32, Content-MD5 -> 0, Date -> 0, If-Match -> 0),false,false,true,akka.util.ConstantFun$$$Lambda$1116/0x00007f1aeb6a7248@35e995d0,akka.util.ConstantFun$$$Lambda$1116/0x00007f1aeb6a7248@35e995d0,akka.util.ConstantFun$$$Lambda$1117/0x00007f1aeb6a7610@3c4dff45),100 milliseconds,None,Http2ClientSettingsImpl(256,65536,10000000,512000,1024,false,0 seconds,0 seconds,0,3 seconds,100 milliseconds,2 minutes,None),TCPTransport),1 second,List()),akka.http.scaladsl.HttpConnectionContext$@5dc42dd8,akka.event.MarkerLoggingAdapter@6c563e20))) has completely filled up because the pool currently does not process requests fast enough to handle the incoming request load. Please retry the request later. See https://doc.akka.io/docs/akka-http/current/scala/http/client-side/pool-overflow.html for more information.","logger_name":"akka.management.cluster.bootstrap.internal.HttpContactPointBootstrap","thread_name":"ditto-cluster-akka.actor.default-dispatcher-11","level":"WARN","level_value":30000,"akkaAddress":"akka://ditto-cluster@10.42.0.18:2551","sourceThread":"ditto-cluster-akka.actor.default-dispatcher-11","akkaSource":"akka://ditto-cluster@10.42.0.18:2551/system/bootstrapCoordinator/contactPointProbe-10-42-0-21.spg-dev.pod.cluster.local-8558"}
{"@timestamp":"2023-09-07T22:19:03.935+02:00","@version":"1","message":"Discovered [5] contact points, confirmed [0], which is less than the required [5], retrying","logger_name":"akka.management.cluster.bootstrap.LowestAddressJoinDecider","thread_name":"ditto-cluster-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000,"akkaAddress":"akka://ditto-cluster@10.42.0.18:2551","akkaContactPoints":"10-42-0-21.spg-dev.pod.cluster.local:8558, 10-42-0-22.spg-dev.pod.cluster.local:8558, 10-42-0-17.spg-dev.pod.cluster.local:8558, 10-42-0-18.spg-dev.pod.cluster.local:8558, 10-42-0-19.spg-dev.pod.cluster.local:8558","sourceThread":"ditto-cluster-akka.actor.default-dispatcher-13","akkaSource":"LowestAddressJoinDecider(akka://ditto-cluster)","akkaSeedNodes":"","tags":["akkaBootstrapInProgress"]}
{"@timestamp":"2023-09-07T22:19:04.204+02:00","@version":"1","message":"Probing [http://10-42-0-17.spg-dev.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Exceeded configured max-open-requests value of [32]. This means that the request queue of this pool (HostConnectionPoolSetup(10-42-0-17.spg-dev.pod.cluster.local,8558,ConnectionPoolSetup(ConnectionPoolSettings(4,0,0,32,1,Duration.Inf,100 milliseconds,2 minutes,30 seconds,Duration.Inf,ClientConnectionSettings(Some(User-Agent: akka-http/10.2.10),10 seconds,60 seconds,512,None,WebSocketSettings(<function0>,ping,Duration.Inf,akka.http.impl.settings.WebSocketSettingsImpl$$$Lambda$1168/0x00007f1aeb6e0738@4c674487,false),List(),ParserSettings(2048,16,64,64,8192,64,Some(9223372036854775807),8388608,256,1048576,5,Strict,RFC6265,true,Set(),Full,Error,Error,Error,HashMap(If-Range -> 0, If-Modified-Since -> 0, If-Unmodified-Since -> 0, default -> 12, If-None-Match -> 0, User-Agent -> 32, Content-MD5 -> 0, Date -> 0, If-Match -> 0),false,false,true,akka.util.ConstantFun$$$Lambda$1116/0x00007f1aeb6a7248@35e995d0,akka.util.ConstantFun$$$Lambda$1116/0x00007f1aeb6a7248@35e995d0,akka.util.ConstantFun$$$Lambda$1117/0x00007f1aeb6a7610@3c4dff45),100 milliseconds,None,Http2ClientSettingsImpl(256,65536,10000000,512000,1024,false,0 seconds,0 seconds,0,3 seconds,100 milliseconds,2 minutes,None),TCPTransport),1 second,List()),akka.http.scaladsl.HttpConnectionContext$@5dc42dd8,akka.event.MarkerLoggingAdapter@6c563e20))) has completely filled up because the pool currently does not process requests fast enough to handle the incoming request load. Please retry the request later. See https://doc.akka.io/docs/akka-http/current/scala/http/client-side/pool-overflow.html for more information.","logger_name":"akka.management.cluster.bootstrap.internal.HttpContactPointBootstrap","thread_name":"ditto-cluster-akka.actor.default-dispatcher-12","level":"WARN","level_value":30000,"akkaAddress":"akka://ditto-cluster@10.42.0.18:2551","sourceThread":"ditto-cluster-akka.actor.default-dispatcher-13","akkaSource":"akka://ditto-cluster@10.42.0.18:2551/system/bootstrapCoordinator/contactPointProbe-10-42-0-17.spg-dev.pod.cluster.local-8558"}

[calohmn]

@sgloutnikov Good point. I actually wasn't testing with ditto.nginx.service.type=LoadBalancer. I was testing without any such chart parameters being set (the hono.useLoadBalancer=true is the default anyway - we need to adapt the documentation there).

About ditto.nginx.service.type=LoadBalancer we need to check whether there's an issue there with the Ditto chart. I actually don't see setting this option to LoadBalancer being documented in the Ditto repo.

(EDIT: I've tested using ditto.nginx.service.type=LoadBalancer now and found no issues.)

[sgloutnikov]

@calohmn If I install each chart separately, the Hono chart installed fine. However, the latest Ditto chart has the same problem as above. I am installing with the default command which pulls the latest version: helm install -n ditto my-ditto oci://registry-1.docker.io/eclipse/ditto --wait.

In what environment are you testing the cloud2edge chart that it's able to install properly? I'm unable to get it to install your PR version, with or without extra parameters, on my local single node k3s or minikube.

[calohmn]

@sgloutnikov I'm using minikube v1.29.0 (with v1.31.2 I have the issue that pods are stuck in "terminating" state when uninstalling the chart). I'm running minikube tunnel.

[sgloutnikov]

I'm convinced the installation issues above are specific to my environment, most likely due to DNS inside the Kubernetes cluster and pods being unable to reach each other for the Akka readiness checks. I was able to install the Ditto Hono chart in minikube with a VirtualBox driver when the Docker one before was giving me the above-mentioned problems. Can't pin it down exactly, but will report if I find more.

[sgloutnikov]

Resolved in #493