Connection between Hono and Ditto - New tenant - Failed

[paulocabrita-ionos]

Hello everyone.

I'm following the tour: https://www.eclipse.org/packages/packages/cloud2edge/tour/ and my aim is to create a new tenant, create some devices, send data to Hono and manipulate Ditto. Everything works fine except that Ditto does not receive any data.

I also noticed that the site has instructions to create a new connection between Hono and Ditto using AMQP but... the example that is also deployed uses Kafka. The example uses the domain "org.eclipse.packages.c2e".

If I add devices to the "org.eclipse.packages.c2e" domain everything works fine.

My curls are the following:

• Tenant
  curl -i -X POST http://${REGISTRY_IP}:${REGISTRY_PORT_HTTP}/v1/tenants/skynet
• New device
  curl -i -X POST http://${REGISTRY_IP}:${REGISTRY_PORT_HTTP}/v1/devices/skynet/skynet:my-device-1
• Device password
  curl -i -X PUT -H "Content-Type: application/json" --data '[ { "type": "hashed-password", "auth-id": "userRobot", "secrets": [{ "pwd-plain": "topSecret" }] }]' http://${REGISTRY_IP}:${REGISTRY_PORT_HTTP}/v1/credentials/skynet/skynet:my-device-1
• AMQP Connection (following the "tour"). HONO_TENANT and DITTO_DEVOPS_PWD are correct.
  curl -i -X POST -u devops:${DITTO_DEVOPS_PWD} -H 'Content-Type: application/json' --data '{ "targetActorSelection": "/system/sharding/connection", "headers": { "aggregate": false }, "piggybackCommand": { "type": "connectivity.commands:createConnection", "connection": { "id": "hono-connection-for-'"${HONO_TENANT}"'", "connectionType": "amqp-10", "connectionStatus": "open", "uri": "amqp://consumer%40HONO:verysecret@'"${RELEASE}"'-dispatch-router-ext:15672", "failoverEnabled": true, "sources": [ { "addresses": [ "telemetry/'"${HONO_TENANT}"'", "event/'"${HONO_TENANT}"'" ], "authorizationContext": [ "pre-authenticated:hono-connection" ], "enforcement": { "input": "{{ header:device_id }}", "filters": [ "{{ entity:id }}" ] }, "headerMapping": { "hono-device-id": "{{ header:device_id }}", "content-type": "{{ header:content-type }}" }, "replyTarget": { "enabled": true, "address": "{{ header:reply-to }}", "headerMapping": { "to": "command/'"${HONO_TENANT}"'/{{ header:hono-device-id }}", "subject": "{{ header:subject | fn:default(topic:action-subject) | fn:default(topic:criterion) }}-response", "correlation-id": "{{ header:correlation-id }}", "content-type": "{{ header:content-type | fn:default('"'"'application/vnd.eclipse.ditto+json'"'"') }}" }, "expectedResponseTypes": [ "response", "error" ] }, "acknowledgementRequests": { "includes": [], "filter": "fn:filter(header:qos,'"'"'ne'"'"','"'"'0'"'"')" } }, { "addresses": [ "command_response/'"${HONO_TENANT}"'/replies" ], "authorizationContext": [ "pre-authenticated:hono-connection" ], "headerMapping": { "content-type": "{{ header:content-type }}", "correlation-id": "{{ header:correlation-id }}", "status": "{{ header:status }}" }, "replyTarget": { "enabled": false, "expectedResponseTypes": [ "response", "error" ] } } ], "targets": [ { "address": "command/'"${HONO_TENANT}"'", "authorizationContext": [ "pre-authenticated:hono-connection" ], "topics": [ "_/_/things/live/commands", "_/_/things/live/messages" ], "headerMapping": { "to": "command/'"${HONO_TENANT}"'/{{ thing:id }}", "subject": "{{ header:subject | fn:default(topic:action-subject) }}", "content-type": "{{ header:content-type | fn:default('"'"'application/vnd.eclipse.ditto+json'"'"') }}", "correlation-id": "{{ header:correlation-id }}", "reply-to": "{{ fn:default('"'"'command_response/'"${HONO_TENANT}"'/replies'"'"') | fn:filter(header:response-required,'"'"'ne'"'"','"'"'false'"'"') }}" } }, { "address": "command/'"${HONO_TENANT}"'", "authorizationContext": [ "pre-authenticated:hono-connection" ], "topics": [ "_/_/things/twin/events", "_/_/things/live/events" ], "headerMapping": { "to": "command/'"${HONO_TENANT}"'/{{ thing:id }}", "subject": "{{ header:subject | fn:default(topic:action-subject) }}", "content-type": "{{ header:content-type | fn:default('"'"'application/vnd.eclipse.ditto+json'"'"') }}", "correlation-id": "{{ header:correlation-id }}" } } ] } } }' http://${DITTO_API_IP}:${DITTO_API_PORT_HTTP}/devops/piggyback/connectivity
• Creation of the Digital Twin. I used a json file to do it (I created before the policy)
  curl -u ditto:ditto -X PUT -H 'Content-Type: application/json' -d @../robotArm_v1.json http://${DITTO_API_IP}:${DITTO_API_PORT_HTTP}/api/2/things/skynet:my-device-1
• I send data but nothing gets to the Digital Twin;
• I repeated the same process but this time I created a Kafka connection like this (I'm not sure if it's correct)
  curl -i -X POST -u devops:${DITTO_DEVOPS_PWD} -H 'Content-Type: application/json' --data '{ "targetActorSelection": "/system/sharding/connection", "headers": { "aggregate": false }, "piggybackCommand": { "type": "connectivity.commands:createConnection", "connection": { "id": "hono-skynet", "name": "skynet", "connectionType": "kafka", "connectionStatus": "open", "uri": "ssl://ditto-c2e:verysecret@c2e-kafka:9092", "sources": [ { "addresses": [ "hono.telemetry.skynet" ], "consumerCount": 1, "qos": 0, "authorizationContext": [ "pre-authenticated:hono-connection" ], "enforcement": { "input": "{{ header:device_id }}", "filters": [ "{{ entity:id }}" ] }, "acknowledgementRequests": { "includes": [], "filter": "fn:delete()" }, "headerMapping": {}, "replyTarget": { "address": "hono.command.skynet/{{ thing:id }}", "headerMapping": { "device_id": "{{ thing:id }}", "subject": "{{ header:subject | fn:default(topic:action-subject) | fn:default(topic:criterion) }}-response", "correlation-id": "{{ header:correlation-id }}" }, "expectedResponseTypes": [ "response", "error" ], "enabled": true } }, { "addresses": [ "hono.event.skynet" ], "consumerCount": 1, "qos": 1, "authorizationContext": [ "pre-authenticated:hono-connection" ], "enforcement": { "input": "{{ header:device_id }}", "filters": [ "{{ entity:id }}" ] }, "acknowledgementRequests": { "includes": [] }, "headerMapping": {}, "replyTarget": { "address": "hono.command.skynet/{{ thing:id }}", "headerMapping": { "device_id": "{{ thing:id }}", "subject": "{{ header:subject | fn:default(topic:action-subject) | fn:default(topic:criterion) }}-response", "correlation-id": "{{ header:correlation-id }}" }, "expectedResponseTypes": [ "response", "error" ], "enabled": true } }, { "addresses": [ "hono.command_response.skynet" ], "consumerCount": 1, "qos": 0, "authorizationContext": [ "pre-authenticated:hono-connection" ], "enforcement": { "input": "{{ header:device_id }}", "filters": [ "{{ entity:id }}" ] }, "acknowledgementRequests": { "includes": [], "filter": "fn:delete()" }, "headerMapping": { "correlation-id": "{{ header:correlation-id }}", "status": "{{ header:status }}" }, "replyTarget": { "enabled": false } } ], "targets": [ { "address": "hono.command.skynet/{{ thing:id }}", "topics": [ "_/_/things/live/commands", "_/_/things/live/messages" ], "authorizationContext": [ "pre-authenticated:hono-connection" ], "headerMapping": { "device_id": "{{ thing:id }}", "subject": "{{ header:subject | fn:default(topic:action-subject) }}", "response-required": "{{ header:response-required }}", "correlation-id": "{{ header:correlation-id }}" } }, { "address": "hono.command.skynet/{{thing:id}}", "topics": [ "_/_/things/twin/events", "_/_/things/live/events" ], "authorizationContext": [ "pre-authenticated:hono-connection" ], "headerMapping": { "device_id": "{{ thing:id }}", "subject": "{{ header:subject | fn:default(topic:action-subject) }}", "correlation-id": "{{ header:correlation-id }}" } } ], "clientCount": 1, "failoverEnabled": true, "validateCertificates": true, "processorPoolSize": 1, "specificConfig": { "saslMechanism": "plain", "bootstrapServers": "c2e-kafka:9092", "groupId": "skynet_{{ connection:id }}" } } } }' http://${DITTO_API_IP}:${DITTO_API_PORT_HTTP}/devops/piggyback/connectivity
• When I check the connection, it seems that everything is ok
  curl -X POST -i -u devops:${DITTO_DEVOPS_PWD} -H 'Content-Type: application/json' -d '{ "targetActorSelection": "/system/sharding/connection", "headers": {}, "piggybackCommand": { "type": "connectivity.commands:retrieveConnection", "connectionId": "hono-skynet" } }' http://${DITTO_API_IP}:${DITTO_API_PORT_HTTP}/devops/piggyback/connectivity?timeout=8s

Like I said, if I use the tenant from the example "org.eclipse.packages.c2e", it works fine but not when I create a new tenant.

I think this a related issue:

• New Hono connection type configuration misses configuring the Hono tenant eclipse-ditto/ditto#1574
• C2E "Tour" documentation is outdated, still refers to AMQP 1.0 connections #432

Also, I think it should be great to update the "tour" site in order to have instructions to connect Hono and Ditto using Kafka.

Does anyone have a working new kafka connection running with a new tenant?

Please advise.

Thanks
PJC

[BHUVANESHWARI967]

Hi,
Did you get any solution for this?

[paulocabrita-ionos]

Hi.

Unfortunately, no.

I have to give it a try again but with an another version.

Are you facing the same issue?

[BHUVANESHWARI967]

Hi,

Was you connection getting created?

While I try to create connection it throws error saying "The connection Id is invalid". Do you know what should be given as connection id for amqp or kafka connection?

[calohmn]

@paulocabrita-ionos I have created PR #493 to update the cloud2edge tour to create a Kafka connection there.
One thing that is missing in the connection payload shown in the issue text above is the "ca" entry with the Kafka broker self-signed certificate - I think this is needed for the used "validateCertificates": true option. The c2e tour instructions will include adding this ca entry.
In general, #493 isn't merged yet - as noted in the comments there I still see an issue in using that connection.

@BHUVANESHWARI967 You could already have a look at the updated tour chapter in the PR with an example Kafka connection JSON that gets created successfully. Following the instructions there requires the setCloud2EdgeEnv.sh file updated in that PR.

[calohmn]

@paulocabrita-ionos @BHUVANESHWARI967 The PR #493 has been merged now. Along with cloud2edge chart version 0.5.0, the updated c2e tour is online, including the updated chapter about creating the connection. (To follow the steps there, the setCloud2EdgeEnv.sh script has to be downloaded again.)

One issue in the used connection JSON above is, that the authorizationContext value (pre-authenticated:hono-connection) should be unique to the connection, containing pre-authenticated:hono-connection-skynet for example.

[paulocabrita-ionos]

@calohmn thanks a lot. I will take a look and tell something ASAP.

[paulocabrita-ionos]

@calohmn I followed the new tour guide and I had success creating a new tenant using Kafka.

If you agree, I close this issue.

Just a side note: the script setCloud2EdgeEnv.sh has an issue. It's assuming that the NODE_IP is being populated using the InternalIP field but if we use an external K8S there is no InternalIP, there'se the ExternalIP field. I installed the package using NodePort on a IONOS K8S. In that case, the populate of all the environment variables will fail.

[calohmn]

If you agree, I close this issue.

Yes, I think this can be closed.

Regarding the setCloud2EdgeEnv.sh script, does it work when replacing the NODE_IP=... line with

NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="ExternalIP")].address}' 2> /dev/null)
if [[ -z "$NODE_IP" ]] ; then
  NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}' 2> /dev/null)
fi

?

[paulocabrita-ionos]

I updated my previous answer because of some type-os.

I did something similar with the script. What you suggested worked :) So I think that the script can be updated on the repo.

Closing this issue.

Thanks ;)