Optimize MQTT topic structure for main and child devices

[reubenmiller]

Is your refactoring request related to a problem? Please describe.

Review MQTT topic structure for both the existing and planned topics to create a consistent interface across the different telemetry data types and commands.

The current tedge topic structure:

Topic	Description
tedge/alarms/{severity}/{type}	Alarm for the main device
tedge/alarms/{severity}/{type}/{child}	Alarm for child device
tedge/events/{type}	Event for main device
tedge/events/{type}/{child}	Event for child device
tedge/measurements	Measurement for main device
tedge/measurements/{child}	Measurement for child device
tedge/health/{service}	Service status for main device
tedge/health/{child}/{service}	Service status for child device
tedge/commands/#	Commands for main device
tedge/{child}/commands/#	Commands for child device

And the following list is a rough approximation of the future/planned topics (subject to change):

Topic	Description
tedge/register/child/{child}	Register a child device from the main device
tedge/{parent}/register/child/{child}	(future proofing only) Register a child device of another child
tedge/inventory/update	Update inventory of main device
tedge/{child}/inventory/update	Update inventory of child device
tedge/inventory/update/{fragment}	Update inventory fragment of main device
tedge/{child}/inventory/update/{fragment}	Update inventory fragment of child device
tedge/inventory/delete/{fragment}	Delete inventory fragment of child device
tedge/{child}/inventory/delete/{fragment}	Delete inventory fragment of child device

Problem areas

• Measurements don't support specifying a type in the topic structure, e.g. tedge/measurements/{type} Using the topic benefit of MQTT for Measurement Mappings #509
• No distinction between a child device and a service. This is a problem because if a service publishes to the tedge/measurement/{service} topic, tedge creates a child device (which is unexpected as a service should not be a child device) Support publishing service telemetry data #1971
• Creating Mosquitto Access Controls (via the Dynamic Security Plugin) for the topic structure is overly complicated due to the child device id being used as a suffix of a variable length topic name
• Subscribing to all data related to a single child device is difficult when the child id is a suffix. It would be much easier for child devices to subscribe using tedge/{child}/# Guideline for tedge MQTT topic schemes to accommodate child devices #1351
• Subscribing to all main device data is not possible as it is difficult to write a mqtt topic wildcard pattern to only select tedge main device data
• It difficult to have optional type part in the topic as it makes it difficult to detect which part is the device/child and which is the type.
• tedge/health topics don't support filtering by service type. e.g. if a device only wants to subscribe to services of a specific type, then it will have to subscribe to all services and filter based on the content of the message payload (which is inefficient)
• Topic structure always assumes that the data is going directly to the cloud and will not go through some intermediate processing before being sent to the cloud (e.g. streaming analytics)

Future proofing

• How would sending telemetry data for nested child devices look like? Whilst this is not necessary at the moment, the design should be flexible enough that it can be adapted to meet this requirement in the future.

Other considerations

• AWS IoT Core topic path limitation: The maximum number of forward slashes (/) in the MQTT topic name for AWS IoT Core is seven.

Describe the solution you'd like

• A topic structure which addresses each of the Problem areas detailed in the previous section
• A MQTT client should be able to subscribe to all data for either the main device or a single child device via one subscription pattern

Describe alternatives you've considered

Additional context

Below are some links to provide some insight into best practices by other systems.

• AWS Topic Design
• MQTT Sparkplug for IOT

[didier-wenzek]

Here are the specifications for the new MQTT topic schema: https://thin-edge.github.io/thin-edge.io/next/references/mqtt-api/