Get Device Twin request from IoT Hub not working

[engemil]

Describe the bug

I'm working on a code sample to utilize the IoT Hub device twin, where the first step is to request the device twin from cloud to device. After the request is sent, there is no message received and there is no telling if the cloud (IoT Hub) received the request.

The code is based on the Azure IoT Hub Arduino Nano RP2040 Connect example from https://github.com/Azure/azure-sdk-for-c-arduino/tree/main/examples/Azure_IoT_Hub_Arduino_Nano_RP2040_Connect . I built the example with PlatformIO, which worked. However, the device twin request doesn't seem execute properly. The device twin code is based on:

• https://github.com/Azure/azure-sdk-for-c/blob/main/sdk/src/azure/iot/az_iot_hub_client_twin.c
• https://github.com/djaus2/Azure_IoT_Hub_Arduino_RPI_Pico_Telemetry

Exception or Stack Trace

Serial reading from the microcontroller

(Sensitive information sensured out with XXXXX)

[INFO] Attempting to connect to WIFI SSID: XXXXX

[INFO] WiFi connected, IP address: XXXXX, Strength (dBm): -48
[INFO] Syncing time.
.........
[INFO] Time synced!
[INFO] Initializing Azure IoT Hub client.
[INFO] Azure IoT Hub hostname: XXXXX.azure-devices.net
[INFO] Azure IoT Hub client initialized.
[INFO] Initializing MQTT client.
[INFO] UTC Current time: 2023-03-17 12:31:57 (epoch: 1679056317 secs)
[INFO] UTC Expiry time: 2023-03-17 13:31:57 (epoch: 1679059917 secs)
[INFO] Local Current time: 2023-03-17 14:31:57
[INFO] Local Expiry time: 2023-03-17 15:31:57
[INFO] Client ID: XXXXX
[INFO] Username: XXXXX.azure-devices.net/XXXXX/?api-version=2020-09-30&DeviceClientType=c/1.5.0-beta.1(ard;nanorp2040connect)
[INFO] MQTT client initialized.
[INFO] Connecting to Azure IoT Hub.
[INFO] Connected to your Azure IoT Hub!
[INFO] Subscribed to MQTT topic: devices/+/messages/devicebound/#
[INFO] Subscribed to MQTT topic: $iothub/twin/res/#
[INFO] Subscribed to MQTT topic: $iothub/twin/PATCH/properties/desired/#
[INFO] Arduino Nano RP2040 Connect requesting device twin document . . . 
[INFO] MQTT Client publishing to $iothub/twin/GET/?$rid=get_twin
[INFO] Request for Device Twin Document sent.

To Reproduce

• Set up a IoT Hub with a IoT device with default settings (symmetric key).
• Clone this repo, and use the Arduino Nano RP2040 example: https://github.com/Azure/azure-sdk-for-c-arduino/tree/main/examples/Azure_IoT_Hub_Arduino_Nano_RP2040_Connect
• Change the main file (<main_file>.ino if using Arduino IDE, or main.cpp if using PlatformIO in VS Code)
• Update the iot_configs.h file, build, upload and monitor.

Code Snippet

The .ino-file (main.cpp file for PlatformIO projects)

/*--- Libraries ---*/
// Arduino
#include <Arduino.h> // Needs to be added when not using Arduino IDE

// C99 libraries.
#include <cstdbool>
#include <cstdlib>
#include <cstring>
#include <time.h>

// Libraries for SSL client, MQTT client, and WiFi connection.
#include <ArduinoBearSSL.h>
#include <ArduinoMqttClient.h>
#include <WiFiNINA.h>

// Libraries for SAS token generation.
#include <ECCX08.h>

// Azure IoT SDK for C includes.
#include <az_core.h>
#include <az_iot.h>

// Azure IoT Device Twin Utilitie(s)
//#include <ArduinoJson.h>

// Sample header.
//#include "iot_configs.h"
#include "iot_configs_secrets.h"

// Logging
#include "SerialLogger.h"

/*--- Macros ---*/
#define BUFFER_LENGTH_MQTT_CLIENT_ID 256
#define BUFFER_LENGTH_MQTT_PASSWORD 256
#define BUFFER_LENGTH_MQTT_TOPIC 128
#define BUFFER_LENGTH_MQTT_USERNAME 512
#define BUFFER_LENGTH_SAS 32
#define BUFFER_LENGTH_SAS_ENCODED_SIGNED_SIGNATURE 64
#define BUFFER_LENGTH_SAS_SIGNATURE 512
#define BUFFER_LENGTH_DATETIME_STRING 256

// QoS levels
#define QOS_0 0 // At most once (not guaranteed)
#define QOS_1 1 // At least once (guaranteed)
#define QOS_2 2 // Exactly once (guaranteed)

#define LED_PIN 2 // High on error. Briefly high for each successful send.

// Time and Time Zone.
#define SECS_PER_MIN 60
#define SECS_PER_HOUR (SECS_PER_MIN * 60)
#define GMT_OFFSET_SECS (IOT_CONFIG_DAYLIGHT_SAVINGS ? \
                        ((IOT_CONFIG_TIME_ZONE + IOT_CONFIG_TIME_ZONE_DAYLIGHT_SAVINGS_DIFF) * SECS_PER_HOUR) : \
                        (IOT_CONFIG_TIME_ZONE * SECS_PER_HOUR))

// Exit into infinite loop
#define EXIT_LOOP(condition, errorMessage) \
  do \
  { \
    if (condition) { \
      Logger.Error(errorMessage); \
      while (1); \
    } \
  } while (0)
  

/*--- Sample static variables --*/
// Clients for WiFi connection, SSL, MQTT, and Azure IoT SDK for C.
static WiFiClient wiFiClient;
static BearSSLClient bearSSLClient(wiFiClient);
static MqttClient mqttClient(bearSSLClient);
static az_iot_hub_client azIoTHubClient;

// MQTT variables.
static char mqttClientId[BUFFER_LENGTH_MQTT_CLIENT_ID];
static char mqttUsername[BUFFER_LENGTH_MQTT_USERNAME];
static char mqttPassword[BUFFER_LENGTH_MQTT_PASSWORD];

// Device Twin variables
static az_span const twin_document_topic_request_id = AZ_SPAN_LITERAL_FROM_STR("get_twin");
static void requestDeviceTwinDocument();

/*--- Functions ---*/
// Initialization and connection functions.
void connectToWiFi();
void initializeAzureIoTHubClient();
void initializeMQTTClient();
void connectMQTTClientToAzureIoTHub();

// Telemetry and message-callback functions.
void onMessageReceived(int messageSize);

// SAS Token related functions.
static void generateMQTTPassword();
static void generateSASBase64EncodedSignedSignature(
    uint8_t const* sasSignature, size_t const sasSignatureSize,
    uint8_t* encodedSignedSignature, size_t encodedSignedSignatureSize,
    size_t* encodedSignedSignatureLength);
static uint64_t getSASTokenExpirationTime(uint32_t minutes);

// Time and Error functions.
static unsigned long getTime();
static String getFormattedDateTime(unsigned long epochTimeInSeconds);
static String mqttErrorCodeName(int errorCode);

/*---------------------------*/
/*    Main code execution    */
/*---------------------------*/

/*
 * setup:
 * Initialization and connection of serial communication, WiFi client, Azure IoT SDK for C client, 
 * and MQTT client.
 */
void setup() 
{
  while (!Serial);
  delay(1000);

  Serial.begin(SERIAL_LOGGER_BAUD_RATE);

  pinMode(LED_PIN, OUTPUT);
  digitalWrite(LED_PIN, HIGH);

  connectToWiFi();
  initializeAzureIoTHubClient();
  initializeMQTTClient();
  connectMQTTClientToAzureIoTHub();

  requestDeviceTwinDocument();

  digitalWrite(LED_PIN, LOW);
}

/*
 * loop:
 * Check for connection and reconnect if necessary.
 * Send Telemetry and receive messages.
 */
void loop() 
{
  if (WiFi.status() != WL_CONNECTED) 
  {
    connectToWiFi();
  }

  // MQTT loop must be called to process Telemetry and Cloud-to-Device (C2D) messages.
  mqttClient.poll();
  delay(50);
}

/*-----------------------------------------------*/
/*    Initialization and connection functions    */
/*-----------------------------------------------*/

/*
 * connectToWifi:
 * The WiFi client connects, using the provided SSID and password.
 * The WiFi client synchronizes the time on the device. 
 */
void connectToWiFi() 
{
  Logger.Info("Attempting to connect to WIFI SSID: " + String(IOT_CONFIG_WIFI_SSID));

  WiFi.begin(IOT_CONFIG_WIFI_SSID, IOT_CONFIG_WIFI_PASSWORD);
  while (WiFi.status() != WL_CONNECTED) 
  {
    Serial.print(".");
    delay(IOT_CONFIG_WIFI_CONNECT_RETRY_MS);
  }
  Serial.println();

  Logger.Info("WiFi connected, IP address: " + String(WiFi.localIP()) + ", Strength (dBm): " + WiFi.RSSI());
  Logger.Info("Syncing time.");

  while (getTime() == 0) 
  {
    Serial.print(".");
    delay(500);
  }
  Serial.println();

  Logger.Info("Time synced!");
}

/*
 * initializeAzureIoTHubClient:
 * The Azure IoT SDK for C client uses the provided hostname, device id, and user agent.
 */
void initializeAzureIoTHubClient() 
{
  Logger.Info("Initializing Azure IoT Hub client.");

  az_span hostname = AZ_SPAN_FROM_STR(IOT_CONFIG_IOTHUB_FQDN);
  az_span deviceId = AZ_SPAN_FROM_STR(IOT_CONFIG_DEVICE_ID);

  az_iot_hub_client_options options = az_iot_hub_client_options_default();
  options.user_agent = AZ_SPAN_FROM_STR(IOT_CONFIG_AZURE_SDK_CLIENT_USER_AGENT);

  int result = az_iot_hub_client_init(&azIoTHubClient, hostname, deviceId, &options);

  EXIT_LOOP(az_result_failed(result), "Failed to initialize Azure IoT Hub client. Return code: " + result);

  Logger.Info("Azure IoT Hub hostname: " + String(IOT_CONFIG_IOTHUB_FQDN));
  Logger.Info("Azure IoT Hub client initialized.");
}

/*
 * initializeMQTTClient:
 * The MQTT client uses the client id and username from the Azure IoT SDK for C client.
 * The MQTT client uses the generated password (the SAS token).
 */
void initializeMQTTClient() 
{
  Logger.Info("Initializing MQTT client.");
  
  int result;

  result = az_iot_hub_client_get_client_id(
      &azIoTHubClient, mqttClientId, sizeof(mqttClientId), NULL);
  EXIT_LOOP(az_result_failed(result), "Failed to get MQTT client ID. Return code: " + result);
  
  result = az_iot_hub_client_get_user_name(
      &azIoTHubClient, mqttUsername, sizeof(mqttUsername), NULL);
  EXIT_LOOP(az_result_failed(result), "Failed to get MQTT username. Return code: " + result);

  generateMQTTPassword(); // SAS Token

  // MQTT Client ID, username, and password
  mqttClient.setId(mqttClientId);
  mqttClient.setUsernamePassword(mqttUsername, mqttPassword);

  mqttClient.onMessage(onMessageReceived); // Set callback

  Logger.Info("Client ID: " + String(mqttClientId));
  Logger.Info("Username: " + String(mqttUsername));

  Logger.Info("MQTT client initialized.");
}

/*
 * connectMQTTClientToAzureIoTHub:
 * The SSL library sets a callback to validate the server certificate.
 * The MQTT client connects to the provided hostname. The port is pre-set.
 * The MQTT client subscribes to IoT topics.
 */
void connectMQTTClientToAzureIoTHub() 
{
  Logger.Info("Connecting to Azure IoT Hub.");

  // Set a callback to get the current time used to validate the server certificate.
  ArduinoBearSSL.onGetTime(getTime);

  while (!mqttClient.connect(IOT_CONFIG_IOTHUB_FQDN, AZ_IOT_DEFAULT_MQTT_CONNECT_PORT)) 
  {
    int code = mqttClient.connectError();
    Logger.Error("Cannot connect to Azure IoT Hub. Reason: " + mqttErrorCodeName(code) + ", Code: " + code);
    delay(5000);
  }

  Logger.Info("Connected to your Azure IoT Hub!");

  //mqttClient.subscribe(AZ_IOT_HUB_CLIENT_C2D_SUBSCRIBE_TOPIC);
  //Logger.Info("Subscribed to MQTT topic: " + String(AZ_IOT_HUB_CLIENT_C2D_SUBSCRIBE_TOPIC));

  //mqtt_client.subscribe(AZ_IOT_HUB_CLIENT_METHODS_SUBSCRIBE_TOPIC);
  //Logger.Info("Subscribed to MQTT topic: " + String(AZ_IOT_HUB_CLIENT_METHODS_SUBSCRIBE_TOPIC));

  mqttClient.subscribe(AZ_IOT_HUB_CLIENT_TWIN_RESPONSE_SUBSCRIBE_TOPIC, QOS_1); // (Default: QoS 0 At most once, not guaranteed)
  Logger.Info("Subscribed to MQTT topic: " + String(AZ_IOT_HUB_CLIENT_TWIN_RESPONSE_SUBSCRIBE_TOPIC));

  mqttClient.subscribe(AZ_IOT_HUB_CLIENT_TWIN_PATCH_SUBSCRIBE_TOPIC, QOS_1); // (Default: QoS 0 At most once, not guaranteed)
  Logger.Info("Subscribed to MQTT topic: " + String(AZ_IOT_HUB_CLIENT_TWIN_PATCH_SUBSCRIBE_TOPIC));

}

/*--------------------------------------------------------------*/
/*    Device Twins, Telemetry and message-callback functions    */
/*--------------------------------------------------------------*/

/*
 * onMessageReceived:
 * The function called when device receives a message on the subscribed topics.
 * Callback function signature is defined by the ArduinoMQTTClient library.
 * Message received is printed to the terminal.
 */
void onMessageReceived(int messageSize){

  Logger.Info("Message received: Topic: " + mqttClient.messageTopic() + ", Length: " + messageSize);
  //receivedMessageDeviceTwinDoc(messageSize);

/*
  Logger.Info("Message: ");
  while (mqttClient.available()) 
  {
    Serial.print((char)mqttClient.read());
  }
  Serial.println();
*/
}

/*
 * requestDeviceTwinDocument:
 * The Azure IoT SDK for C client creates the MQTT topic to publish a telemetry message.
 * The MQTT client creates and sends the telemetry mesage on the topic.
 */
static void requestDeviceTwinDocument() 
{
  Logger.Info("Arduino Nano RP2040 Connect requesting device twin document . . . ");

  char deviceTwinTopic[BUFFER_LENGTH_MQTT_TOPIC];
  // Get the Twin Document topic to publish the twin document request.
  int result = az_iot_hub_client_twin_document_get_publish_topic(
      &azIoTHubClient, twin_document_topic_request_id, deviceTwinTopic, sizeof(deviceTwinTopic), NULL);
  EXIT_LOOP(az_result_failed(result), "Failed to get twin document publish topic. Return code: " + result);

  // Publish the twin document request.
  mqttClient.beginMessage(deviceTwinTopic);
  //mqttClient.print("");
  mqttClient.endMessage();

  Logger.Info("MQTT Client publishing to " + (String)deviceTwinTopic);
  Logger.Info("Request for Device Twin Document sent.");
  delay(100);
}



/*************************************/
/*    SAS Token related functions    */
/*************************************/

/*
 * generateMQTTPassword:
 * The MQTT password is the generated SAS token. The process is:
 *    1. Get the SAS token expiration time from the provided value. (Default 60 minutes).
 *    2. Azure IoT SDK for C creates the SAS signature from this expiration time.
 *    3. Sign and encode the SAS signature.
 *    4. Azure IoT SDK for C creates the MQTT Password from the expiration time and the encoded,
 *       signed SAS signature.
 */
static void generateMQTTPassword() 
{
  int result;

  uint64_t sasTokenDuration = 0;
  uint8_t signature[BUFFER_LENGTH_SAS_SIGNATURE] = {0};
  az_span signatureAzSpan = AZ_SPAN_FROM_BUFFER(signature);
  uint8_t encodedSignedSignature[BUFFER_LENGTH_SAS_ENCODED_SIGNED_SIGNATURE] = {0};
  size_t encodedSignedSignatureLength = 0;

  // Get the signature. It will be signed later with the decoded device key.
  // To change the sas token duration, see IOT_CONFIG_SAS_TOKEN_EXPIRY_MINUTES in iot_configs.h
  sasTokenDuration = getSASTokenExpirationTime(IOT_CONFIG_SAS_TOKEN_EXPIRY_MINUTES);
  result = az_iot_hub_client_sas_get_signature(
      &azIoTHubClient, sasTokenDuration, signatureAzSpan, &signatureAzSpan);
  EXIT_LOOP(az_result_failed(result), "Could not get the signature for SAS Token. Return code: " + result);

  // Sign and encode the signature (b64 encoded, HMAC-SHA256 signing).
  // Uses the decoded device key.
  generateSASBase64EncodedSignedSignature(
      az_span_ptr(signatureAzSpan), az_span_size(signatureAzSpan),
      encodedSignedSignature, sizeof(encodedSignedSignature), &encodedSignedSignatureLength);

  // Get the MQTT password (SAS Token) from the base64 encoded, HMAC signed bytes.
  az_span encodedSignedSignatureAzSpan = az_span_create(encodedSignedSignature, 
                                                        encodedSignedSignatureLength);
  result = az_iot_hub_client_sas_get_password(
      &azIoTHubClient, sasTokenDuration, encodedSignedSignatureAzSpan, AZ_SPAN_EMPTY,
      mqttPassword, sizeof(mqttPassword), NULL);
  EXIT_LOOP(az_result_failed(result), "Could not get the MQTT password. Return code: " + result);
}

/*
 * generateSASBase64EncodedSignedSignature:
 * Sign and encode a signature. It is signed using the provided device key.
 * The process is:
 *    1. Decode the encoded device key.
 *    2. Sign the signature with the decoded device key.
 *    3. Encode the signed signature.
 */
static void generateSASBase64EncodedSignedSignature(
    uint8_t const* sasSignature, size_t const sasSignatureSize,
    uint8_t* encodedSignedSignature, size_t encodedSignedSignatureSize,
    size_t* encodedSignedSignatureLength) 
{
  int result;
  unsigned char sasDecodedKey[BUFFER_LENGTH_SAS] = {0};
  az_span sasDecodedKeySpan = AZ_SPAN_FROM_BUFFER(sasDecodedKey);
  int32_t sasDecodedKeyLength = 0;
  uint8_t sasHMAC256SignedSignature[BUFFER_LENGTH_SAS] = {0};

  // Decode the SAS base64 encoded device key to use for HMAC signing.
  az_span configDeviceKeySpan = az_span_create((uint8_t*)IOT_CONFIG_DEVICE_KEY, sizeof(IOT_CONFIG_DEVICE_KEY) - 1);
  result = az_base64_decode(sasDecodedKeySpan, configDeviceKeySpan, &sasDecodedKeyLength);
  EXIT_LOOP(result != AZ_OK, "az_base64_decode failed. Return code: " + result);

  // HMAC-SHA256 sign the signature with the decoded device key.
  result = ECCX08.begin();
  EXIT_LOOP(!result, "Failed to communicate with ATECC608.");
  
  result = ECCX08.nonce(sasDecodedKey);
  EXIT_LOOP(!result, "Failed to do nonce.");

  result = ECCX08.beginHMAC(0xFFFF);
  EXIT_LOOP(!result, "Failed to start HMAC operation.");

  result = ECCX08.updateHMAC(sasSignature, sasSignatureSize);
  EXIT_LOOP(!result, "Failed to update HMAC with signature.");

  result = ECCX08.endHMAC(sasHMAC256SignedSignature);
  EXIT_LOOP(!result, "Failed to end HMAC operation.");

  // Base64 encode the result of the HMAC signing.
  az_span signedSignatureSpan = az_span_create(sasHMAC256SignedSignature, sizeof(sasHMAC256SignedSignature));
  az_span encodedSignedSignatureSpan = az_span_create(encodedSignedSignature, encodedSignedSignatureSize);
  result = az_base64_encode(encodedSignedSignatureSpan, signedSignatureSpan, (int32_t*) encodedSignedSignatureLength);
  EXIT_LOOP(result != AZ_OK, "az_base64_encode failed. Return code: " + result);
}

/*
 * getSASTokenExpirationTime:
 * Calculate expiration time from current time and duration value.
 */
static uint64_t getSASTokenExpirationTime(uint32_t minutes) 
{
  unsigned long now = getTime();  // GMT
  unsigned long expiryTime = now + (SECS_PER_MIN * minutes); // For SAS Token
  unsigned long localNow = now + GMT_OFFSET_SECS;
  unsigned long localExpiryTime = expiryTime + GMT_OFFSET_SECS;

  Logger.Info("UTC Current time: " + getFormattedDateTime(now) + " (epoch: " + now + " secs)");
  Logger.Info("UTC Expiry time: " + getFormattedDateTime(expiryTime) + " (epoch: " + expiryTime + " secs)");
  Logger.Info("Local Current time: " + getFormattedDateTime(localNow));
  Logger.Info("Local Expiry time: " + getFormattedDateTime(localExpiryTime));

  return (uint64_t)expiryTime;
}

/**********************************/
/*    Time and Error functions    */
/**********************************/

/*
 * getTime:
 * WiFi client returns the seconds corresponding to GMT epoch time.
 * This function used as a callback by the SSL library to validate the server certificate
 * and in SAS token generation.
 */
static unsigned long getTime()
{
  return WiFi.getTime();
}

/*
 * getFormattedDateTime:
 * Custom formatting for epoch seconds. Used in logging.
 */
static String getFormattedDateTime(unsigned long epochTimeInSeconds) 
{
  char dateTimeString[BUFFER_LENGTH_DATETIME_STRING];

  time_t epochTimeInSecondsAsTimeT = (time_t)epochTimeInSeconds;
  struct tm* timeInfo = localtime(&epochTimeInSecondsAsTimeT);

  strftime(dateTimeString, 20, "%F %T", timeInfo);

  return String(dateTimeString);
}

/*
 * mqttErrorCodeName:
 * Legibly prints AruinoMqttClient library error enum values. 
 */
static String mqttErrorCodeName(int errorCode) 
{
  String errorMessage;
  switch (errorCode) 
  {
  case MQTT_CONNECTION_REFUSED:
    errorMessage = "MQTT_CONNECTION_REFUSED";
    break;
  case MQTT_CONNECTION_TIMEOUT:
    errorMessage = "MQTT_CONNECTION_TIMEOUT";
    break;
  case MQTT_SUCCESS:
    errorMessage = "MQTT_SUCCESS";
    break;
  case MQTT_UNACCEPTABLE_PROTOCOL_VERSION:
    errorMessage = "MQTT_UNACCEPTABLE_PROTOCOL_VERSION";
    break;
  case MQTT_IDENTIFIER_REJECTED:
    errorMessage = "MQTT_IDENTIFIER_REJECTED";
    break;
  case MQTT_SERVER_UNAVAILABLE:
    errorMessage = "MQTT_SERVER_UNAVAILABLE";
    break;
  case MQTT_BAD_USER_NAME_OR_PASSWORD:
    errorMessage = "MQTT_BAD_USER_NAME_OR_PASSWORD";
    break;
  case MQTT_NOT_AUTHORIZED:
    errorMessage = "MQTT_NOT_AUTHORIZED";
    break;
  default:
    errorMessage = "Unknown";
    break;
  }

  return errorMessage;
}

Expected behavior
The expected behavior of the code is to receive a "message" which includes the device twin Json document. However, this code only checks if there is any message received, and prints the message-topic and message length. The code has not yet implemented code to read the message. It ONLY verifies if the device twin was sent to device.

Setup:

• OS: Windows 11
• Microcontroller: Arduino Nano RP2040 Connect
• IDE: VS Code + PlatformIO or Arduino IDE 2.0.4
• PlatformIO libraries
  • Azure SDK for C 1.1.3
  • ArduinoBearSSL 1.7.3
  • ArduinoECCX08 1.3.7
  • ArduinoMqttClient 0.1.6
  • WiFiNINA 1.8.13

Additional context

Note1: If the problem does not relate to Azure SDK for C, I suspect limitations/problems with ArduinoMQTT. I would like to give an request for adding a device twin example in the Azure SDK for C Arduino repository.

Note 2: (Update 20.03.2023) I made another test with two functionaliteis implemented, twin get request and D2C message in the same code. The code first sends a D2C message, which I detect in Azure Portal (with az iot hub montir-events --hub-name ...). The second step is to send a twin get request over MQTT connection. After the request is sent, I can no longer recieve D2C messages in the cloud (IoT Hub). Not able to see this happen. Will try to replace the ArduinoMqttClient with SubPubClient.

Note 3: (Update 20.03.2023) Tested PubSubClient library, same result. Suspect problem with the arduino porting repository (https://github.com/Azure/azure-sdk-for-c-arduino). NB! Im posting this issue here, as the arduino-ported repo does not have a issue tab.

Note 4: (Update 21.03.2023) In the arduinoMqttClient library there is a macro (MQTT_CLIENT_DEBUG), commented out by default. With it enabled, it confirmed that the topics where subscripted to correctly (MQTT received data for each subscription). And, as suspected only sends and does NOT receive data when sending a "get_twin" request.

Note 5: (Update 21.03.2023) In D2C / Telemetry messages, the Azure SDK for C (for arduino) seem to treat the payload section as non-JSON syntax. Due to the backslashes for each quotation mark, as shown here:

{
    "event": {
        "origin": "XXXXXXXXXX",
        "module": "",
        "interface": "",
        "component": "",
        "payload": "{ \"msgCount\": 0 }"
    }
}

Is this as expected? or is the code in the Arduino framework affecting the handling of Strings/characters, such that the payload behaves wierd? And could this possible affect the "get_twin" request aswell?

Information Checklist
Please make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[meta-space]

I was able to successfully implement the twin document request by extending the example Azure_IoT_Central_ESP32 based on the sample ino that you've provided above. So thank you for that.

I noticed you are not subscribing to the topic '$iothub/twin/GET/?$rid=get_twin'. The response to the GET request will be posted there. Have you tried adding an additional subscription to that topic?

For what it's worth here's an extract from my log output which shows all topics I'm using

[ 17910][I] esp_mqtt_event_handler(): MQTT client connected (session_present=0x00000000).
[ 17914][I] mqtt_client_subscribe_function(): MQTT client subscribing to '$iothub/methods/POST/#'
[ 18015][I] esp_mqtt_event_handler(): MQTT topic subscribed (message id=0x00009e5c).
[ 18024][I] mqtt_client_subscribe_function(): MQTT client subscribing to '$iothub/twin/res/#'
[ 18099][I] esp_mqtt_event_handler(): MQTT topic subscribed (message id=0x0000569f).
[ 18104][I] mqtt_client_subscribe_function(): MQTT client subscribing to '$iothub/twin/PATCH/properties/desired/#'
[ 18174][I] esp_mqtt_event_handler(): MQTT topic subscribed (message id=0x0000c7b9).
[ 18184][I] azure_iot_do_work(): *******************  REQUESTING TWIN DOCUMENT  *********************
[ 18206][I] mqtt_client_publish_function(): MQTT client publishing to '$iothub/twin/GET/?$rid=get_twin'
[ 18220][I] mqtt_client_subscribe_function(): MQTT client subscribing to '$iothub/twin/GET/?$rid=get_twin'
[ 18290][I] esp_mqtt_event_handler(): MQTT message received.
[ 18290][I] on_device_twin_received(): Device Twin received: {"desired":{"state":"on","temp":55,"deviceName":"foo" .... 
[ 18425][I] esp_mqtt_event_handler(): MQTT topic subscribed (message id=0x00009019).
[ 18430][I]                  (): *******************  TWIN DOCUMENT GOTTEN  *************************

Hope this helps

[engemil]

@meta-space Interesting to see that it works for you with ESP32.

When it comes to $iothub/twin/GET/?$rid=get_twin, from the official examples, you are not supposed to subscribe to it. You only need to publish to it. I gave it a try to see if subscribing to it worked aswell. However, the same result.

...
[INFO] MQTT client initialized.
[INFO] Connecting to Azure IoT Hub.
[INFO] Connected to your Azure IoT Hub!
[INFO] Subscribed to MQTT topic: devices/+/messages/devicebound/#
[INFO] Subscribed to MQTT topic: $iothub/methods/POST/#
[INFO] Subscribed to MQTT topic: $iothub/twin/res/#
[INFO] Subscribed to MQTT topic: $iothub/twin/PATCH/properties/desired/#
[INFO] Subscribed to MQTT topic: $iothub/twin/GET/?$rid=get_twin
[INFO] Requesting device twin document . . . 
[INFO] MQTT Client publishing to $iothub/twin/GET/?$rid=get_twin

And no reply.

And as mentioned, the odd part is that I can't receive any C2D Messages nor publish with D2C messages,
after I tried to publish to $iothub/twin/GET/?$rid=get_twin.

[CIPop]

@engemil , first, to confirm everything is set up correctly on the Hub side, please use the same device identity with the sample within this repository: https://github.com/Azure/azure-sdk-for-c/blob/main/sdk/samples/iot/paho_iot_hub_twin_sample.c
I'm able to run the sample and use Azure IoT Explorer to modify twin's desired properties and receive updates on the device side.

The two topics you must subscribe to are: AZ_IOT_HUB_CLIENT_TWIN_PATCH_SUBSCRIBE_TOPIC and AZ_IOT_HUB_CLIENT_TWIN_RESPONSE_SUBSCRIBE_TOPIC.
Then, you must use the az_iot_hub_client_twin_document_get_publish_topic and az_iot_hub_client_twin_patch_get_publish_topic APIs to obtain the correct publish topics. This is described in the diagram here: https://github.com/Azure/azure-sdk-for-c/blob/main/sdk/docs/iot/mqtt_state_machine.md#iot-hub We do not recommend hardcoding any of the topics when using the SDK (they are subject to change in future versions).

Also, please keep in mind that Twin is not available on all IoT Hub SKUs: https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-scaling?branch=release-iotbasic (since you mentioned C2D works, I assume this is not a Basic hub).

Based on the logs above, the topics seem to be correct and Hub should have responded to the request. Since the logs though do not show MQTT acknowledgements, we don't know if the connection was terminated or if the subscriptions were successful.

To further investigate this issue, we need service-side logs. Please follow https://learn.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request to open a ticket.

[meta-space]

@engemil You are correct. I removed the subscription call and the twin document still arrived.

I think I had accidentally messed up the state in the state machine which resulted in the response not being processed previously

[engemil]

@CIPop The IoT Hub (free tier) seem to be set up correctly. I managed to run the paho_iot_hub_twin_sample.c from Windows, and I got the following result (Iot hub name sensured with XXXXXXXXX):

AZ_IOT_HUB_HOSTNAME = XXXXXXXXX.azure-devices.net
AZ_IOT_HUB_DEVICE_ID = paho-sample-device1
AZ_IOT_DEVICE_X509_CERT_PEM_FILE_PATH = C:\azure-sdk-for-c\sdk\samples\iot\device_cert_store.pem
AZ_IOT_DEVICE_X509_TRUST_PEM_FILE_PATH = C:\azure-sdk-for-c\sdk\samples\iot\BaltimoreCyberTrustRoot.crt.pem

SUCCESS:        MQTT endpoint created at "ssl://XXXXXXXXX.azure-devices.net:8883".
SUCCESS:        Client created and configured.
                MQTT client username: XXXXXXXXX.azure-devices.net/paho-sample-device1/?api-version=2020-09-30&DeviceClientType=azsdk-c%2F1.6.0-beta.1

SUCCESS:        Client connected to IoT Hub.
SUCCESS:        Client subscribed to IoT Hub topics.
                Client requesting device twin document from service.

                Waiting for device twin message.

SUCCESS:        Client received a device twin message from the service.
SUCCESS:        Client received a valid topic response.
                Topic: $iothub/twin/res/200/?$rid=get_twin
                Payload: {"desired":{"$version":1},"reported":{"$version":1}}
                Status: 200
SUCCESS:        Client parsed device twin message.
                Message Type: GET

                Client sending reported property to service.
SUCCESS:        Client published the Twin Patch reported property message.
                Payload: {"device_count":0}

                Waiting for device twin message.

SUCCESS:        Client received a device twin message from the service.
SUCCESS:        Client received a valid topic response.
                Topic: $iothub/twin/res/204/?$rid=reported_prop&$version=2
                Payload:
                Status: 204
SUCCESS:        Client parsed device twin message.
                Message Type: Reported Properties

                Waiting for device twin message.

                Receive message timeout expired.
SUCCESS:        Client disconnected from IoT Hub.

C:\azure-sdk-for-c\build\sdk\samples\iot\Debug\paho_iot_hub_twin_sample.exe (process 21612) exited with code 0.
To automatically close the console when debugging stops, enable Tools->Options->Debugging->Automatically close the console when debugging stops.
Press any key to close this window . . .

I tried to open a ticket with the instructions given in the link you gave, however, the "New support request"-interface sent me to a Diagnostics summary, summarizing that nothing seems to be wrong and recommended steps given where "no action should be needed".

I suspect the issue is in the arduino ported version of Azure SDK for C (https://github.com/Azure/azure-sdk-for-c-arduino), or in my own code. I'm available for further troubleshooting if you have more ideas for what should be tested.

[CIPop]

Thank you @engemil for the test and logs!

We'll try to repro on our side with the Arduino port.