IoTEdge module is running in the Jetson TX2 device but not reporting back in the Azure Portal

[manojrampur]

Hi Team,

I'm building the Kirkstone Yocto Image for Jetson TX2 Device. I have cloned the meta iotedge layer along with it's dependecy layers

meta-iotedge Commit ID: 6518416

While testing the image in JetsonT TX2 Device, I could see the EdgeAgent module is running in the Jetson TX2 device but it's not reporting back and showing the EdgeAgent module as running in Azure Portal.

Please find the logs below:

iotedge system status

System services:
    aziot-edged             Running
    aziot-identityd         Running
    aziot-keyd              Running
    aziot-certd             Running
    aziot-tpmd              Ready

iotedge check --verbose

Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
× aziot-identity-service package is up-to-date - Error
    could not query https://aka.ms/latest-aziot-identity-service for latest available version
        caused by: could not query https://aka.ms/latest-aziot-identity-service for latest available version
        caused by: error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
        caused by: dns error: failed to lookup address information: Temporary failure in name resolution
        caused by: failed to lookup address information: Temporary failure in name resolution
‼ host time is close to reference time - Warning
    Could not query NTP server
        caused by: Could not query NTP server
        caused by: could not resolve NTP pool hostname: failed to lookup address information: Temporary failure in name resolution
        caused by: failed to lookup address information: Temporary failure in name resolution
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
√ read all preloaded certificates from the Certificates Service - OK
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
× host can connect to and perform TLS handshake with iothub AMQP port - Error
    Could not connect to deviceupdatehub.azure-devices.net : could not complete TLS handshake
        caused by: Could not connect to deviceupdatehub.azure-devices.net : could not complete TLS handshake
        caused by: dns error: failed to lookup address information: Temporary failure in name resolution
× host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Error
    Could not connect to deviceupdatehub.azure-devices.net : could not complete TLS handshake
        caused by: Could not connect to deviceupdatehub.azure-devices.net : could not complete TLS handshake
        caused by: dns error: failed to lookup address information: Temporary failure in name resolution
× host can connect to and perform TLS handshake with iothub MQTT port - Error
    Could not connect to deviceupdatehub.azure-devices.net : could not complete TLS handshake
        caused by: Could not connect to deviceupdatehub.azure-devices.net : could not complete TLS handshake
        caused by: dns error: failed to lookup address information: Temporary failure in name resolution

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
√ configuration has correct URIs for daemon mgmt endpoint - OK
× aziot-edge package is up-to-date - Error
    Error while fetching latest versions of edge components: could not send HTTP request
        caused by: Error while fetching latest versions of edge components: could not send HTTP request
        caused by: error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
        caused by: dns error: failed to lookup address information: Temporary failure in name resolution
        caused by: failed to lookup address information: Temporary failure in name resolution
√ container time is close to host time - OK
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
        caused by: Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
                   Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
                   You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.
        caused by: Container engine is not configured to rotate module logs which may cause it run out of disk space.
                   Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
                   You can ignore this warning if you are setting log policy per module in the Edge deployment.
‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
    The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
        caused by: The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.
                   Data might be lost if the module is deleted or updated.
                   Please see https://aka.ms/iotedge-storage-host for best practices.
× production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error
    Could not check current state of edgeHub container
        caused by: Could not check current state of edgeHub container
        caused by: docker returned exit status: 1, stderr = Error: No such object: edgeHub
× Agent image is valid and can be pulled from upstream - Error
    Failed to get edge Agent image
        caused by: Failed to get edge Agent image
        caused by: docker returned exit status: 1, stderr = Error response from daemon: Get "https://mcr.microsoft.com/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
× container on the default network can connect to upstream AMQP port - Error
    Container on the default network could not connect to deviceupdatehub.azure-devices.net:5671
        caused by: Container on the default network could not connect to deviceupdatehub.azure-devices.net:5671
        caused by: docker returned exit status: 1, stderr = One or more errors occurred. (Resource temporarily unavailable)
× container on the default network can connect to upstream HTTPS / WebSockets port - Error
    Container on the default network could not connect to deviceupdatehub.azure-devices.net:443
        caused by: Container on the default network could not connect to deviceupdatehub.azure-devices.net:443
        caused by: docker returned exit status: 1, stderr = One or more errors occurred. (Resource temporarily unavailable (deviceupdatehub.azure-devices.net:443))
√ container on the default network can connect to upstream MQTT port - OK
    skipping because of not required in this configuration
× container on the IoT Edge module network can connect to upstream AMQP port - Error
    Container on the azure-iot-edge network could not connect to deviceupdatehub.azure-devices.net:5671
        caused by: Container on the azure-iot-edge network could not connect to deviceupdatehub.azure-devices.net:5671
        caused by: docker returned exit status: 1, stderr = One or more errors occurred. (Resource temporarily unavailable)
× container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - Error
    Container on the azure-iot-edge network could not connect to deviceupdatehub.azure-devices.net:443
        caused by: Container on the azure-iot-edge network could not connect to deviceupdatehub.azure-devices.net:443
        caused by: docker returned exit status: 1, stderr = One or more errors occurred. (Resource temporarily unavailable (deviceupdatehub.azure-devices.net:443))
√ container on the IoT Edge module network can connect to upstream MQTT port - OK
    skipping because of not required in this configuration
20 check(s) succeeded.
4 check(s) raised warnings.
11 check(s) raised errors.
2 check(s) were skipped due to errors from other checks.

iotedge system logs

Mar 25 11:28:40 jetson-tx2-devkit systemd[1]: Started Azure IoT Certificates Service.
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Using existing Edge CA certificate
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Certificate aziot-edged-ca will be auto-renewed. Next renewal at 2024-06-05T07:43:14+00:00.
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Stopping all modules...
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Stopping module edgeAgent...
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - All modules stopped
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Detecting if device information has changed...
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Device information has not changed
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Starting management API...
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Starting new listener for module edgeAgent
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Starting workload API...
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Starting workload API...
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Stopping listener for module edgeAgent
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Workload API stopped
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Starting watchdog with 60 second period...
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Starting image garbage collection task...
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Watchdog checking Edge runtime status
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Edge runtime status is failed, starting module now...
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Starting module edgeAgent...
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Starting new listener for module edgeAgent
Mar 25 11:28:40 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:40Z [INFO] - Starting workload API...
Mar 25 11:28:41 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:41Z [INFO] - Started Edge runtime module edgeAgent
Mar 25 11:28:43 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:43Z [INFO] - <-- GET /trust-bundle?api-version=2020-07-07 {"accept": "application/json", "host": "workload.sock:80", "connection": "close"}
Mar 25 11:28:43 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:43Z [INFO] - --> 200 {"content-type": "application/json"}
Mar 25 11:28:43 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:43Z [INFO] - <-- GET /systeminfo?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Mar 25 11:28:43 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:43Z [INFO] - Querying system info...
Mar 25 11:28:43 jetson-tx2-devkit aziot-edged[8153]: none
Mar 25 11:28:43 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:43Z [INFO] - Successfully queried system info
Mar 25 11:28:43 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:43Z [INFO] - --> 200 {"content-type": "application/json"}
Mar 25 11:28:43 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:43Z [INFO] - <-- GET /systeminfo?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Mar 25 11:28:43 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:43Z [INFO] - Querying system info...
Mar 25 11:28:43 jetson-tx2-devkit aziot-edged[8169]: none
Mar 25 11:28:43 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:43Z [INFO] - Successfully queried system info
Mar 25 11:28:43 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:43Z [INFO] - --> 200 {"content-type": "application/json"}
Mar 25 11:28:45 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:45Z [INFO] - <-- GET /systeminfo?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Mar 25 11:28:45 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:45Z [INFO] - Querying system info...
Mar 25 11:28:45 jetson-tx2-devkit aziot-edged[8202]: none
Mar 25 11:28:45 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:45Z [INFO] - Successfully queried system info
Mar 25 11:28:45 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:45Z [INFO] - --> 200 {"content-type": "application/json"}
Mar 25 11:28:45 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:45Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Mar 25 11:28:45 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:45Z [INFO] - --> 200 {"content-type": "application/json"}
Mar 25 11:28:55 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:55Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Mar 25 11:28:55 jetson-tx2-devkit aziot-edged[7883]: 2024-03-25T11:28:55Z [INFO] - --> 200 {"content-type": "application/json"}

iotedge logs edgeAgent

System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.Azure.Devices.Edge.Util.PeriodicTask.DoWork() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/PeriodicTask.cs:line 87
<6> 2024-03-25 11:50:03.348 +00:00 [INF] - Disposing module connection object
2024-03-25 11:50:06  Starting Edge Agent
2024-03-25 11:50:06  Changing ownership of storage folder: /tmp/edgeAgent to 13622
2024-03-25 11:50:06  Changing ownership of backup folder: /tmp/edgeAgent_backup to 13622
2024-03-25 11:50:06  Changing ownership of management socket: /var/run/iotedge/mgmt.sock
2024-03-25 11:50:06  Completed necessary setup. Starting Edge Agent.
2024-03-25 11:50:06.493 +00:00 Edge Agent Main()
<6> 2024-03-25 11:50:06.929 +00:00 [INF] - Initializing Edge Agent.
<6> 2024-03-25 11:50:07.304 +00:00 [INF] - Version - 1.4.33.89334537 (71f4974213cfab7447117af13a30319ba2c40357)
<6> 2024-03-25 11:50:07.305 +00:00 [INF] -
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2024-03-25 11:50:07.311 +00:00 [INF] - ModuleUpdateMode: NonBlocking
<6> 2024-03-25 11:50:07.457 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false}
<6> 2024-03-25 11:50:07.790 +00:00 [INF] - Installing certificates [CN=aziot-edge CA Yocto-Jetson:06/23/2024 07:43:14] to Root
<6> 2024-03-25 11:50:08.184 +00:00 [INF] - Starting metrics listener on Host: *, Port: 9600, Suffix: metrics
<6> 2024-03-25 11:50:08.220 +00:00 [INF] - Updating performance metrics every 05m:00s
<6> 2024-03-25 11:50:08.233 +00:00 [INF] - Started operation Get system resources
<6> 2024-03-25 11:50:08.237 +00:00 [INF] - Collecting metadata metrics
<6> 2024-03-25 11:50:08.411 +00:00 [INF] - Set metadata metrics: 1.4.33.89334537 (71f4974213cfab7447117af13a30319ba2c40357), {"Enabled":false,"DisableCloudSubscriptions":false}, {"OperatingSystemType":"Linux","Architecture":"aarch64","Version":"1.4.27","Provisioning":{"Type":"ProvisioningType","DynamicReprovisioning":false,"AlwaysReprovisionOnStartup":false},"ServerVersion":"20.10.21-ce","KernelVersion":"#1 SMP PREEMPT Fri Dec 2 12:17:09 UTC 2022","OperatingSystem":"tegrademo","NumCpus":4,"TotalMemory":8235610112,"Virtualized":"no"}, True
<6> 2024-03-25 11:50:08.457 +00:00 [INF] - Started operation Checkpoint Availability
<6> 2024-03-25 11:50:08.464 +00:00 [INF] - Started operation refresh twin config
<6> 2024-03-25 11:50:08.507 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
<6> 2024-03-25 11:50:09.150 +00:00 [INF] - Created persistent store at /tmp/edgeAgent
<6> 2024-03-25 11:50:09.243 +00:00 [INF] - Started operation Metrics Scrape
<6> 2024-03-25 11:50:09.243 +00:00 [INF] - Started operation Metrics Upload
Scraping frequency: 01:00:00
Upload Frequency: 1.00:00:00
<6> 2024-03-25 11:50:10.158 +00:00 [INF] - Registering request handler UploadModuleLogs
<6> 2024-03-25 11:50:10.159 +00:00 [INF] - Registering request handler GetModuleLogs
<6> 2024-03-25 11:50:10.159 +00:00 [INF] - Registering request handler UploadSupportBundle
<6> 2024-03-25 11:50:10.159 +00:00 [INF] - Registering request handler RestartModule
<4> 2024-03-25 11:50:15.319 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead
<6> 2024-03-25 11:50:15.325 +00:00 [INF] - Edge agent config backup file does not exist - /tmp/edgeAgent/backup.json
<4> 2024-03-25 11:50:25.346 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead
<4> 2024-03-25 11:50:35.353 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead
<6> 2024-03-25 11:50:41.597 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_WebSocket_Only...
<4> 2024-03-25 11:50:45.358 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead
<4> 2024-03-25 11:50:55.360 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead
<4> 2024-03-25 11:51:05.361 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead
<4> 2024-03-25 11:51:15.053 +00:00 [WRN] - Error creating a device-to-cloud connection
System.AggregateException: One or more errors occurred. (Transient network error occurred, please retry.) (Transient network error occurred, please retry.)
 ---> Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Transient network error occurred, please retry.
 ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable
   at Microsoft.Azure.Amqp.ExceptionDispatcher.Throw(Exception exception)
   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.Azure.Amqp.Transport.AmqpTransportInitiator.ConnectAsyncResult.End(IAsyncResult result)
   at Microsoft.Azure.Amqp.Transport.AmqpTransportInitiator.<>c.<ConnectAsync>b__17_1(IAsyncResult r)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.InitializeAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpIoTConnector.OpenConnectionAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.EnsureConnectionAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.EnsureConnectionAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.OpenSessionAsync(DeviceIdentity deviceIdentity, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.EnsureSessionIsOpenAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.EnsureSessionIsOpenAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.OpenAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass27_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass44_0.<<OpenInternalAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.OpenInternalAsync(Boolean withRetry, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(Boolean withRetry, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider.CreateAndOpenSdkModuleClient(UpstreamProtocol upstreamProtocol, ConnectionStatusChangesHandler statusChangedHandler) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.IoTHub/ModuleClientProvider.cs:line 231
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider.<>c__DisplayClass20_1.<<CreateSdkModuleClient>b__2>d.MoveNext() in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.IoTHub/ModuleClientProvider.cs:line 189
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Edge.Util.Fallback.ExecuteAsync[T](Func`1[] options)
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Transient network error occurred, please retry.
 ---> System.Net.WebSockets.WebSocketException (0x80004005): Unable to connect to the remote server
 ---> System.Net.Http.HttpRequestException: Resource temporarily unavailable (deviceupdatehub.azure-devices.net:443)
 ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.WebSockets.WebSocketHandle.ConnectAsync(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)
   at System.Net.WebSockets.WebSocketHandle.ConnectAsync(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)
   at System.Net.WebSockets.ClientWebSocket.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.CreateClientWebSocketAsync(Uri websocketUri, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.CreateClientWebSocketTransportAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.InitializeAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpIoTConnector.OpenConnectionAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.EnsureConnectionAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.EnsureConnectionAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.OpenSessionAsync(DeviceIdentity deviceIdentity, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.EnsureSessionIsOpenAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.EnsureSessionIsOpenAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.OpenAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass27_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass44_0.<<OpenInternalAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.OpenInternalAsync(Boolean withRetry, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(Boolean withRetry, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider.CreateAndOpenSdkModuleClient(UpstreamProtocol upstreamProtocol, ConnectionStatusChangesHandler statusChangedHandler) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.IoTHub/ModuleClientProvider.cs:line 231
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider.<>c__DisplayClass20_1.<<CreateSdkModuleClient>b__3>d.MoveNext() in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.IoTHub/ModuleClientProvider.cs:line 195
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Edge.Util.Fallback.ExecuteAsync[T](Func`1[] options)<---
<6> 2024-03-25 11:51:15.118 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
<4> 2024-03-25 11:51:15.371 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead
<4> 2024-03-25 11:51:25.374 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead
<4> 2024-03-25 11:51:35.378 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead
<4> 2024-03-25 11:51:45.386 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead
<6> 2024-03-25 11:51:48.546 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_WebSocket_Only...
<4> 2024-03-25 11:51:55.391 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead
<4> 2024-03-25 11:52:05.393 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead
<4> 2024-03-25 11:52:15.398 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead

Expected behavior
The IoTEdge Device should be provisioned with the IoT Edge Runtime in Jetson TX2 Device and also it should report back into the Azure Portal and show the status as Running for IoTEdge Modules.

Please help me to resolve this issue ASAP. Thanks in Advance!!