Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug Report] Thread leak of IotHubReconnectTask #1563

Closed
janusz-com2m opened this issue Jun 28, 2022 · 2 comments
Closed

[Bug Report] Thread leak of IotHubReconnectTask #1563

janusz-com2m opened this issue Jun 28, 2022 · 2 comments
Assignees
@brycewang-microsoft
Labels
bug fix checked in Fix checked into main or preview, but not yet released. IoTSDK

Comments

@janusz-com2m
Copy link

Context

  • OS and version used: Docker Image eclipse-temurin:11-alpine (Linux f74e9f6057d9 4.15.0-184-generic Minor changes to enable compilation under JDK 9. Mainly bringing the … #194-Ubuntu SMP Thu Jun 2 18:54:48 UTC 2022 x86_64 Linux)
  • Java runtime used: openjdk version "11.0.15" 2022-04-19
    OpenJDK Runtime Environment Temurin-11.0.15+10 (build 11.0.15+10)
    OpenJDK 64-Bit Server VM Temurin-11.0.15+10 (build 11.0.15+10, mixed mode)
  • SDK version used:
  • iot-device-client: 2.0.3
  • provisioning-device-client: 2.0.0

Description of the issue

We are using the sdk to connect multiple devices within an IoT-Edge module to IoT Central. The DPS is used for the device provisioning. We are seeing an increasing threadcount over the time in the Java Process, until the underlying VM is not able to spawn any more threads.
I have retrieved a threaddump from the process after it was running for ~20 hours and I could find 18 Instances of the IotHubReconnectTask for one device in the threaddump, this behaviour occurs for other devices to. So it looks like that the IotHubReconnectTask-Threads aren't cleaned up correctly.

Code sample exhibiting the issue

@service
public class AzureIotEdgeService {

private static final String TELEMETRY_COMPONENT_NAME = "$.sub";
private static final String ENCODING_UTF_8 = StandardCharsets.UTF_8.name();
private static final String CONTENT_APPLICATION_JSON = "application/json";

private final Logger logger;
private final AzureDeviceProvisioningService azureDeviceProvisioningService;

private Map<String, DeviceClient> deviceClients = new ConcurrentHashMap<>();

public AzureIotEdgeService(
		Logger logger,
		AzureDeviceProvisioningService azureDeviceProvisioningService
) {
	this.logger = logger;
	this.azureDeviceProvisioningService = azureDeviceProvisioningService;
}

@Async
public void sendMessageToIotHubForDevice(String topic, String payload, Date timestamp) {
	String componentId = topic.split("/")[3].replace("@", "_");
	String subComponent = topic.split("/")[topic.split("/").length - 1];
	Message message = createIotHubMessageUtf8(payload, subComponent, timestamp);

	DeviceClient deviceClient;
	if (deviceClients.containsKey(componentId)) {
		deviceClient = deviceClients.get(componentId);
	} else {
		deviceClient = azureDeviceProvisioningService.provisionDevice(componentId);
		if (deviceClient == null) {
			// There should be an error logged already in the AzureDeviceProvisioningService
			return;
		}
		deviceClients.put(componentId, deviceClient);
	}

	deviceClient.sendEventAsync(message, (msg, error, callbackContext) -> {
		if (error != null) {
			logger.error("Error during sending message {} to IoT Hub for device {}: {}", msg.getMessageId(), componentId, error.getMessage(), error);
		} else {
			logger.info("Telemetry - Response from IoT Hub for message Id={}", msg.getMessageId());
		}
	}, message);
	logger.info("After send async");

}

private Message createIotHubMessageUtf8(String json, String componentName, Date creationTime) {
	Message message = new Message(json);
	message.setContentEncoding(ENCODING_UTF_8);
	message.setContentType(CONTENT_APPLICATION_JSON);
	message.setConnectionDeviceId(componentName);
	if (creationTime != null) {
		message.setCreationTimeUTC(creationTime);
	}

	if (componentName != null) {
		message.setProperty(TELEMETRY_COMPONENT_NAME, componentName);
	}

	return message;
}

}

@service
public class AzureDeviceProvisioningService {

// Plug and play features are available over MQTT, MQTT_WS, AMQPS, and AMQPS_WS.
private static final ProvisioningDeviceClientTransportProtocol provisioningProtocol = ProvisioningDeviceClientTransportProtocol.MQTT;

// Environmental variables for Dps
private final String scopeId;
private final String globalEndpoint;
private final String enrollmentGroupSymmetricKey;
private final int maxTimeToWaitForRegistration;

private final Logger logger;

AzureDeviceProvisioningService(@Value("${IOTHUB_DEVICE_DPS_ID_SCOPE}") String scopeId,
							   @Value("${IOTHUB_DEVICE_DPS_ENDPOINT}") String globalEndpoint,
							   @Value("${IOTHUB_DEVICE_DPS_DEVICE_KEY}") String enrollmentGroupSymmetricKey,
							   @Value("${MAX_TIME_TO_WAIT_FOR_REGISTRATION}") int maxTimeToWaitForRegistration,
							   Logger logger) {
	this.scopeId = scopeId;
	this.globalEndpoint = globalEndpoint;
	this.enrollmentGroupSymmetricKey = enrollmentGroupSymmetricKey;
	this.maxTimeToWaitForRegistration = maxTimeToWaitForRegistration;
	this.logger = logger;
}

static class ProvisioningStatus
{
	ProvisioningDeviceClientRegistrationResult provisioningDeviceClientRegistrationInfoClient = new ProvisioningDeviceClientRegistrationResult();
	Exception exception;
}

public DeviceClient provisionDevice(String registrationId) {
	DeviceClient deviceClient;
	try {
		deviceClient = initializeAndProvisionDevice(registrationId);
	} catch (Exception e) {
		logger.error("Error during provision for device {}: {}", registrationId, e.getMessage(), e);
		return null;
	}
	return deviceClient;
}

private DeviceClient initializeAndProvisionDevice(String registrationId) throws ProvisioningDeviceClientException, IOException, URISyntaxException, InterruptedException, NoSuchAlgorithmException, InvalidKeyException {
	byte[] deviceSymmetricKey = generateSymmetricKey(enrollmentGroupSymmetricKey, registrationId);
	SecurityProviderSymmetricKey securityClientSymmetricKey = new SecurityProviderSymmetricKey(deviceSymmetricKey, registrationId);

	ProvisioningStatus provisioningStatus = new ProvisioningStatus();
	ProvisioningDeviceClient provisioningDeviceClient = ProvisioningDeviceClient.create(globalEndpoint, scopeId, provisioningProtocol, securityClientSymmetricKey);
	AdditionalData additionalData = new AdditionalData();

	provisioningDeviceClient.registerDevice((provisioningDeviceClientRegistrationResult, e, context) -> {
		if (context instanceof ProvisioningStatus)
		{
			ProvisioningStatus status = (ProvisioningStatus) context;
			status.provisioningDeviceClientRegistrationInfoClient = provisioningDeviceClientRegistrationResult;
			status.exception = e;
			logger.error(e.getMessage(), e);
		}
		else
		{
			logger.error("Received unknown context");
		}
	}, provisioningStatus, additionalData);

	while (provisioningStatus.provisioningDeviceClientRegistrationInfoClient.getProvisioningDeviceClientStatus() != ProvisioningDeviceClientStatus.PROVISIONING_DEVICE_STATUS_ASSIGNED)
	{
		if (provisioningStatus.provisioningDeviceClientRegistrationInfoClient.getProvisioningDeviceClientStatus() == ProvisioningDeviceClientStatus.PROVISIONING_DEVICE_STATUS_ERROR ||
				provisioningStatus.provisioningDeviceClientRegistrationInfoClient.getProvisioningDeviceClientStatus() == ProvisioningDeviceClientStatus.PROVISIONING_DEVICE_STATUS_DISABLED ||
				provisioningStatus.provisioningDeviceClientRegistrationInfoClient.getProvisioningDeviceClientStatus() == ProvisioningDeviceClientStatus.PROVISIONING_DEVICE_STATUS_FAILED)
		{
			logger.error("Registration error, bailing out", provisioningStatus.exception);
			break;
		}
		logger.info("Waiting for Provisioning Service to register");
		Thread.sleep(maxTimeToWaitForRegistration);
	}

	if (provisioningStatus.provisioningDeviceClientRegistrationInfoClient.getProvisioningDeviceClientStatus() == ProvisioningDeviceClientStatus.PROVISIONING_DEVICE_STATUS_ASSIGNED) {
		logger.info("IotHUb Uri: {}", provisioningStatus.provisioningDeviceClientRegistrationInfoClient.getIothubUri());
		logger.info("Device ID: {}", provisioningStatus.provisioningDeviceClientRegistrationInfoClient.getDeviceId());

		String iotHubUri = provisioningStatus.provisioningDeviceClientRegistrationInfoClient.getIothubUri();
		String deviceId = provisioningStatus.provisioningDeviceClientRegistrationInfoClient.getDeviceId();

		logger.debug("Opening the device client.");
		DeviceClient deviceClient = new DeviceClient(iotHubUri, deviceId, securityClientSymmetricKey, IotHubClientProtocol.MQTT);
		try {
			deviceClient.open(false);
		} catch (IotHubClientException e) {
			throw new ProvisioningDeviceClientException(e);
		}
		return deviceClient;
	} else {
		throw new ProvisioningDeviceClientException(String.format("Provisioning time exceeded for device %s", registrationId));
	}
}

private byte[] generateSymmetricKey(String enrollmentGroupSymmetricKey, String registrationId) throws NoSuchAlgorithmException, InvalidKeyException {
	return SecurityProviderSymmetricKey
			.ComputeDerivedSymmetricKey(
							enrollmentGroupSymmetricKey.getBytes(StandardCharsets.UTF_8),
							registrationId);
}

}

Console log of the issue

I extracted all threads from threaddump for one device and removed to references to the customer in this snippet. The full dump can be provided on request.
"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxn48c4b792-1539-4f77-93eb-288848b72fb6-azure-iot-sdk-IotHubReconnectTask" #4624 prio=5 os_prio=0 cpu=168.43ms elapsed=67207.03s tid=0x00007f1b8ad50800 nid=0x12a6 waiting on condition [0x00007f1b7b5ce000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxn6ca16f68-8829-4ed2-b906-352a46403006-azure-iot-sdk-IotHubReconnectTask" #6132 prio=5 os_prio=0 cpu=75.89ms elapsed=63307.14s tid=0x00007f1b8b199800 nid=0x1889 waiting on condition [0x00007f1ac5a73000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxncff75684-baad-4de1-95d9-a928975b7e9a-azure-iot-sdk-IotHubReconnectTask" #7903 prio=5 os_prio=0 cpu=56.13ms elapsed=59404.88s tid=0x00007f1b8b19b000 nid=0x1f72 waiting on condition [0x00007f1ad90be000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxnb0058747-96e2-4a19-919b-5e6bdf49c41b-azure-iot-sdk-IotHubReconnectTask" #9431 prio=5 os_prio=0 cpu=23.64ms elapsed=55500.32s tid=0x00007f1b8b633800 nid=0x2581 waiting on condition [0x00007f1b7aa3e000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxn25de35cc-3e1e-4e25-b55f-78337b327154-azure-iot-sdk-IotHubReconnectTask" #10937 prio=5 os_prio=0 cpu=129.48ms elapsed=51598.67s tid=0x00007f1b8b1a2000 nid=0x2b60 waiting on condition [0x00007f1b7fc46000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxnd7d869b7-e2d2-4ce0-9cd8-d2e1ceefb8d0-azure-iot-sdk-IotHubReconnectTask" #12460 prio=5 os_prio=0 cpu=123.15ms elapsed=47696.86s tid=0x00007f1b8bbfe000 nid=0x3159 waiting on condition [0x00007f1b80bc8000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxn85942809-f38a-424b-8fd3-7fb64618e05e-azure-iot-sdk-IotHubReconnectTask" #14003 prio=5 os_prio=0 cpu=42.24ms elapsed=43796.60s tid=0x00007f1b8b195000 nid=0x3760 waiting on condition [0x00007f1b8075e000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxn83ce62dc-aa98-4898-8584-a865e7ffd839-azure-iot-sdk-IotHubReconnectTask" #15554 prio=5 os_prio=0 cpu=307.74ms elapsed=39895.64s tid=0x00007f1b8b643000 nid=0x3d6f waiting on condition [0x00007f1b80fc4000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxnb0058747-96e2-4a19-919b-5e6bdf49c41b-azure-iot-sdk-IotHubReconnectTask" #17081 prio=5 os_prio=0 cpu=0.62ms elapsed=35995.12s tid=0x00007f1b3275b800 nid=0x4366 waiting on condition [0x00007f1a6c997000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxn48c4b792-1539-4f77-93eb-288848b72fb6-azure-iot-sdk-IotHubReconnectTask" #18625 prio=5 os_prio=0 cpu=4.90ms elapsed=32094.36s tid=0x00007f1b33d0d000 nid=0x496e waiting on condition [0x00007f1a5eb82000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxn6ca16f68-8829-4ed2-b906-352a46403006-azure-iot-sdk-IotHubReconnectTask" #20161 prio=5 os_prio=0 cpu=0.23ms elapsed=28192.82s tid=0x00007f1b81610000 nid=0x4f6e waiting on condition [0x00007f1a52193000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxn25de35cc-3e1e-4e25-b55f-78337b327154-azure-iot-sdk-IotHubReconnectTask" #21697 prio=5 os_prio=0 cpu=2.32ms elapsed=24291.61s tid=0x00007f1b81b15000 nid=0x556e waiting on condition [0x00007f1a45bb2000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxnd7d869b7-e2d2-4ce0-9cd8-d2e1ceefb8d0-azure-iot-sdk-IotHubReconnectTask" #23233 prio=5 os_prio=0 cpu=0.39ms elapsed=20390.32s tid=0x00007f1b83231000 nid=0x5b6e waiting on condition [0x00007f1a393cf000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxn93103c2d-df0f-4702-930b-8dd91362a617-azure-iot-sdk-IotHubReconnectTask" #25202 prio=5 os_prio=0 cpu=0.91ms elapsed=16489.13s tid=0x00007f1b84908800 nid=0x631f waiting on condition [0x00007f1ad07a0000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxn85942809-f38a-424b-8fd3-7fb64618e05e-azure-iot-sdk-IotHubReconnectTask" #26607 prio=5 os_prio=0 cpu=3.10ms elapsed=12586.36s tid=0x00007f1b84912000 nid=0x689c waiting on condition [0x00007f1aca9af000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxncff75684-baad-4de1-95d9-a928975b7e9a-azure-iot-sdk-IotHubReconnectTask" #28009 prio=5 os_prio=0 cpu=0.49ms elapsed=8683.07s tid=0x00007f1b82d88800 nid=0x6e15 waiting on condition [0x00007f1a1e4af000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxn83ce62dc-aa98-4898-8584-a865e7ffd839-azure-iot-sdk-IotHubReconnectTask" #29402 prio=5 os_prio=0 cpu=0.68ms elapsed=4781.12s tid=0x00007f1b81164800 nid=0x7387 waiting on condition [0x00007f1a1316d000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)
"MQTT Rec: DEVICE_ID" #30780 prio=5 os_prio=0 cpu=1619.11ms elapsed=879.68s tid=0x00007f1b81163800 nid=0x78e9 runnable [0x00007f1ad3ea5000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(java.base@11.0.15/Native Method)
at java.net.SocketInputStream.socketRead(java.base@11.0.15/SocketInputStream.java:115)
at java.net.SocketInputStream.read(java.base@11.0.15/SocketInputStream.java:168)
at java.net.SocketInputStream.read(java.base@11.0.15/SocketInputStream.java:140)
at sun.security.ssl.SSLSocketInputRecord.read(java.base@11.0.15/SSLSocketInputRecord.java:478)
at sun.security.ssl.SSLSocketInputRecord.readHeader(java.base@11.0.15/SSLSocketInputRecord.java:472)
at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(java.base@11.0.15/SSLSocketInputRecord.java:70)
at sun.security.ssl.SSLSocketImpl.readApplicationRecord(java.base@11.0.15/SSLSocketImpl.java:1454)
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(java.base@11.0.15/SSLSocketImpl.java:1065)
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(java.base@11.0.15/SSLSocketImpl.java:972)
at java.io.DataInputStream.readByte(java.base@11.0.15/DataInputStream.java:270)
at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:92)
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:137)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"MQTT Snd: DEVICE_ID" #30783 prio=5 os_prio=0 cpu=128.60ms elapsed=879.54s tid=0x00007f1b7ef62800 nid=0x78eb in Object.wait() [0x00007f1ac4b64000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.15/Native Method)
- waiting on
at java.lang.Object.wait(java.base@11.0.15/Object.java:328)
at org.eclipse.paho.client.mqttv3.internal.ClientState.get(ClientState.java:825)
- waiting to re-lock in wait() <0x00000000f184fdb8> (a java.lang.Object)
at org.eclipse.paho.client.mqttv3.internal.CommsSender.run(CommsSender.java:129)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"MQTT Call: DEVICE_ID" #30785 prio=5 os_prio=0 cpu=23.24ms elapsed=879.51s tid=0x00007f1b7ef72800 nid=0x78ec in Object.wait() [0x00007f1ab7f24000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.15/Native Method)
- waiting on
at java.lang.Object.wait(java.base@11.0.15/Object.java:328)
at org.eclipse.paho.client.mqttv3.internal.CommsCallback.run(CommsCallback.java:181)
- waiting to re-lock in wait() <0x00000000f184c770> (a java.lang.Object)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"MQTT Ping: DEVICE_ID" #30787 prio=5 os_prio=0 cpu=20060.37ms elapsed=879.35s tid=0x00007f1b828d2000 nid=0x78f0 in Object.wait() [0x00007f1b7d21c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.15/Native Method)
- waiting on
at java.util.TimerThread.mainLoop(java.base@11.0.15/Timer.java:553)
- waiting to re-lock in wait() <0x00000000f184c7f0> (a java.util.TaskQueue)
at java.util.TimerThread.run(java.base@11.0.15/Timer.java:506)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxne86b35d5-5258-4756-9808-51a92f7ee86f-azure-iot-sdk-IotHubSendTask" #30788 prio=5 os_prio=0 cpu=164.29ms elapsed=879.18s tid=0x00007f1b4c27d000 nid=0x78f2 waiting on condition [0x00007f1a1e3ae000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f638> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubSendTask.run(IotHubSendTask.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxne86b35d5-5258-4756-9808-51a92f7ee86f-azure-iot-sdk-IotHubReceiveTask" #30790 prio=5 os_prio=0 cpu=0.32ms elapsed=879.18s tid=0x00007f1b73f18000 nid=0x78f3 waiting on condition [0x00007f1a13470000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f680> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReceiveTask.run(IotHubReceiveTask.java:62)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"iotc-IOTHUB_ID.azure-devices.net-DEVICE_ID-af5e22bd-Cxne86b35d5-5258-4756-9808-51a92f7ee86f-azure-iot-sdk-IotHubReconnectTask" #30792 prio=5 os_prio=0 cpu=0.32ms elapsed=879.14s tid=0x00007f1b46530800 nid=0x78f4 waiting on condition [0x00007f1a1326e000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
- parking to wait for <0x00000000e343f6c8> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.15/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.15/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.Semaphore.acquire(java.base@11.0.15/Semaphore.java:318)
at com.microsoft.azure.sdk.iot.device.transport.IotHubReconnectTask.run(IotHubReconnectTask.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.15/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.15/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.15/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)
@brycewang-microsoft brycewang-microsoft self-assigned this Jul 1, 2022
@timtay-microsoft timtay-microsoft changed the title [Bug Report] Possible threadleak with IotHubReconnectTask [Bug Report] Thread leak of IotHubReconnectTask Jul 5, 2022
@brycewang-microsoft brycewang-microsoft mentioned this issue Jul 6, 2022
Merged
@brycewang-microsoft
Copy link
Collaborator

brycewang-microsoft commented Jul 6, 2022 •
edited
Loading

Hi @janusz-com2m, thanks for bringing this to our attention! #1567 should fix this issue. I will update here again once we release a newer version.

@brycewang-microsoft brycewang-microsoft added the fix checked in Fix checked into main or preview, but not yet released. label Jul 6, 2022
@brycewang-microsoft
Copy link
Collaborator

Device client package 2.0.4 has been released with the fix for this issue. Closing this thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brycewang-microsoft brycewang-microsoft

Labels
bug fix checked in Fix checked into main or preview, but not yet released. IoTSDK
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@brycewang-microsoft @janusz-com2m