AMQP OpenAsync doesn't timeout after 1 min when network is offline

[ancaantochi]

• OS, version, SKU and CPU architecture used: Ubuntu 18.04 x86
• Application's .NET Target Framework : 2.1.13
• Device: Azur VM
• SDK version used: 1.21.3

Description of the issue:

Create ModuleClient with AMQP transport settings and call OpenAsync when the network is offline. Would expect the call to timeout in 1 minute which I can see in the logs is used for timeout, but it times out after 2 minutes, sometimes longer.

Code sample exhibiting the issue:

Console log of the issue:

[davilu]

It should be the same cause as RecevieAsync. While OpenAsync get called, there is a CancellationToken created with 1 minute expiry. But since AMQP doesn't accept CancellationToken, first attempt was timed out after 1 minute. As we found is CancellationToken doesn't throw after expired for several milliseconds, RetryDelegatingHandler will start another attempt which makes it to be 2 minutes. I believe it shouldn't happen if OperationTimeout is set to 59 seconds. It could be fixed from DeviceClient pipeline to use CancellationToken only when there is a CancellationToken passed to the DeviceClient function call and use TimeoutHelper for all other cases. But it will be a big change.

[drwill-ms]

I can repro this issue, and even saw a case where it took over 4 minutes to time out.

I worry about fixing the behavior only because we've had a fair number of complaints for internal and external customers anytime the default behavior changes, even when it isn't an API surface change.

However, I may have a reliable solution for you. I did find that if I included a cancellation token to the call to OpenAsync, from a CancellationTokenSource built with a custom timeout, then it looks to be respecting that. Here's my sample code:

var deviceClient = DeviceClient.CreateFromConnectionString(deviceConnectionString);

var sw = new Stopwatch();
sw.Start();

using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(15)); // also tried 30 and 60 seconds
try
{
    await deviceClient.OpenAsync(cts.Token).ConfigureAwait(false); //cts.Token
}
catch (Exception ex)
{
    Console.WriteLine(ex.Message); // no host found
}

sw.Stop();
Console.WriteLine(sw.Elapsed); // just slightly over the desired timeout

Will this work for you?

@davilu is right, that a big part of the problem is in the AMQP library we use - not to say this code couldn't have been written to actually work. Still, with what seems to be a viable workaround to me, perhaps we are best to let the existing behavior remain.

[az-iot-builder-01]

@ancaantochi, @davilu, thank you for your contribution to our open-sourced project! Please help us improve by filling out this 2-minute customer satisfaction survey