NullReferenceException thrown from AmazonS3Client.Begin/EndGetObject if no internet connection

[callorico]

Using 3.1.3.5 with .NET 4.0 Client Profile

The following works fine if I have an active internet connection:

using (IAmazonS3 s3Client = new AmazonS3Client())
{
    // Setup request for putting an object in S3.                 
    GetObjectRequest request = new GetObjectRequest
    {
        BucketName = "<bucket>",
        Key = "<key>",
    };

    IAsyncResult result = s3Client.BeginGetObject(request, null, null);

    GetObjectResponse response = s3Client.EndGetObject(result);
    response.WriteResponseStreamToFile("test.out");
}

However, if I disable my network adapter prior to running the above, a NullReferenceException is thrown out of the SDK with the following stack trace:

   at Amazon.Runtime.Internal.RedirectHandler.HandleRedirect(IExecutionContext executionContext) in d:\Jenkins\jobs\v3-stage-release\workspace\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\RedirectHandler.cs:line 101
   at Amazon.Runtime.Internal.RedirectHandler.InvokeAsyncCallback(IAsyncExecutionContext executionContext) in d:\Jenkins\jobs\v3-stage-release\workspace\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\RedirectHandler.cs:line 75
   at Amazon.Runtime.Internal.PipelineHandler.AsyncCallback(IAsyncExecutionContext executionContext) in d:\Jenkins\jobs\v3-stage-release\workspace\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\PipelineHandler.cs:line 95
   at Amazon.Runtime.AmazonServiceClient.EndInvoke[TResponse](IAsyncResult result) in d:\Jenkins\jobs\v3-stage-release\workspace\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\AmazonServiceClient.cs:line 212
   at Amazon.S3.AmazonS3Client.EndGetObject(IAsyncResult asyncResult) in d:\Jenkins\jobs\v3-stage-release\workspace\AWSDotNetPublic\sdk\src\Services\S3\Generated\_bcl35\AmazonS3Client.cs:line 1956
   at S3Test.Program.Main(String[] args) in c:\Users\callorico\Desktop\S3Test\S3Test\Program.cs:line 28

In this scenario, I would expect to see a network exception of some type. If I replace the APM code with a synchronous s3Client.GetObject call then I do see an AmazonServiceException with a nested WebException.

[jimfl]

This is an obvious bug in the request handling logic in the 3.5 async client. Thanks for pointing it out.

[gokarnm]

The fix for this issue is available in the latest version of the AWS S3 SDK (version 3.1.3.7). The APM API should now throw the same exception as the synchronous version.

[callorico]

OK, I tried testing with 3.1.3.8 and this doesn't seem to be working correctly:

I tried running the following with no Internet connection:

class Program
{
    static IAmazonS3 s3Client;

    static void GetObjectCallback(IAsyncResult result)
    {
        Console.WriteLine("{0} GetObjectCallback was triggered", DateTime.UtcNow);
        GetObjectResponse response = s3Client.EndGetObject(result);
        Console.WriteLine("{0} EndGetObject returned", DateTime.UtcNow);
    }

    static void Main(string[] args)
    {
        s3Client = new AmazonS3Client();

        GetObjectRequest request = new GetObjectRequest
        {
            BucketName = "<bucket-name>",
            Key = "<key-name>",
        };

        Console.WriteLine("{0} About to call BeginGetObject", DateTime.UtcNow);
        try
        {
            s3Client.BeginGetObject(request, new AsyncCallback(GetObjectCallback), null);
        }
        catch (Exception e)
        {
            Console.WriteLine("{0} BeginGetObject threw {1}", DateTime.UtcNow, e);
        }
        finally
        {
            Console.WriteLine("{0} BeginGetObject Elapsed time", DateTime.UtcNow);
        }

        Console.WriteLine("Hit return to exit");
        Console.ReadLine();
    }
}

Here is the output from my machine:

PS C:\Users\Callorico\Desktop\S3Test\S3Test\bin\Debug> .\S3Test.exe
11/24/2015 5:36:33 AM About to call BeginGetObject
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM BeginGetObject threw System.NullReferenceException: Object reference not set to an instance of an
object.
   at Amazon.Runtime.Internal.PipelineHandler.AsyncCallback(IAsyncExecutionContext executionContext) in d:\Jenkins\jobs\
v3-stage-release\workspace\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\PipelineHandler.cs:line 111
   at Amazon.Runtime.Internal.HttpHandler`1.GetResponseCallback(IAsyncResult result) in d:\Jenkins\jobs\v3-stage-release
\workspace\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\HttpHandler.cs:line 302
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.ContextAwareResult.CaptureOrComplete(ExecutionContext& cachedContext, Boolean returnContext)
   at System.Net.HttpWebRequest.BeginGetResponse(AsyncCallback callback, Object state)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync(IAsyncExecutionContext executionContext) in d:\Jenkins\jobs\v3-s
tage-release\workspace\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\HttpHandler.cs:line 253
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync(IAsyncExecutionContext executionContext) in d:\Jenkins\jobs\v3-
stage-release\workspace\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\MetricsHandler.cs:line 87
   at Amazon.Runtime.AmazonServiceClient.BeginInvoke[TRequest](TRequest request, IMarshaller`2 marshaller, ResponseUnmar
shaller unmarshaller, AsyncCallback callback, Object state) in d:\Jenkins\jobs\v3-stage-release\workspace\AWSDotNetPubli
c\sdk\src\Core\Amazon.Runtime\AmazonServiceClient.cs:line 169
   at Amazon.S3.AmazonS3Client.BeginGetObject(GetObjectRequest request, AsyncCallback callback, Object state) in d:\Jenk
ins\jobs\v3-stage-release\workspace\AWSDotNetPublic\sdk\src\Services\S3\Generated\_bcl35\AmazonS3Client.cs:line 1943
   at S3Test.Program.Main(String[] args) in c:\Users\Ryan\Desktop\S3Test\S3Test\Program.cs:line 35
11/24/2015 5:37:07 AM BeginGetObject Elapsed time
Hit return to exit

1. First, it looks like there is still a NullReferenceException being thrown in this scenario.
2. The thread that calls BeginGetObject is blocked for ~30 seconds (presumably while going through the retry loop). In the APM model, I believe the Begin* methods should return immediately.
3. Why is the GetObjectCallback method called so many times? I would have expected it to just be called once.

Is there anyway to use the *Async methods with .NET 4.0 instead?

[gokarnm]

Hi,

1 & 3. You need to have a try catch block around the EndGetObject call. In the current code, the EndGetObject throwns an exception as expected, which is not handled by GetObjectCallback method, causing the callback method to be called again. The callback should not be called multiple time by the SDK, this is a bug and we are working on a fix for this. If you catch the appropriate exception in the callback, it won't be called again.

try
{
    GetObjectResponse response = s3Client.EndGetObject(result);
}
catch (AmazonServiceException exception)
{
    Console.WriteLine("{0} EndGetObject threw {1}", DateTime.UtcNow, exception);
    Console.WriteLine("{0} EndGetObject threw inner exception {1}", DateTime.UtcNow, exception.InnerException);
}

2 . You are correct, the BeginGetObject should not block for the retries, I'll let you know when a fix is available for this. The work around for this may be to reduce the number of retries, the default value for max retries is 4.

s3Client = new AmazonS3Client(new AmazonS3Config
    {
        MaxErrorRetry = 2
    });

You cannot use the *Async methods on .NET 4.0, they are only available .NET 4.5 onward.

[gokarnm]

A fix for this issue is available in version 3.1.3.10 of AWS S3 SDK . This fix catches unhandled exceptions being thrown from the user code in the async callback, and does not block the BeginXyz call under certain conditions when retries are performed.