[Bug] Storage PutFileAsync/PutBytesAsync stops transfering bytes after exactly 5 minutes, throws "Firebase.Storage.StorageException: The operation retry limit has been exceeded" after exactly 5 more minutes

[TobiasWehrum]

[REQUIRED] Please fill in the following fields:

• Unity editor version: 2022.3.17f1
• Firebase Unity SDK version: 11.6.0
• Source you installed the SDK: .unitypackage
• Problematic Firebase Component: Storage
• Other Firebase Components in use: Auth, Firestore, Functions
• Additional SDKs you are using: Sentry, Cesium
• Platform you are using the Unity editor on: Windows 10
• Platform you are targeting: Desktop
• Scripting Runtime: Mono
• Pre-built SDK from the website or open-source from this repo: Pre-built SDK

[REQUIRED] Please describe the issue here:

I'm trying to upload a 423 MB video file with PutFileAsync while using a progress handler to show the upload progress.

After exactly 5 minutes the upload always seems to stop: The progress handler continues to be called, but BytesTransferred will keep the value it had at the 5 minute mark (instead of steadily progressing like before).

After exactly 5 more minutes PutFileAsync throws Firebase.Storage.StorageException: The operation retry limit has been exceeded.

If I try to upload anything else after that exception in the Unity Editor the whole Editor will freeze and has to be closed via Task manager.

If I try to upload anything else after that exception in a build, the app will either directly crash or will throw a System.ApplicationException: ret == ((((DWORD )0x00000000L) ) + 0 ) ---> System.ApplicationException: ret == ((((DWORD )0x00000000L) ) + 0 ) immediately.

This happens in the Unity Editor and in a desktop build on Windows 10 every single time I tested it. I also tested PutBytesAsync in the Unity Editor, and it has the exact same behaviour.

We also tested tested this in a build on another Windows 10 machine, and there too we got Firebase.Storage.StorageException: The operation retry limit has been exceeded after pretty much exactly 10 minutes (and then the System.ApplicationException if we tried to upload again).

If I set MaxDownloadRetryTime, MaxUploadRetryTime and MaxOperationRetryTime to 1 second, Firebase.Storage.StorageException: The operation retry limit has been exceeded will be thrown right after BytesTransferred stops increasing, further suggesting that something goes wrong there and that it starts retrying at that point.

[paulinon]

Hi @TobiasWehrum,

Does setting MaxUploadRetryTime to a higher value make a difference? You may use this code to have a maximum upload retry time of one hour:

storage.MaxUploadRetryTime = new TimeSpan(TimeSpan.TicksPerHour);

If that doesn't work, could you provide a minimal, reproducible example of your implementation so that we can identify what's causing this to happen?

[TobiasWehrum]

@paulinon When I tested that yesterday (with new TimeSpan(1, 0, 0)) the only difference it made was how long it took to throw Firebase.Storage.StorageException: The operation retry limit has been exceeded after it stopped uploading at the 5 minute mark.

I'll make an example later.

[TobiasWehrum]

@paulinon, here is how to reproduce the problem using Unity 2022.3.17f1 on Windows 10 in the editor. I get a 100% reproduction rate on this.

1. Make an empty project.
2. Add FirebaseStorage.unitypackage version 11.6.0 or 11.7.0 (I tested both).
3. Deactivate "Validate References" in the inspector for Assets/ExternalDependencyManager/Editor/1.2.179/Google.IOSResolver.dll and Assets/Firebase/Editor/Firebase.Editor.dll to resolve the errors in the Console.
4. Create or reuse a Firebase project with a storage component and at least one folder with writing permissions set to if true.
5. Add that project's google-services.json to the project.
6. Add UploadTest.cs (below) to the project.
7. Open the example scene.
8. Create an empty GameObject and attach UploadTest as a component.
9. Fill out UploadTest:

• firebaseStorageUrl should point to your bucket, e.g. gs://my-app-name.appspot.com
• storagePathStringForUpload should point to a place inside a folder with if true writing permissions, e.g. bugtest/testfile
• localFilePathForUpload should point to a file that is big enough that the upload will take longer than 5 minutes (e.g. 423 MB for me), e.g. C:\bigfile.mp4.

10. Run the scene for 10 minutes and observe the console output:

• At 05:00 it will stop transfering bytes.
• A long time later on it might or might not restart for a bit (in the log below at 09:29, in another test it happened at 09:57).
• Around 10:01 it will throw StorageException: The operation retry limit has been exceeded..

11. Bonus step: After the StorageException, stop the player, and close Unity. Most of the time Unity will crash.

If you try step 10 again while commenting in the second PutFileAsync block in UploadTest.cs, you can observe that any new call to PutFileAsync after the StorageException: The operation retry limit has been exceeded. will immediately freeze the Unity Editor. In the build it would probably crash.

Example log output excerpt:

Database URL not set in the Firebase config.
[00:00:00] Transfered: 0 of 433315522 bytes
[00:00:00] Transfered: 65536 of 433315522 bytes
[00:00:01] Transfered: 131072 of 433315522 bytes
[00:00:02] Transfered: 196608 of 433315522 bytes
[...]
[00:04:54] Transfered: 29753344 of 433315522 bytes
[00:04:55] Transfered: 29818880 of 433315522 bytes
[00:04:56] Transfered: 29884416 of 433315522 bytes
[00:04:56] Transfered: 29949952 of 433315522 bytes
[00:04:57] Transfered: 30015488 of 433315522 bytes
[00:04:57] Transfered: 30081024 of 433315522 bytes
[00:04:58] Transfered: 30146560 of 433315522 bytes
[00:04:58] Transfered: 30212096 of 433315522 bytes
[00:04:59] Transfered: 30277632 of 433315522 bytes
[00:05:00] Transfered: 30343168 of 433315522 bytes
[00:05:01] Transfered: 30343168 of 433315522 bytes
[00:05:01] Transfered: 30343168 of 433315522 bytes
[00:05:02] Transfered: 30343168 of 433315522 bytes
[00:05:03] Transfered: 30343168 of 433315522 bytes
[00:05:04] Transfered: 30343168 of 433315522 bytes
[00:05:04] Transfered: 30343168 of 433315522 bytes
[00:05:05] Transfered: 30343168 of 433315522 bytes
[00:05:06] Transfered: 30343168 of 433315522 bytes
[...]
[00:09:23] Transfered: 30343168 of 433315522 bytes
[00:09:23] Transfered: 30343168 of 433315522 bytes
[00:09:24] Transfered: 30343168 of 433315522 bytes
[00:09:24] Transfered: 30343168 of 433315522 bytes
[00:09:25] Transfered: 30343168 of 433315522 bytes
[00:09:26] Transfered: 30343168 of 433315522 bytes
[00:09:26] Transfered: 30343168 of 433315522 bytes
[00:09:27] Transfered: 30343168 of 433315522 bytes
[00:09:27] Transfered: 30343168 of 433315522 bytes
[00:09:28] Transfered: 30343168 of 433315522 bytes
[00:09:29] Transfered: 30343168 of 433315522 bytes
[00:09:29] Transfered: 30408704 of 433315522 bytes
[00:09:30] Transfered: 30474240 of 433315522 bytes
[00:09:30] Transfered: 30539776 of 433315522 bytes
[00:09:31] Transfered: 30605312 of 433315522 bytes
[00:09:31] Transfered: 30670848 of 433315522 bytes
[00:09:32] Transfered: 30736384 of 433315522 bytes
[00:09:32] Transfered: 30801920 of 433315522 bytes
[00:09:33] Transfered: 30867456 of 433315522 bytes
[...]
[00:09:57] Transfered: 33619968 of 433315522 bytes
[00:09:58] Transfered: 33685504 of 433315522 bytes
[00:09:58] Transfered: 33751040 of 433315522 bytes
[00:09:59] Transfered: 33816576 of 433315522 bytes
[00:09:59] Transfered: 33882112 of 433315522 bytes
[00:10:00] Transfered: 33947648 of 433315522 bytes
[00:10:01] Transfered: 34013184 of 433315522 bytes
StorageException: The operation retry limit has been exceeded.

UploadTest.cs:

using System;
using System.Threading;
using Firebase;
using Firebase.Storage;
using UnityEngine;

public class UploadTest : MonoBehaviour
{
    [SerializeField] private string firebaseStorageUrl;
    [SerializeField] private string storagePathStringForUpload;
    [SerializeField] private string localFilePathForUpload;

    private CancellationTokenSource cancellationTokenSource;

    protected async void Awake()
    {
        var depencencyResult = await FirebaseApp.CheckAndFixDependenciesAsync();
        if (depencencyResult != DependencyStatus.Available)
        {
            Debug.LogError("FirebaseApp.CheckAndFixDependenciesAsync failed.");
            return;
        }

        var storage = FirebaseStorage.DefaultInstance;
        var storageRef = storage.GetReferenceFromUrl(firebaseStorageUrl);

        cancellationTokenSource = new CancellationTokenSource();

        var startTime = Time.realtimeSinceStartup;

        var progressHandler = new StorageProgress<UploadState>(state =>
        {
            var currentTime = TimeSpan.FromSeconds(Mathf.FloorToInt(Time.realtimeSinceStartup - startTime));
            Debug.Log($"[{currentTime}] Transfered: {state.BytesTransferred} of {state.TotalByteCount} bytes");
        });

        try
        {
            await storageRef.Child(storagePathStringForUpload).PutFileAsync(localFilePathForUpload, null, progressHandler, cancellationTokenSource.Token, null);

            Debug.Log("Upload finished");
        }
        catch (Exception e)
        {
            Debug.LogError(e);
        }

        /*
        // If you comment this block in, Unity will freeze immediately when calling PutFileAsync() (after the previous block failed with a StorageException).
        // Editor.log will only show a single line before ending abruptly:
        // ERROR: ret == ((((DWORD )0x00000000L) ) + 0 )
        try
        {
            await storageRef.Child(storagePathStringForUpload).PutFileAsync(localFilePathForUpload, null, progressHandler, cancellationTokenSource.Token, null);

            Debug.Log("Upload finished");
        }
        catch (Exception e)
        {
            Debug.LogError(e);
        }
        */

        cancellationTokenSource = null;
    }

    protected void OnDestroy()
    {
        if (cancellationTokenSource != null)
        {
            cancellationTokenSource.Cancel();
            cancellationTokenSource = null;
        }
    }
}

[paulinon]

Hi @TobiasWehrum,

Thanks for the additional information. I was able to reproduce the issue, and it persists even after setting the upload retry time. That said, I'll be relaying my observations to our team. You may refer to this thread for updates.

[TobiasWehrum]

Thank you, @paulinon - looking forward to this getting resolved.

On a related note, I think the same bug might apply not just to uploading, but also to downloading. I didn't have time to investigate yet, but yesterday I observed a particularly big download on a slow internet connection that seemed to stop after about 5 minutes, and then threw a StorageException: The operation retry limit has been exceeded. not long after.

[TobiasWehrum]

@paulinon I just tested it to confirm it: Literally the same (stops transfering after 5 minutes, operation retry limit StorageException at 10 minutes) happens when downloading via GetFileAsync.

[a-maurice]

This should be fixed with the latest release, https://github.com/firebase/firebase-unity-sdk/releases/tag/v11.9.0
The 5 minute timeout was the default set by Curl, so we just adjusted that.

[TobiasWehrum]

@a-maurice Thanks for the fix and the info what it was!

It doesn't sound like the curl timeout should lead to the observed crashes/freezes though. There might be a secondary bug here that could occur under other conditions too, like e.g. when "The operation retry limit has been exceeded" is thrown for other reasons.

[a-maurice]

Yeah, it is certainly possible, and we will keep an eye out for possible fixes for that.