Upload of buffer to Google Storage creates File with no buffer data

[GannonSystems]

I am attempting to write a data buffer to a Google Storage Bucket using the CloudStorage/Async/NoCallback/Upload example as a starting point. Rather than pass a file from a FS, I am using BlobConfig to upload a test buffer. I plan on passing ASCII text and binary data in the future, that will be uploaded on ~30mins intervals.

FirebaseClient appears to connect and I can see the the correct filename created in the Cloud Storage dashboard. However, the file does not contain the buffer data, but rather this data when the file is downloaded via the Cloud Storage dashboard:

{"metadata":{"firebaseStorageDownloadTokens":"<TOKEN>"}}

I suspect there is something simple I am missing either in FirebaseClient or in the way Google Could Storage works, as it is my first time using these.

Here is the code modified from the CloudStorage/Async/NoCallback/Upload example:

As Globals:

uint8_t source[17] = "TEST IS A TEST.\n";
uint8_t dest[2048];

BlobConfig upload_data(source, 17);
BlobConfig download_data(dest, 2048);

The main Loop():

void loop()
{
    // The JWT token processor required for ServiceAuth and CustomAuth authentications.
    // JWT is a static object of JWTClass and it's not thread safe.
    // In multi-threaded operations (multi-FirebaseApp), you have to define JWTClass for each FirebaseApp,
    // and set it to the FirebaseApp via FirebaseApp::setJWTProcessor(<JWTClass>), before calling initializeApp.
    JWT.loop(app.getAuth());

    app.loop();

    cstorage.loop();

    if (app.ready() && !taskCompleted)
    {
        taskCompleted = true;

        Serial.println("Upload file...");

        GoogleCloudStorage::uploadOptions options;
        options.mime = "text/plain";
        options.uploadType = GoogleCloudStorage::upload_type_resumable;
        // options.uploadType = GoogleCloudStorage::upload_type_simple;
        // options.uploadType = GoogleCloudStorage::upload_type_multipart;

        cstorage.upload(aClient, GoogleCloudStorage::Parent(STORAGE_BUCKET_ID, "test.txt"), getBlob(upload_data), options, aResult_no_callback);
        Serial.println("Upload Call returned...");
    }

    printResult(aResult_no_callback);
}

Here is the console output:

Firebase Client v1.3.5
Initializing app...
Event task: task_2, msg: initializing, code: 1
Event task: task_2, msg: token signing, code: 6
Debug task: task_2, msg: Getting current time...
Event task: task_2, msg: authenticating, code: 7
Debug task: task_2, msg: Connecting to server...
Event task: task_2, msg: auth request sent, code: 8
Event task: task_2, msg: auth response received, code: 9
Debug task: task_2, msg: Terminating the server connection...
Upload file...
Upload Call returned...
Event task: task_2, msg: ready, code: 10
Debug task: task_2, msg: Connecting to server...

Thanks for the help!

[mobizt]

You cannot choose the resumable upload for small files or data. It should be 256k up as the Google Cloud Storage API document stated

You should choose a simple upload type for small file instead.

[GannonSystems]

@mobizt Thank you, I missed that. After changing to GoogleCloudStorage::upload_type_simple, I receive the return message but no file is actually uploaded. Not even the empty filename I received previously. Here is the message:

Firebase Client v1.3.5
Initializing app...
Event task: task_2, msg: initializing, code: 1
Event task: task_2, msg: token signing, code: 6
Debug task: task_2, msg: Getting current time...
Event task: task_2, msg: authenticating, code: 7
Debug task: task_2, msg: Connecting to server...
Event task: task_2, msg: auth request sent, code: 8
Event task: task_2, msg: auth response received, code: 9
Debug task: task_2, msg: Terminating the server connection...
Upload file...
Uploaded, task: task_2, -1% (0 of 0)
Upload task: task_2, completed!Download URL: 
Upload Call returned...
Event task: task_2, msg: ready, code: 10
Debug task: task_2, msg: Connecting to server...

I tweaked the code to print the Result message after the upload is called:

    if (app.ready() && !taskCompleted)
    {
        taskCompleted = true;

        Serial.println("Upload file...");

        GoogleCloudStorage::uploadOptions options;
        options.mime = "text/plain";
        // options.uploadType = GoogleCloudStorage::upload_type_resumable;
        options.uploadType = GoogleCloudStorage::upload_type_simple;
        // options.uploadType = GoogleCloudStorage::upload_type_multipart;

        cstorage.upload(aClient, GoogleCloudStorage::Parent(STORAGE_BUCKET_ID, "test2.txt"), getBlob(upload_data), options, aResult);
        
        // TEST
        Firebase.printf("Uploaded, task: %s, %d%s (%d of %d)\n", aResult.uid().c_str(), aResult.uploadInfo().progress, "%", aResult.uploadInfo().uploaded, aResult.uploadInfo().total);
        if (aResult.uploadInfo().total == aResult.uploadInfo().uploaded)
        {
            Firebase.printf("Upload task: %s, completed!", aResult.uid().c_str());
            Serial.print("Download URL: ");
            Serial.println(aResult.uploadInfo().downloadUrl);
        }

        Serial.println("Upload Call returned...");
        // END TEST
    }

Is the -1% and no URL indicative of the upload not actually happening?

[mobizt]

It's a bug and thanks for reporting this.

Now you can update the library to latest v1.3.7 with bugs fixed.

[GannonSystems]

@mobizt Thank you for the quick update. I have updated to 1.37, however I now receive a file error (I assume passing the Blob) that was not observed in the prior build:

The code:

uint8_t source[2048] = "TEST IS A TEST!\n";
uint8_t dest[2048];

BlobConfig upload_data(source, 2048);
BlobConfig download_data(dest, 2048);

    if (app.ready() && !taskCompleted)
    {
        taskCompleted = true;

        Serial.println("Upload file...");

        GoogleCloudStorage::uploadOptions options;
        options.mime = "text/plain";
        // options.uploadType = GoogleCloudStorage::upload_type_resumable;
        options.uploadType = GoogleCloudStorage::upload_type_simple;
        // options.uploadType = GoogleCloudStorage::upload_type_multipart;

        cstorage.upload(aClient, GoogleCloudStorage::Parent(STORAGE_BUCKET_ID, "test2.txt"), getBlob(upload_data), options, aResult_no_callback);
        
    }

The serial output:

Firebase Client v1.3.7
Initializing app...
Event task: task_2, msg: initializing, code: 1
Event task: task_2, msg: token signing, code: 6
Debug task: task_2, msg: Getting current time...
Event task: task_2, msg: authenticating, code: 7
Debug task: task_2, msg: Connecting to server...
Event task: task_2, msg: auth request sent, code: 8
Event task: task_2, msg: auth response received, code: 9
Debug task: task_2, msg: Terminating the server connection...
Upload file...
Event task: task_2, msg: ready, code: 10
Error task: task_2, msg: error reading file, code: -104
Debug task: task_2, msg: Connecting to server...

I assume it is possible there was always a Blob issue, it had just not gotten to that entry point yet.

[mobizt]

For the filesystem in ESP32 Core v3.x.x, it has changed from the v2.x.x that causes the internal defined FS object in the library does not work as expected.

This is the FS.h and ESP32 Core issue in v3.x.x.

You have to modify the example by replacing these lines.

#include <FirebaseClient.h>

#if defined(ENABLE_FS)      // Defined in this library
#if defined(FLASH_SUPPORTS) // Defined in this library
#if defined(ESP32)
#include <SPIFFS.h>
#endif
#define MY_FS SPIFFS
#else
#include <SPI.h>
#include <SD.h>
#define MY_FS SD
#endif
#endif

With these lines for SPIFFS.

#include <FS.h>
File myFile;

#if defined(ESP32)
#include <SPIFFS.h>
#endif
#define MY_FS SPIFFS

#include <FirebaseClient.h>

And these lines

void fileCallback(File &file, const char *filename, file_operating_mode mode)
{
    // FILE_OPEN_MODE_READ, FILE_OPEN_MODE_WRITE and FILE_OPEN_MODE_APPEND are defined in this library
    // MY_FS is defined in this example
    switch (mode)
    {
    case file_mode_open_read:
        file = MY_FS.open(filename, FILE_OPEN_MODE_READ);
        break;
    case file_mode_open_write:
        file = MY_FS.open(filename, FILE_OPEN_MODE_WRITE);
        break;
    case file_mode_open_append:
        file = MY_FS.open(filename, FILE_OPEN_MODE_APPEND);
        break;
    case file_mode_remove:
        MY_FS.remove(filename);
        break;
    default:
        break;
    }
}

With these lines.

void fileCallback(File &file, const char *filename, file_operating_mode mode) {
  // FILE_OPEN_MODE_READ, FILE_OPEN_MODE_WRITE and FILE_OPEN_MODE_APPEND are defined in this library
  // MY_FS is defined in this example
  switch (mode) {
    case file_mode_open_read:
      myFile = MY_FS.open(filename, FILE_OPEN_MODE_READ);
      break;
    case file_mode_open_write:
      myFile = MY_FS.open(filename, FILE_OPEN_MODE_WRITE);
      break;
    case file_mode_open_append:
      myFile = MY_FS.open(filename, FILE_OPEN_MODE_APPEND);
      break;
    case file_mode_remove:
      MY_FS.remove(filename);
      break;
    default:
      break;
  }

  file = myFile;
}

For the BLOB, library works normally.

[mobizt]

This means FS object should be defined globally in ESP32 core v3.x.x.

I will update the examples and documentation later to work around latest ESP32 Core FS issue.

[GannonSystems]

I am back to attempting to write a file to Google Storage passed as a Blob instead of a File. For my use case, all files written are <50KB. I have ENABLE_FS commented out in Config.h. I received a error reading file, code: -104 and traced it this this line in CloudStorage.h (~Line 613):

if (sData->request.file_data.file_size == 0)
    return setClientError(request, FIREBASE_ERROR_FILE_READ);

When using BlobConfig, file_data.file_size is always =0. I believe this should be the following in CloudStorage.h:

#if defined(ENABLE_FS)
            if (sData->request.file_data.file_size == 0)
                return setClientError(request, FIREBASE_ERROR_FILE_READ);
#else
            if (sData->request.file_data.data_size == 0)
                return setClientError(request, FIREBASE_ERROR_FILE_READ);
#endif

This does resolve the error reading file, code: -104, however no data is written to storage. I am guessing that the file_data.file_size=0 is causing the Blob not to be written somewhere down the line, but I have not found it yet. Here are my current results:

Firebase Client v1.3.7
Initializing app...
Event task: task_2, msg: initializing, code: 1
Event task: task_2, msg: token signing, code: 6
Debug task: task_2, msg: Getting current time...
Event task: task_2, msg: authenticating, code: 7
Debug task: task_2, msg: Connecting to server...
Event task: task_2, msg: auth request sent, code: 8
Event task: task_2, msg: auth response received, code: 9
Debug task: task_2, msg: Terminating the server connection...
Upload file...
Event task: task_2, msg: ready, code: 10
Debug task: task_2, msg: Connecting to server...

...and stops here with no file being written to Cloud Storage.

[mobizt]

You should get the compilation error instead when you remove ENABLE_FS macro.

If you don't get the compilation error, your Arduino IDE is actually used the pre-compiled (cached) library.

The library was updated with the issue fixed, please try.

[GannonSystems]

I missed you had posted an update. Everything is working now. I have no problem sending a buffer as Blob with ENABLE_FS commented out.

Thanks again for taking the time to work through this with me!

Firebase Client v1.3.9
Initializing app...
Event task: task_2, msg: initializing, code: 1
Event task: task_2, msg: token signing, code: 6
Debug task: task_2, msg: Getting current time...
Event task: task_2, msg: authenticating, code: 7
Debug task: task_2, msg: Connecting to server...
Event task: task_2, msg: auth request sent, code: 8
Event task: task_2, msg: auth response received, code: 9
Debug task: task_2, msg: Terminating the server connection...
Upload file...
Event task: task_2167, msg: ready, code: 10
Debug task: task_2167, msg: Connecting to server...
Uploaded, task: task_2167, 100% (17 of 17)
Upload task: task_2167, completed!
Download URL: