GCP InputStream in GCP actually readsAllBytes

[eahrold]

When fetching larger files, we are seeing throwIfFatal detected a jvm fatal exception, which is thrown and logged below: java.lang.OutOfMemoryError: Java heap space

It looks like what's happening is that the underlying method used in objectstorage, is actually just pulling all bytes from the Blob into memory, and not creating a usable stream.

micronaut-object-storage/object-storage-gcp/src/main/java/io/micronaut/objectstorage/googlecloud/GoogleCloudStorageEntry.java

Lines 49 to 51 in e0ec2d9

	public InputStream getInputStream() {
	return new ByteArrayInputStream(blob.getContent());
	}

The relevant stack.

        at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.storage.Retrying.run(Retrying.java:65)
	at com.google.cloud.storage.StorageImpl.run(StorageImpl.java:1527)
	at com.google.cloud.storage.StorageImpl.readAllBytes(StorageImpl.java:622)
	at com.google.cloud.storage.Blob.getContent(Blob.java:778)
	at io.micronaut.objectstorage.googlecloud.GoogleCloudStorageEntry.getInputStream(GoogleCloudStorageEntry.java:50)

What we've got under the hood is similar to this.

7.2. Download endpoint from https://guides.micronaut.io/latest/micronaut-object-storage-gcp-gradle-java.html

[eahrold]

There may be some autoclose foot-guns here, but I've implemented the equivalent of this, and seems to work for files of any size

    @NonNull
    @Override
     public InputStream getInputStream() {
          return Channels.newInputStream(blob.reader());
     }

[alvarosanchez]

@eahrold thank you very much! Would you be willing to send a PR so that you get credit for the fix?

[eahrold]

@alvarosanchez here you go, #467