Skip to content

test: add new StorageNativeCanaryTest #1872

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Feb 21, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions google-cloud-storage/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -340,5 +340,15 @@
</dependency>
</dependencies>
</profile>
<profile>
<id>native</id>
<properties>
<!--
Override the default match pattern for native image tests since we have a custom
test specifically for native image validation.
-->
<test>com.google.cloud.storage.it.StorageNativeCanary</test>
</properties>
</profile>
</profiles>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
/*
* Copyright 2023 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.google.cloud.storage.it;

import static com.google.cloud.storage.TestUtils.assertAll;
import static com.google.common.truth.Truth.assertThat;

import com.google.api.gax.paging.Page;
import com.google.cloud.ReadChannel;
import com.google.cloud.WriteChannel;
import com.google.cloud.storage.Blob;
import com.google.cloud.storage.BlobId;
import com.google.cloud.storage.BlobInfo;
import com.google.cloud.storage.BucketInfo;
import com.google.cloud.storage.DataGenerator;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.Storage.BlobListOption;
import com.google.cloud.storage.Storage.BlobSourceOption;
import com.google.cloud.storage.Storage.BlobWriteOption;
import com.google.cloud.storage.StorageOptions;
import com.google.common.collect.ImmutableList;
import com.google.common.io.ByteStreams;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.nio.channels.Channels;
import java.nio.channels.WritableByteChannel;
import java.util.List;
import java.util.UUID;
import org.junit.Test;

// Intentionally avoid StorageITRunner here. It touches lots of code at a semi-static level making
// native-test have a hard time.
public final class StorageNativeCanary {

private static final int _256KiB = 256 * 1024;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider renaming this to CHUNK_SIZE

Copy link
Collaborator Author

@BenWhitehead BenWhitehead Feb 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For gcs 256 KiB is an important number, as it is the minimum number of bytes which can be in a resumable upload chunk and still keep the resumable session active. The fact that we are setting the chunk size to this value is important and I feel the code reads more like it is important having the name it does.

readChannel.setChunkSize(_256KiB); is more clear about the boundary which is being set rather than readChannel.setChunkSize(CHUNK_SIZE);.

private static final byte[] bytes = DataGenerator.base64Characters().genBytes(512 * 1024);

@Test
public void canary_happyPath_http() throws Exception {
assertBehaviorOfPrimaryStorageActions(StorageOptions.http().build().getService());
}

@Test
public void canary_happyPath_grpc() throws Exception {
assertBehaviorOfPrimaryStorageActions(StorageOptions.grpc().build().getService());
}

/**
* When testing on Native Image, we're primarily wanting to verify the primary code paths are
* properly detected by the native image compiler.
*
* <p>For Storage, we have a few "primary code paths" we want to ensure are validated:
*
* <ul>
* <li>Can a (Unary) Request Succeed?
* <li>Can a (ServerStream) Object Read Request Succeed?
* <li>Can a (ClientStream) Object Write Request Succeed?
* <li>Can a (Page over Unary) Paginated Request Succeed?
* </ul>
*
* To validate this, our happy path test is as follows:
*
* <ul>
* <li>Create a temporary bucket (Unary)
* <li>Insert two (2) objects (Unary, ServerStream)
* <li>List all objects, using a pageSize of 1 (Page over Unary)
* <li>Read all bytes of each object (ServerStream)
* <li>Delete each object (Unary)
* <li>Delete temporary bucket (Unary)
* </ul>
*/
private static void assertBehaviorOfPrimaryStorageActions(Storage storage) throws Exception {
// create a temporary bucket
try (TemporaryBucket temporaryBucket =
TemporaryBucket.newBuilder()
.setStorage(storage)
.setBucketInfo(BucketInfo.of("java-storage-grpc-" + UUID.randomUUID()))
.build()) {
String bucketName = temporaryBucket.getBucket().getName();
String obj1Name = UUID.randomUUID().toString();
String obj2Name = UUID.randomUUID().toString();

// insert 2 objects
BlobInfo info1 = BlobInfo.newBuilder(bucketName, obj1Name).build();
BlobInfo info2 = BlobInfo.newBuilder(bucketName, obj2Name).build();
uploadUsingWriter(storage, info1);
uploadUsingWriter(storage, info2);

// list objects
Page<Blob> page = storage.list(bucketName, BlobListOption.pageSize(1));
List<Blob> blobs = ImmutableList.copyOf(page.iterateAll());

// read all bytes of each object
List<BlobWithContent> actual =
blobs.stream()
.map(info -> readAll(storage, info))
.collect(ImmutableList.toImmutableList());

List<Boolean> deletes =
blobs.stream()
.map(b -> storage.delete(b.getBlobId(), BlobSourceOption.generationMatch()))
.collect(ImmutableList.toImmutableList());

assertAll(
() -> {
List<String> actualNames =
actual.stream()
.map(BlobWithContent::getInfo)
.map(BlobInfo::getBlobId)
.map(BlobId::getName)
.collect(ImmutableList.toImmutableList());

assertThat(actualNames).containsExactly(info1.getName(), info2.getName());
},
() -> assertThat(actual.get(0).getContent()).isEqualTo(bytes),
() -> assertThat(actual.get(1).getContent()).isEqualTo(bytes),
() -> assertThat(deletes.get(0)).isTrue(),
() -> assertThat(deletes.get(1)).isTrue());
}
}

private static void uploadUsingWriter(Storage storage, BlobInfo info) throws IOException {
try (WriteChannel writeChannel = storage.writer(info, BlobWriteOption.doesNotExist())) {
// set our size to the smallest resumable size, so we can send multiple requests
writeChannel.setChunkSize(_256KiB);
ByteStreams.copy(Channels.newChannel(new ByteArrayInputStream(bytes)), writeChannel);
}
}

private static BlobWithContent readAll(Storage storage, BlobInfo info) {
try (ReadChannel readChannel =
storage.reader(info.getBlobId(), BlobSourceOption.generationMatch());
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
WritableByteChannel writeChannel = Channels.newChannel(outputStream)) {
// only buffer up to half the object
readChannel.setChunkSize(_256KiB);
ByteStreams.copy(readChannel, writeChannel);
return new BlobWithContent(info, outputStream.toByteArray());
} catch (IOException e) {
throw new RuntimeIOException(e);
}
}

private static final class BlobWithContent {
private final BlobInfo info;
private final byte[] content;

private BlobWithContent(BlobInfo info, byte[] content) {
this.info = info;
this.content = content;
}

public BlobInfo getInfo() {
return info;
}

public byte[] getContent() {
return content;
}
}

private static final class RuntimeIOException extends RuntimeException {
private RuntimeIOException(IOException cause) {
super(cause);
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -5,29 +5,4 @@
# build time. Initializing these classes explicitly at build time results in a
# successful build.
Args = \
--initialize-at-build-time=com.google.cloud.conformance.storage.v1,\
com.google.protobuf,\
com.google.auth.oauth2,\
com.google.cloud.storage.conformance.retry,\
com.google.common.base.Charsets,\
com.google.gson.stream.JsonReader,\
com.google.api.client.util,\
com.google.api.client.http.javanet.NetHttpTransport,\
com.google.api.client.http.HttpTransport,\
com.google.api.client.json.JsonParser$1,\
com.google.api.client.json.gson.GsonParser$1,\
com.google.common.io.BaseEncoding,\
com.google.common.math.IntMath$1,\
com.google.common.collect.Platform,\
com.google.gson.Gson,\
com.google.common.truth,\
com.google.common.collect,\
com.google.gson.internal.reflect,\
com.google.gson.internal.bind,\
com.google.gson.internal,\
com.google.gson.internal.sql.SqlTypesSupport,\
com.google.gson.FieldNamingPolicy$3,\
com.google.gson.LongSerializationPolicy$2,\
net.jqwik


--initialize-at-build-time=net.jqwik
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,4 @@
"allDeclaredFields":true,
"allDeclaredMethods":true,
"methods":[{"name":"<init>","parameterTypes":[] }]}
,
{
"name":"com.google.cloud.storage.conformance.retry.TestBench$RetryTestResource",
"allDeclaredFields":true,
"methods":[{"name":"<init>","parameterTypes":[] }]}
]