Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Google Cloud Storage NIO Alpha support #1090

Merged
merged 38 commits into from
Jul 3, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
affed4d
Cloud Storage Java 7 nio library
jart Feb 3, 2016
045dce2
Migrated GCS NIO from App Engine to gcloud-storage
jean-philippe-martin Mar 2, 2016
9784842
Run google-java-format on GCS NIO
jart Mar 3, 2016
a7c4679
Correct more style issues and remove warnings
jart Mar 4, 2016
b5733fb
Add FileSystemProvider, tests now pass
jean-philippe-martin Mar 11, 2016
9b0da03
Remove spi.FileSystemProvider
jean-philippe-martin Mar 17, 2016
7ca9fcb
Add license headers
jart Mar 22, 2016
d5cf006
Split off the "normal case" example from the "legacy case" one.
jean-philippe-martin Apr 5, 2016
46596d6
Fix javadoc in examples to match the README
jean-philippe-martin Apr 6, 2016
2125f8a
Add url to the pom for nio and nio-examples (#909)
jean-philippe-martin Apr 12, 2016
73b2917
Implement bare-bones blob listing
jean-philippe-martin Apr 14, 2016
e874b76
Move gcloud->cloud (issue #877) and update version number
jean-philippe-martin Apr 18, 2016
10180eb
Integration test for ls (#931)
jean-philippe-martin Apr 19, 2016
dc71290
Move Stat and clarify example command line
jean-philippe-martin Apr 26, 2016
a538f94
Fix markdown and simplify command
jean-philippe-martin Apr 27, 2016
9e94824
NIO CountBytes example (#975)
jean-philippe-martin May 4, 2016
ded161f
Add ParallelCountBytes
jean-philippe-martin May 4, 2016
db89ffe
Refactor to use ExecutorService and BaseEncoding
jean-philippe-martin May 5, 2016
be727b5
Add shutdown
jean-philippe-martin May 5, 2016
a176250
Close channels
jean-philippe-martin May 6, 2016
f2aa350
Close channel and fix typo (#1036)
jean-philippe-martin Jun 1, 2016
1305c4d
Update pom.xml versions
jart Jun 13, 2016
85257a0
FakeStorageRpc with master changes
jart Jun 13, 2016
29ea8b0
Use appassembler for Compute and Dns examples (#1057)
jean-philippe-martin Jun 14, 2016
312565c
File system close declares it throws IOException (#1060)
jean-philippe-martin Jun 20, 2016
37fe1eb
Handle files over 2GB (#1065)
jean-philippe-martin Jun 21, 2016
595c16a
Rename LocalGcsHelper to LocalStorageHelper
jean-philippe-martin Jun 21, 2016
1e6b5aa
Document limitations of FakeStorageRpc (#1069)
jean-philippe-martin Jun 21, 2016
39ba077
More precise description of FakeStorageRpc limitations (#1070)
jean-philippe-martin Jun 21, 2016
5917ae7
Make sure every java file starts with the Apache license. (#1067)
jean-philippe-martin Jun 23, 2016
3b55bfb
Improve NIO documentation (#1076)
jean-philippe-martin Jun 24, 2016
bb34177
Temporarily hide FakeStorageRpc and LocalStorageHelper (#1079)
mziccard Jun 27, 2016
8c12052
Improve nio and contrib READMEs (#1077)
jean-philippe-martin Jun 28, 2016
d31301d
Make examples' javadoc more homogenous and minor html fixes
mziccard Jun 28, 2016
5dd4692
Use appassembler in Pub/Sub example
mziccard Jun 29, 2016
24b73ab
Minor checkstyle and codacy fixes to gcloud-java-nio
mziccard Jun 30, 2016
c65d2fc
Minor javadoc fixes to gcloud-java-nio
mziccard Jun 30, 2016
90f2c4b
Refactor nio's package-info and add snippets
mziccard Jul 3, 2016
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,3 +1,19 @@
/*
* Copyright 2016 Google Inc. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.google.cloud.bigquery;

import com.google.api.services.bigquery.model.ErrorProto;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,19 @@
/*
* Copyright 2016 Google Inc. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.google.cloud.bigquery;

import com.google.api.services.bigquery.model.JobStatistics2;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,19 @@
/*
* Copyright 2016 Google Inc. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.google.cloud.bigquery;

import com.google.common.base.MoreObjects;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,19 @@
/*
* Copyright 2016 Google Inc. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.google.cloud.bigquery;

import com.google.api.services.bigquery.model.UserDefinedFunctionResource;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,19 @@
/*
* Copyright 2016 Google Inc. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.google.cloud.bigquery;

import static org.junit.Assert.assertEquals;
Expand Down
18 changes: 18 additions & 0 deletions gcloud-java-contrib/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,17 @@ Packages that provide higher-level abstraction/functionality for common gcloud-j
[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java)
[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969)

Contents
--------

* [gcloud-java-nio](./gcloud-java-nio/): NIO Filesystem Provider for Google Cloud Storage.
* [gcloud-java-nio-examples](./gcloud-java-nio-examples/): How to add Google Cloud Storage NIO after the fact.

Quickstart
----------

### gcloud-java-nio

If you are using Maven, add this to your pom.xml file
```xml
<dependency>
Expand All @@ -28,6 +37,10 @@ If you are using SBT, add this to your dependencies
libraryDependencies += "com.google.cloud" % "gcloud-java-contrib" % "0.2.4"
```

### gcloud-java-nio-examples

See its [README](./gcloud-java-nio-examples/README.md) for instructions on how to run it.

Java Versions
-------------

Expand All @@ -42,6 +55,11 @@ It is currently in major version zero (``0.y.z``), which means that anything
may change at any time and the public API should not be considered
stable.

See also
--------

* [gcloud-java-examples](../gcloud-java-examples) for an example of how to use NIO normally.

Contributing
------------

Expand Down
44 changes: 44 additions & 0 deletions gcloud-java-contrib/gcloud-java-nio-examples/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
Example of adding the Google Cloud Storage NIO Provider to a legacy jar
=======================================================================

This project shows how to add Google Cloud Storage capabilities to a jar file for a Java 7
application that uses Java NIO without the need to recompile.

Note that whenever possible, you instead want to recompile the app and use the normal
dependency mechanism to add a dependency to gcloud-java-nio. You can see examples of
this in the [gcloud-java-examples](../../gcloud-java-examples) project.

To run this example:

1. Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled.

2. Log in using gcloud SDK (`gcloud auth login` in command line)

3. Compile the JAR with:
```
mvn package -DskipTests -Dmaven.javadoc.skip=true -Dmaven.source.skip=true
```

4. Run the sample with:

```
java -cp gcloud-java-contrib/gcloud-java-nio/target/gcloud-java-nio-0.2.5-SNAPSHOT-shaded.jar:gcloud-java-contrib/gcloud-java-nio-examples/target/gcloud-java-nio-examples-0.2.5-SNAPSHOT.jar com.google.cloud.nio.examples.ListFilesystems
```

Notice that it lists Google Cloud Storage, which it wouldn't if you ran it without the NIO jar:
```
java -cp gcloud-java-contrib/gcloud-java-nio-examples/target/gcloud-java-nio-examples-0.2.5-SNAPSHOT.jar com.google.cloud.nio.examples.ListFilesystems
```

The sample doesn't have anything about Google Cloud Storage in it. It gets that ability from the NIO
jar that we're adding to the classpath. You can use the NIO "fat shaded" jar for this purpose as it
also includes the dependencies for gcloud-java-nio.
The underlying mechanism is Java's standard [ServiceLoader](https://docs.oracle.com/javase/7/docs/api/java/util/ServiceLoader.html)
facility, the [standard way](http://docs.oracle.com/javase/7/docs/technotes/guides/io/fsp/filesystemprovider.html) to plug in NIO providers like this one.

If you have access to a project's source code you can also simply add gcloud-java-nio as
a dependency and let Maven pull in the required dependencies (this is what the NIO unit tests do).
This approach is preferable as the fat jar approach may waste memory on multiple copies of
dependencies.

[developers-console]:https://console.developers.google.com/
92 changes: 92 additions & 0 deletions gcloud-java-contrib/gcloud-java-nio-examples/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>gcloud-java-nio-examples</artifactId>
<packaging>jar</packaging>
<name>GCloud Java NIO examples</name>
<url>https://github.com/GoogleCloudPlatform/gcloud-java/tree/master/gcloud-java-contrib/gcloud-java-nio-examples</url>
<description>
Demonstrates how to use the gcloud-java-nio jar to add Google Cloud Storage functionality to legacy code.
</description>
<parent>
<groupId>com.google.cloud</groupId>
<artifactId>gcloud-java-contrib</artifactId>
<version>0.2.5-SNAPSHOT</version>
</parent>
<properties>
<site.installationModule>nio</site.installationModule>
</properties>
<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>gcloud-java-storage</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>19.0</version>
</dependency>
<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
<version>2.0.1</version>
</dependency>
<dependency>
<groupId>javax.inject</groupId>
<artifactId>javax.inject</artifactId>
<version>1</version>
</dependency>
<dependency>
<groupId>com.google.auto.service</groupId>
<artifactId>auto-service</artifactId>
<version>1.0-rc2</version>
<scope>provided</scope> <!-- to leave out of the all-deps jar -->
</dependency>
<dependency>
<groupId>com.google.auto.value</groupId>
<artifactId>auto-value</artifactId>
<version>1.1</version>
<scope>provided</scope> <!-- to leave out of the all-deps jar -->
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava-testlib</artifactId>
<version>19.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.truth</groupId>
<artifactId>truth</artifactId>
<version>0.27</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>1.9.5</version>
</dependency>
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.4</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<configuration>
<skip>false</skip>
</configuration>
</plugin>
</plugins>
</build>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
/*
* Copyright 2015 Google Inc. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.google.cloud.nio.examples;

import java.io.IOException;
import java.net.URI;
import java.nio.file.FileSystem;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.spi.FileSystemProvider;

/**
* ListFilesystems is a super-simple program that lists the available NIO filesystems.
*/
public class ListFilesystems {

/**
* See the class documentation.
*/
public static void main(String[] args) throws IOException {
listFilesystems();
}

private static void listFilesystems() {
System.out.println("Installed filesystem providers:");
for (FileSystemProvider p : FileSystemProvider.installedProviders()) {
System.out.println(" " + p.getScheme());
}
}

}
Loading