Skip to content

Commit

Permalink
Merge pull request #582 from ajkannan/test-paging-in-datastore-example
Browse files Browse the repository at this point in the history 
Use pagination in DatastoreExample
  • Loading branch information
@aozarov
aozarov committed Jan 26, 2016
2 parents c5bae84 + 360e048 commit d8bce91
Show file tree
Hide file tree
Showing 2 changed files with 71 additions and 61 deletions.
94 changes: 47 additions & 47 deletions gcloud-java-examples/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,54 +39,54 @@ To run examples from your command line:

4. Run an example using Maven from command line.

Here's an example run of `BigQueryExample`.

Before running the example, go to the [Google Developers Console][developers-console] to ensure
that BigQuery API is enabled. You can upload a CSV file `my_csv_file` to the `my_bucket` bucket
(replace `my_csv_file` and `my_bucket` with actual file and bucket names) using the GCS
[web browser](https://console.developers.google.com/storage/browser). The CSV file will be used to
load data into a BigQuery table and should look something like:
```csv
value1
value2
value3
```
Then you are ready to run the following example:
```
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create dataset new_dataset_id"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="list tables new_dataset_id"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'"
```

Here's an example run of `DatastoreExample`.
* Here's an example run of `BigQueryExample`.

Before running the example, go to the [Google Developers Console][developers-console] to ensure
that BigQuery API is enabled. You can upload a CSV file `my_csv_file` to the `my_bucket` bucket
(replace `my_csv_file` and `my_bucket` with actual file and bucket names) using the GCS
[web browser](https://console.developers.google.com/storage/browser). The CSV file will be used to
load data into a BigQuery table and should look something like:
```csv
value1
value2
value3
```
Then you are ready to run the following example:
```
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create dataset new_dataset_id"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="list tables new_dataset_id"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'"
```
* Here's an example run of `DatastoreExample`.
Note that you have to enable the Google Cloud Datastore API on the [Google Developers Console][developers-console] before running the following commands.
```
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="my_name add my\ comment"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="my_name display"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="my_name delete"
```

Here's an example run of `ResourceManagerExample`.

Be sure to change the placeholder project ID "my-project-id" with your own globally unique project ID.
```
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="create my-project-id"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="list"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="get my-project-id"
```

Here's an example run of `StorageExample`.

Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine.
```
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="upload /path/to/test.txt <bucket_name>"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="list <bucket_name>"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="download <bucket_name> test.txt"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="delete <bucket_name> test.txt"
```
Be sure to change the placeholder project ID "your-project-id" with your own project ID. Also note that you have to enable the Google Cloud Datastore API on the [Google Developers Console][developers-console] before running the following commands.
```
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="your-project-id my_name add my\ comment"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="your-project-id my_name display"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="your-project-id my_name delete"
```
* Here's an example run of `ResourceManagerExample`.
Be sure to change the placeholder project ID "your-project-id" with your own globally unique project ID.
```
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="create your-project-id"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="list"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="get your-project-id"
```
* Here's an example run of `StorageExample`.
Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine.
```
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="upload /path/to/test.txt <bucket_name>"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="list <bucket_name>"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="download <bucket_name> test.txt"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="delete <bucket_name> test.txt"
```
Troubleshooting
---------------
Expand Down
38 changes: 24 additions & 14 deletions gcloud-java-examples/src/main/java/com/google/gcloud/examples/DatastoreExample.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@
import com.google.gcloud.datastore.Key;
import com.google.gcloud.datastore.KeyFactory;
import com.google.gcloud.datastore.Query;
import com.google.gcloud.datastore.Query.ResultType;
import com.google.gcloud.datastore.QueryResults;
import com.google.gcloud.datastore.StructuredQuery;
import com.google.gcloud.datastore.StructuredQuery.PropertyFilter;
import com.google.gcloud.datastore.Transaction;

Expand Down Expand Up @@ -100,21 +100,31 @@ public void run(Transaction tx, Key userKey, String... args) {
return;
}
System.out.printf("User '%s' has %d comment[s].%n", userKey.name(), user.getLong("count"));
// ORDER BY timestamp";
String gql = "SELECT * FROM " + COMMENT_KIND + " WHERE __key__ HAS ANCESTOR @1";
Query<Entity> query = Query.gqlQueryBuilder(ResultType.ENTITY, gql)
.namespace(NAMESPACE)
.addBinding(userKey)
.build();
QueryResults<Entity> results = tx.run(query);
// We could have added "ORDER BY timestamp" to the query to avoid the sorting bellow
// but that would require adding an ancestor index for timestamp
// see: https://cloud.google.com/datastore/docs/tools/indexconfig
int limit = 200;
Map<DateTime, String> sortedComments = new TreeMap<>();
while (results.hasNext()) {
Entity result = results.next();
sortedComments.put(result.getDateTime("timestamp"), result.getString("content"));
StructuredQuery<Entity> query =
Query.entityQueryBuilder()
.namespace(NAMESPACE)
.kind(COMMENT_KIND)
.filter(PropertyFilter.hasAncestor(userKey))
.limit(limit)
.build();
while (true) {
QueryResults<Entity> results = tx.run(query);
int resultCount = 0;
while (results.hasNext()) {
Entity result = results.next();
sortedComments.put(result.getDateTime("timestamp"), result.getString("content"));
resultCount++;
}
if (resultCount < limit) {
break;
}
query = query.toBuilder().startCursor(results.cursorAfter()).build();
}
// We could have added "ORDER BY timestamp" to the query to avoid sorting, but that would
// require adding an ancestor index for timestamp.
// See: https://cloud.google.com/datastore/docs/tools/indexconfig
for (Map.Entry<DateTime, String> entry : sortedComments.entrySet()) {
System.out.printf("\t%s: %s%n", entry.getKey(), entry.getValue());
}
Expand Down

0 comments on commit d8bce91

Please sign in to comment.