Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use pagination in DatastoreExample #582

Merged
merged 2 commits into from
Jan 26, 2016
Merged

Use pagination in DatastoreExample #582

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
830d87c
Use pagination in DatastoreExample and fix readme bug and formatting
Jan 26, 2016
360e048
Increase limit on query size
Jan 26, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 47 additions & 47 deletions gcloud-java-examples/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,54 +39,54 @@ To run examples from your command line:

4. Run an example using Maven from command line.

Here's an example run of `BigQueryExample`.

Before running the example, go to the [Google Developers Console][developers-console] to ensure
that BigQuery API is enabled. You can upload a CSV file `my_csv_file` to the `my_bucket` bucket
(replace `my_csv_file` and `my_bucket` with actual file and bucket names) using the GCS
[web browser](https://console.developers.google.com/storage/browser). The CSV file will be used to
load data into a BigQuery table and should look something like:
```csv
value1
value2
value3
```
Then you are ready to run the following example:
```
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create dataset new_dataset_id"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="list tables new_dataset_id"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'"
```

Here's an example run of `DatastoreExample`.
* Here's an example run of `BigQueryExample`.

Before running the example, go to the [Google Developers Console][developers-console] to ensure
that BigQuery API is enabled. You can upload a CSV file `my_csv_file` to the `my_bucket` bucket
(replace `my_csv_file` and `my_bucket` with actual file and bucket names) using the GCS
[web browser](https://console.developers.google.com/storage/browser). The CSV file will be used to
load data into a BigQuery table and should look something like:
```csv
value1
value2
value3
```
Then you are ready to run the following example:
```
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create dataset new_dataset_id"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="list tables new_dataset_id"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'"
```

* Here's an example run of `DatastoreExample`.

Note that you have to enable the Google Cloud Datastore API on the [Google Developers Console][developers-console] before running the following commands.
```
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="my_name add my\ comment"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="my_name display"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="my_name delete"
```

Here's an example run of `ResourceManagerExample`.

Be sure to change the placeholder project ID "my-project-id" with your own globally unique project ID.
```
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="create my-project-id"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="list"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="get my-project-id"
```

Here's an example run of `StorageExample`.

Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine.
```
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="upload /path/to/test.txt <bucket_name>"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="list <bucket_name>"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="download <bucket_name> test.txt"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="delete <bucket_name> test.txt"
```
Be sure to change the placeholder project ID "your-project-id" with your own project ID. Also note that you have to enable the Google Cloud Datastore API on the [Google Developers Console][developers-console] before running the following commands.
```
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="your-project-id my_name add my\ comment"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="your-project-id my_name display"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="your-project-id my_name delete"
```

* Here's an example run of `ResourceManagerExample`.

Be sure to change the placeholder project ID "your-project-id" with your own globally unique project ID.
```
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="create your-project-id"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="list"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="get your-project-id"
```

* Here's an example run of `StorageExample`.

Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine.
```
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="upload /path/to/test.txt <bucket_name>"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="list <bucket_name>"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="download <bucket_name> test.txt"
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="delete <bucket_name> test.txt"
```

Troubleshooting
---------------
Expand Down
38 changes: 24 additions & 14 deletions gcloud-java-examples/src/main/java/com/google/gcloud/examples/DatastoreExample.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@
import com.google.gcloud.datastore.Key;
import com.google.gcloud.datastore.KeyFactory;
import com.google.gcloud.datastore.Query;
import com.google.gcloud.datastore.Query.ResultType;
import com.google.gcloud.datastore.QueryResults;
import com.google.gcloud.datastore.StructuredQuery;
import com.google.gcloud.datastore.StructuredQuery.PropertyFilter;
import com.google.gcloud.datastore.Transaction;

Expand Down Expand Up @@ -100,21 +100,31 @@ public void run(Transaction tx, Key userKey, String... args) {
return;
}
System.out.printf("User '%s' has %d comment[s].%n", userKey.name(), user.getLong("count"));
// ORDER BY timestamp";
String gql = "SELECT * FROM " + COMMENT_KIND + " WHERE __key__ HAS ANCESTOR @1";
Query<Entity> query = Query.gqlQueryBuilder(ResultType.ENTITY, gql)
.namespace(NAMESPACE)
.addBinding(userKey)
.build();
QueryResults<Entity> results = tx.run(query);
// We could have added "ORDER BY timestamp" to the query to avoid the sorting bellow
// but that would require adding an ancestor index for timestamp
// see: https://cloud.google.com/datastore/docs/tools/indexconfig
int limit = 200;
Map<DateTime, String> sortedComments = new TreeMap<>();
while (results.hasNext()) {
Entity result = results.next();
sortedComments.put(result.getDateTime("timestamp"), result.getString("content"));
StructuredQuery<Entity> query =
Query.entityQueryBuilder()
.namespace(NAMESPACE)
.kind(COMMENT_KIND)
.filter(PropertyFilter.hasAncestor(userKey))
.limit(limit)
.build();
while (true) {
QueryResults<Entity> results = tx.run(query);
int resultCount = 0;
while (results.hasNext()) {
Entity result = results.next();
sortedComments.put(result.getDateTime("timestamp"), result.getString("content"));
resultCount++;
}
if (resultCount < limit) {
break;
}
query = query.toBuilder().startCursor(results.cursorAfter()).build();
}
// We could have added "ORDER BY timestamp" to the query to avoid sorting, but that would
// require adding an ancestor index for timestamp.
// See: https://cloud.google.com/datastore/docs/tools/indexconfig
for (Map.Entry<DateTime, String> entry : sortedComments.entrySet()) {
System.out.printf("\t%s: %s%n", entry.getKey(), entry.getValue());
}
Expand Down