Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Count API #823

Merged
merged 67 commits into from
Oct 17, 2022
Merged
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
ce4c149
Add method in Datastore client to invoke rpc for aggregation query
jainsahab Aug 29, 2022
100115f
Creating count aggregation and using it to populate Aggregation proto
jainsahab Aug 29, 2022
c1956d8
Moving aggregation builder method to root level aggregation class
jainsahab Aug 30, 2022
493edfa
Introducing RecordQuery to represent queries which returns entity rec…
jainsahab Sep 7, 2022
9c7c4ba
Updating gitignore with patch extension
jainsahab Sep 7, 2022
cf49a37
Setting up structure of Aggregation query and its builder
jainsahab Sep 8, 2022
78fd6c9
Introducing ProtoPreparer to populate the request protos
jainsahab Sep 9, 2022
2ae957a
Delegating responsibility of preparing query proto to QueryPreparer
jainsahab Sep 9, 2022
14ac55f
Populating aggregation query with nested structured query
jainsahab Sep 9, 2022
3ff182f
Delegating responsibility of preparing query proto in GqlQuery to Que…
jainsahab Sep 9, 2022
7de95e6
Removing RecordQuery from the query hierarchy and making it a standal…
jainsahab Sep 12, 2022
d62964f
Populating aggregation query with nested gql query
jainsahab Sep 12, 2022
1f5b1a4
Removing deprecation warning by using assertThrows instead of Expecte…
jainsahab Sep 12, 2022
b69b166
Making DatastoreRpc call aggregation query method on client
jainsahab Sep 12, 2022
9589f5c
Creating response transformer to transform aggregation query response…
jainsahab Sep 13, 2022
2103b80
Implementing aggregation query executor to execute AggergationQuery
jainsahab Sep 13, 2022
775c341
Adding missing assertion statements
jainsahab Sep 13, 2022
8872a55
Creating RetryExecutor to inject it as a dependency in other components
jainsahab Sep 14, 2022
1dfafb7
Making RetryExecutor accept RetrySettings when creating it
jainsahab Sep 14, 2022
f6bf0ce
Revert "Making RetryExecutor accept RetrySettings when creating it"
jainsahab Sep 14, 2022
deb63bf
Revert "Creating RetryExecutor to inject it as a dependency in other …
jainsahab Sep 14, 2022
428e9e3
Introducing RetryAndTraceDatastoreRpcDecorator to have retry and trac…
jainsahab Sep 14, 2022
c86913e
Extracting out the responsibility of preparing ReadOption in it's own…
jainsahab Sep 14, 2022
b235935
Making QueryExecutor to execute query with provided ReadOptions
jainsahab Sep 14, 2022
f1c3379
Exposing readTime to the user
jainsahab Sep 14, 2022
168b864
Ignoring runAggregationQuery method from clirr check
jainsahab Sep 14, 2022
005429b
Making readTime final
jainsahab Sep 15, 2022
dfbdf60
Allowing namespace to be optional in AggregationQuery
jainsahab Sep 15, 2022
0565ba6
Add capability to fetch aggregation result by passing alias
jainsahab Sep 15, 2022
0e8e7b9
Implementing User facing datastore.runAggrgation method to run aggreg…
jainsahab Sep 15, 2022
fc0f0dd
Add integration test for count aggregation
jainsahab Sep 15, 2022
e6cbca6
Add transaction Id support in ReadOptionsProtoPreparer
jainsahab Sep 19, 2022
d205997
Supporting aggregation query with transactions
jainsahab Sep 19, 2022
5097cd6
Allowing user to create Aggregation directly without involving its bu…
jainsahab Sep 19, 2022
6508ff6
Preventing creating duplicated aggregation when creating an aggregati…
jainsahab Sep 19, 2022
5f1e419
Marking RecordQuery implemented method as InternalApi
jainsahab Sep 19, 2022
5e2fcb3
Writing comments and JavaDoc for aggregation query related class
jainsahab Sep 20, 2022
372a2d7
Adding a default implementation in the public interfaces to avoid com…
jainsahab Sep 22, 2022
31e0c5a
Merge branch 'main' into count_feature
jainsahab Sep 22, 2022
2c2cf63
covering a scenario to maintain consistent snapshot when executing ag…
jainsahab Sep 22, 2022
4719ae8
Creating emulator proxy to simulate AggregationQuery response from em…
jainsahab Sep 23, 2022
8edebda
Integration test to execute an aggregation query in a read only trans…
jainsahab Sep 27, 2022
ecdc04e
Merge branch 'main' into count_feature
jainsahab Sep 28, 2022
842df2d
Getting rid off limit operation on count aggregation as same behaviou…
jainsahab Sep 29, 2022
8c81b9f
Removing import statement from javadoc and undo changes in .gitignore…
jainsahab Sep 29, 2022
1584d9f
Using Optional instead of returning null from ReadOptionsProtoPreparer
jainsahab Sep 29, 2022
bc425dd
using assertThat from Truth library
jainsahab Sep 29, 2022
8479fd2
Merge branch 'main' into count_feature
jainsahab Oct 7, 2022
41df1bc
fixing unit test
jainsahab Oct 7, 2022
1606584
Getting rid off Double braces initialization syntax
jainsahab Oct 7, 2022
ca27e75
Fixing lint
jainsahab Oct 7, 2022
9ff91d9
Getting rid off emulator proxy and using easy mock to check the aggre…
jainsahab Oct 7, 2022
6953efe
Deleting a entity created locally in other test which is causing fail…
jainsahab Oct 7, 2022
d2e1969
Deleting all keys in datastore in integration test so that new test c…
jainsahab Oct 7, 2022
39812d4
Executing two read write transaction simultaneously and verifying the…
jainsahab Oct 7, 2022
2d9e81a
Removing tests to verify serializability as it's an underlying implem…
jainsahab Oct 7, 2022
90ab7cb
Fixing lint
jainsahab Oct 7, 2022
a48b323
Merge branch 'main' into count_feature
jainsahab Oct 10, 2022
aae2620
Adding runAggregationQuery method to reflect config so that it's avai…
jainsahab Oct 10, 2022
940e981
Fixing equals of CountAggregation
jainsahab Oct 10, 2022
512f753
Fixing lint
jainsahab Oct 10, 2022
89903d4
Adding an integration test of using limit option with aggregation query
jainsahab Oct 12, 2022
9b5adf1
Adding BetaApi annotation to public surface to indicate that aggregat…
jainsahab Oct 12, 2022
a0babe3
Fixing lint
jainsahab Oct 12, 2022
a1822a9
Merge branch 'main' into count_feature
jainsahab Oct 14, 2022
a5b4623
Removing unused functiona and fixing javadoc
jainsahab Oct 17, 2022
f4d8009
fixing variable name
jainsahab Oct 17, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,5 @@ target/
*.iml
__pycache__/

.flattened-pom.xml
.flattened-pom.xml
*.patch
jainsahab marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@
import com.google.datastore.v1.ReserveIdsResponse;
import com.google.datastore.v1.RollbackRequest;
import com.google.datastore.v1.RollbackResponse;
import com.google.datastore.v1.RunAggregationQueryRequest;
import com.google.datastore.v1.RunAggregationQueryResponse;
import com.google.datastore.v1.RunQueryRequest;
import com.google.datastore.v1.RunQueryResponse;
import com.google.rpc.Code;
Expand Down Expand Up @@ -120,4 +122,12 @@ public RunQueryResponse runQuery(RunQueryRequest request) throws DatastoreExcept
throw invalidResponseException("runQuery", exception);
}
}

public RunAggregationQueryResponse runAggregationQuery(RunAggregationQueryRequest request) throws DatastoreException {
try (InputStream is = remoteRpc.call("runAggregationQuery", request)) {
return RunAggregationQueryResponse.parseFrom(is);
} catch (IOException exception) {
throw invalidResponseException("runAggregationQuery", exception);
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@
import com.google.datastore.v1.ReserveIdsResponse;
import com.google.datastore.v1.RollbackRequest;
import com.google.datastore.v1.RollbackResponse;
import com.google.datastore.v1.RunAggregationQueryRequest;
import com.google.datastore.v1.RunAggregationQueryResponse;
import com.google.datastore.v1.RunQueryRequest;
import com.google.datastore.v1.RunQueryResponse;
import com.google.datastore.v1.client.testing.MockCredential;
Expand Down Expand Up @@ -336,6 +338,13 @@ public void runQuery() throws Exception {
expectRpc("runQuery", request.build(), response.build());
}

@Test
public void runAggregationQuery() throws Exception {
RunAggregationQueryRequest.Builder request = RunAggregationQueryRequest.newBuilder();
RunAggregationQueryResponse.Builder response = RunAggregationQueryResponse.newBuilder();
expectRpc("runAggregationQuery", request.build(), response.build());
}

private void expectRpc(String methodName, Message request, Message response) throws Exception {
Datastore datastore = factory.create(options.build());
MockDatastoreFactory mockClient = (MockDatastoreFactory) factory;
Expand Down
15 changes: 15 additions & 0 deletions google-cloud-datastore/clirr-ignored-differences.xml
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,19 @@
<method>com.google.datastore.v1.ReserveIdsResponse reserveIds(com.google.datastore.v1.ReserveIdsRequest)</method>
<differenceType>7012</differenceType>
</difference>
<difference>
kolea2 marked this conversation as resolved.
Show resolved Hide resolved
<className>com/google/cloud/datastore/spi/v1/DatastoreRpc</className>
<method>com.google.datastore.v1.RunAggregationQueryResponse runAggregationQuery(com.google.datastore.v1.RunAggregationQueryRequest)</method>
<differenceType>7012</differenceType>
</difference>
<difference>
<className>com/google/cloud/datastore/Datastore</className>
<method>com.google.cloud.datastore.AggregationResults runAggregation(com.google.cloud.datastore.AggregationQuery, com.google.cloud.datastore.ReadOption[])</method>
<differenceType>7012</differenceType>
</difference>
<difference>
<className>com/google/cloud/datastore/DatastoreReader</className>
<method>com.google.cloud.datastore.AggregationResults runAggregation(com.google.cloud.datastore.AggregationQuery)</method>
<differenceType>7012</differenceType>
</difference>
</differences>
6 changes: 6 additions & 0 deletions google-cloud-datastore/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,12 @@
<artifactId>easymock</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>mockwebserver</artifactId>
<version>4.10.0</version>
<scope>test</scope>
</dependency>
</dependencies>

<build>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
/*
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.google.cloud.datastore;

import static com.google.cloud.datastore.AggregationQuery.Mode.GQL;
import static com.google.cloud.datastore.AggregationQuery.Mode.STRUCTURED;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.base.Preconditions.checkNotNull;

import com.google.cloud.datastore.aggregation.Aggregation;
import com.google.cloud.datastore.aggregation.AggregationBuilder;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

/**
* An implementation of a Google Cloud Datastore Query that returns {@link AggregationResults}, It
* can be constructed by providing a nested query ({@link StructuredQuery} or {@link GqlQuery})
* to run the aggregations on and a set of {@link Aggregation}.

* <p>{@link StructuredQuery} example:</p>
* <pre>{@code
* import static com.google.cloud.datastore.aggregation.Aggregation.count;
*
* EntityQuery selectAllQuery = Query.newEntityQueryBuilder()
* .setKind("Task")
* .build();
* AggregationQuery aggregationQuery = Query.newAggregationQueryBuilder()
* .addAggregation(count().as("total_count"))
* .addAggregation(count().limit(100).as("count_upto_100"))
* .over(selectAllQuery)
* .build();
* AggregationResults aggregationResults = datastore.runAggregation(aggregationQuery);
* for (AggregationResult aggregationResult : aggregationResults) {
* System.out.println(aggregationResult.get("total_count"));
* System.out.println(aggregationResult.get("count_upto_100"));
* }
* }</pre>
*
* <h4>{@link GqlQuery} example:</h4>
* <pre>{@code
* import static com.google.cloud.datastore.aggregation.Aggregation.count;
*
* GqlQuery<?> selectAllGqlQuery = Query.newGqlQueryBuilder(
* "AGGREGATE COUNT(*) AS total_count, COUNT_UP_TO(100) AS count_upto_100 OVER(SELECT * FROM Task)"
* )
* .setAllowLiteral(true)
* .build();
* AggregationQuery aggregationQuery = Query.newAggregationQueryBuilder()
* .over(selectAllGqlQuery)
* .build();
* AggregationResults aggregationResults = datastore.runAggregation(aggregationQuery);
* for (AggregationResult aggregationResult : aggregationResults) {
* System.out.println(aggregationResult.get("total_count"));
* System.out.println(aggregationResult.get("count_upto_100"));
* }
* }</pre>
*
* @see <a href="https://cloud.google.com/appengine/docs/java/datastore/queries">Datastore
* queries</a>
*/
public class AggregationQuery extends Query<AggregationResults> {
jainsahab marked this conversation as resolved.
Show resolved Hide resolved

private Set<Aggregation> aggregations;
private StructuredQuery<?> nestedStructuredQuery;
private final Mode mode;
private GqlQuery<?> nestedGqlQuery;

AggregationQuery(String namespace, Set<Aggregation> aggregations,
StructuredQuery<?> nestedQuery) {
super(namespace);
checkArgument(!aggregations.isEmpty(),
"At least one aggregation is required for an aggregation query to run");
this.aggregations = aggregations;
this.nestedStructuredQuery = nestedQuery;
this.mode = STRUCTURED;
}

AggregationQuery(String namespace, GqlQuery<?> gqlQuery) {
super(namespace);
this.nestedGqlQuery = gqlQuery;
this.mode = GQL;
}

/** Returns the {@link Aggregation}(s) for this Query. */
public Set<Aggregation> getAggregations() {
kolea2 marked this conversation as resolved.
Show resolved Hide resolved
return aggregations;
}

/** Returns the underlying {@link StructuredQuery for this Query}.
* Returns null if created with {@link GqlQuery}
* */
public StructuredQuery<?> getNestedStructuredQuery() {
return nestedStructuredQuery;
}

/** Returns the underlying {@link GqlQuery for this Query}.
* Returns null if created with {@link StructuredQuery}
* */
public GqlQuery<?> getNestedGqlQuery() {
return nestedGqlQuery;
}

/** Returns the {@link Mode} for this query. */
public Mode getMode() {
return mode;
}

public static class Builder {
gforgurups marked this conversation as resolved.
Show resolved Hide resolved

private String namespace;
private Mode mode;
private final Set<Aggregation> aggregations;
private StructuredQuery<?> nestedStructuredQuery;
private GqlQuery<?> nestedGqlQuery;

public Builder() {
this.aggregations = new HashSet<>();
}

public Builder setNamespace(String namespace) {
this.namespace = namespace;
return this;
}

public Builder addAggregation(AggregationBuilder<?> aggregationBuilder) {
this.aggregations.add(aggregationBuilder.build());
return this;
}

public Builder addAggregation(Aggregation aggregation) {
this.aggregations.add(aggregation);
return this;
}

public Builder over(StructuredQuery<?> nestedQuery) {
this.nestedStructuredQuery = nestedQuery;
this.mode = STRUCTURED;
return this;
}

public Builder over(GqlQuery<?> nestedQuery) {
this.nestedGqlQuery = nestedQuery;
this.mode = GQL;
return this;
}

public AggregationQuery build() {
boolean nestedQueryProvided = nestedGqlQuery != null || nestedStructuredQuery != null;
checkArgument(nestedQueryProvided,
"Nested query is required for an aggregation query to run");

if (mode == GQL) {
return new AggregationQuery(namespace, nestedGqlQuery);
}
return new AggregationQuery(namespace, aggregations, nestedStructuredQuery);
}
}

public enum Mode {
STRUCTURED,
GQL,
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
/*
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.google.cloud.datastore;

import com.google.common.base.MoreObjects;
import com.google.common.base.MoreObjects.ToStringHelper;
import java.util.Iterator;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Objects;

/**
* Represents a result of an {@link AggregationQuery} query submission.
*/
public class AggregationResult {

private final Map<String, LongValue> properties;

public AggregationResult(Map<String, LongValue> properties) {
this.properties = properties;
}

/**
* Returns a result value for the given alias.
* @param alias A custom alias provided in the query or an autogenerated alias in the form of 'property_\d'
* @return An aggregation result value for the given alias.
*/
public Long get(String alias) {
return properties.get(alias).get();
}

@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
AggregationResult that = (AggregationResult) o;
return properties.equals(that.properties);
}

@Override
public int hashCode() {
return Objects.hash(properties);
}

@Override
public String toString() {
ToStringHelper toStringHelper = MoreObjects.toStringHelper(this);
for (Entry<String, LongValue> entry : properties.entrySet()) {
toStringHelper.add(entry.getKey(), entry.getValue().get());
}
return toStringHelper.toString();
}
}
Loading