Queries for aggregations #5184

lutter · 2024-02-01T02:20:50Z

This PR enables querying of aggregations through the GraphQL interface. It required a major rework of the GraphQL query infrastructure as prefetch now needs to translate from types in the API schema to types in the InputSchema as the concrete types for the aggregations do not exist in the API schema.

Before, we relied a lot on the fact that the API schema contained the same types as the input schema with the exact same names and structure. But conceptually, prefetch and query building deals with types from the input schema as that describes how data is ultimately stored in the database.

To try this out, deploy the aggregation-example subgraph that builds aggregations over ethereum block numbers as those are easy to check.

incrypto32 · 2024-02-07T06:25:34Z

graph/src/schema/input_schema.rs

+    }
+
+    /// Return an `EntityType` that either references the object type `name`
+    /// or, if `name` refernces an aggregation, return the object type of


Suggested change

/// or, if `name` refernces an aggregation, return the object type of

/// or, if `name` references an aggregation, return the object type of

Small typo

incrypto32 · 2024-02-07T06:40:50Z

graphql/src/store/mod.rs

@@ -2,5 +2,5 @@ mod prefetch;
 mod query;
 mod resolver;

-pub use self::query::parse_subgraph_id;
+//pub use self::query::parse_subgraph_id;


Suggested change

//pub use self::query::parse_subgraph_id;

Woops .. forgot to remove that. Done

incrypto32 · 2024-02-08T07:02:24Z

@lutter i was unable to test the aggregation-example subgraph, im getting graph-cli error, how did you get over it?

Error: Error in schema.graphql:

  Block:
  - Field 'id': Entity ids must be of type Bytes! or String!

  BlockTime:
  - Field 'id': Entity ids must be of type Bytes! or String!

  Group1:
  - Defined without @entity directive
  - Field 'id': Entity ids must be of type Bytes! or String!

lutter · 2024-02-08T20:45:35Z

@lutter i was unable to test the aggregation-example subgraph, im getting graph-cli error, how did you get over it?

I totally forgot that I have this PR against graph-cli open and that I linked that in my local checkout. I just asked to have that merged and released to save people from that extra hoop. Would love to hear if you can get things working once that's out (or if you want to go through the whole trouble of linking a local checkout in)

incrypto32

Was able to test it out with the example subgraph with the CLI alpha version
0.68.0-alpha-20240208230340-9ea4027 LGTM

We use 'order by timestamp' on the raw data to make sure that first and last work correctly. But the name 'timestamp' was ambiguous as we select a timestamp and the data table has one. The selected timestamp is the same for all data points in the interval, whereas the timestamp from the data table is the block's timestamp, which is what we need to sort by. This change makes sure we sort by the timestamp from the data table as intended.

…tion We don't add it in the first place

Fixes #3574

Change the block time for test blocks to 45 minutes so that block 2 is in a different hour from block 0. In addition, make the query tests process block 2 with no changes. There's no other changes to the tests other than adapting them to the additional processed block.

Previously, we missed adding the filters etc. to query fields that were collections of aggregations

…ties

This required a major rework of the GraphQL query infrastructure as prefetch now needs to translate from types in the API schema to types in the InputSchema as the concrete types for the aggregations do not exist in the API schema. Before, we relied a lot on the fact that the API schema contained the same types as the input schema with the exact same names and structure. But conceptually, prefetch and query building deals with types from the input schema as that describes how data is ultimately stored in the database.

The meaning of that is very murky, but it changes API schema and has been there for a long time

lutter · 2024-02-14T00:49:45Z

I just finished testing queries against the hosted service. The biggest changes this introduces for existing queries are:

In some subgraphs, the names of collections have changed because of the fix described in this issue AFAICT, that affects three subgraphs on the hosted service.
Some constructs that we used to copy from the subgraph schema to the API schema are now removed. Those are the schema directive, type extensions, and definition of scalar and union types. I also wanted to drop input object definitions, but there's one subgraph that uses that to pass a where filter in as a variable. Instead of dropping them, I wanted to make that a validation error, but that turned out trickier than I thought
There's a change in the response for one query that uses the nonsensical ordering described in issue [Bug] Disallow sorting by child fields that are arrays #5207

lutter force-pushed the lutter/agg-schema branch from f20c521 to 0cdfd97 Compare February 1, 2024 16:39

fordN assigned lutter Feb 5, 2024

fordN requested review from zorancv and incrypto32 and removed request for zorancv February 5, 2024 16:46

lutter mentioned this pull request Feb 5, 2024

Declarative state & aggregations in Graph Node #3555

Open

5 tasks

incrypto32 approved these changes Feb 8, 2024

View reviewed changes

incrypto32 approved these changes Feb 12, 2024

View reviewed changes

lutter force-pushed the lutter/agg-schema branch 2 times, most recently from f018c57 to a8d204f Compare February 13, 2024 23:57

lutter added 18 commits February 13, 2024 15:58

graph: Pass the InputSchema to api_schema

cb6606e

graph: Rename output schema to 'api' in api_schema

29508cb

graph: Qualify all GraphQL parser types in api_schema with s::

aa0aff7

graph: Remove import of prelude::* from api.rs

b0b296b

graph: Use our internal types for API schema generation

3515c31

graph: Simplify what gets passed around for API schema generation

77fdf09

graph: No need to filter out _Schema_ any more from API schema genera…

35774c2

…tion We don't add it in the first place

graph: Handle non-pluralizable field names in the API schema

ebe91ad

Fixes #3574

graph: Refactor how we enumerate different kinds of types in InputSchema

a43a0be

graph: Simplify add_filter_type

45c30d3

graph: Simplify field_filter_input_values

1c02b6d

graph: Refactor how we copy from the input schema to the API schema

a29f793

graph: Add aggregation fields and filters to the API schema

d7475a0

store: Make it a easier to understand introspection test failures

e08f437

graph, store: Add an Aggregation_interval enum to meta.graphql

7b84fb3

graph: Change type of 'interval' field to an enum

3cc039b

graph: Simple test for aggregation API schema generation

94f5616

lutter added 12 commits February 13, 2024 15:58

graphql: Remove use graph::prelude::*

a1be80b

graph, runtime: Clarify a little how we classify types internally

1ab345e

graph: Properly treat references to aggregations as collections

e0c7f92

Previously, we missed adding the filters etc. to query fields that were collections of aggregations

all: Distinguish between declared typename and internal name for enti…

10d32a4

…ties

graphql: Remove unnecessary argument from build_query

3b3ebe5

graphql: Small simplification in build_query

5f152e4

graph: Add QueryExecutionError::ConstraintViolation

18f7d06

gaphql: Address review comments

80e10cf

graph: Allow interface fields to be derived

3be6d94

The meaning of that is very murky, but it changes API schema and has been there for a long time

graph: Do not produce errors for unsupported schema constructs

9c154cc

lutter force-pushed the lutter/agg-schema branch from a8d204f to 9c154cc Compare February 14, 2024 00:37

lutter merged commit 9c154cc into master Feb 14, 2024
7 checks passed

lutter deleted the lutter/agg-schema branch February 14, 2024 00:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Queries for aggregations #5184

Queries for aggregations #5184

lutter commented Feb 1, 2024

incrypto32 Feb 7, 2024

incrypto32 Feb 7, 2024

lutter Feb 8, 2024

incrypto32 commented Feb 8, 2024

lutter commented Feb 8, 2024

incrypto32 left a comment

lutter commented Feb 14, 2024 •

edited

Loading

	/// or, if `name` refernces an aggregation, return the object type of
	/// or, if `name` references an aggregation, return the object type of

Queries for aggregations #5184

Queries for aggregations #5184

Conversation

lutter commented Feb 1, 2024

incrypto32 Feb 7, 2024

Choose a reason for hiding this comment

incrypto32 Feb 7, 2024

Choose a reason for hiding this comment

lutter Feb 8, 2024

Choose a reason for hiding this comment

incrypto32 commented Feb 8, 2024

lutter commented Feb 8, 2024

incrypto32 left a comment

Choose a reason for hiding this comment

lutter commented Feb 14, 2024 • edited Loading

lutter commented Feb 14, 2024 •

edited

Loading