Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add new API Version to validate when setting fields not defined in the schema #4894

Merged
merged 7 commits into from
Oct 8, 2023
Merged

Add new API Version to validate when setting fields not defined in the schema #4894

merged 7 commits into from
Oct 8, 2023

Conversation

incrypto32
Copy link
Member

No description provided.

dependabot bot and others added 6 commits September 25, 2023 15:12
@dependabot
build(deps): bump chrono from 0.4.26 to 0.4.31 (#4876)
02c4328 
Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.26 to 0.4.31.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](chronotope/chrono@v0.4.26...v0.4.31)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
build(deps): bump webpki from 0.22.0 to 0.22.1 (#4857)
ce27a4d 
Bumps [webpki](https://github.com/briansmith/webpki) from 0.22.0 to 0.22.1.
- [Commits](https://github.com/briansmith/webpki/commits)

---
updated-dependencies:
- dependency-name: webpki
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@incrypto32
runtime: only include valid fields in entity for store_set
648dedd
@incrypto32
graph, runtime: add new apiVersion to validate fields not defined in …
7e132f0 
…the schema
@incrypto32
graph: update tests for setting invalid field
af14d42
@incrypto32
tests: add runner tests for undefined field setting validation in api…
2289fea 
…Version 0.0.8
@incrypto32 incrypto32 self-assigned this Sep 28, 2023
@incrypto32
Copy link
Member Author

@lutter this has conflicts after you merged the recent PR. If its ok to include that in the release I'll rebase ( i have rebased locally ).
Else we can merge this into the branch incrypto32/v0.33.0 after the review.

lutter
lutter approved these changes Sep 28, 2023
View reviewed changes
Copy link
Collaborator

@lutter lutter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

graph/src/data/subgraph/api_version.rs
@@ -15,6 +15,9 @@ pub const API_VERSION_0_0_6: Version = Version::new(0, 0, 6);
/// Enables event handlers to require transaction receipts in the runtime.
pub const API_VERSION_0_0_7: Version = Version::new(0, 0, 7);

/// Enables validation for fields that doesnt exist in the schema for an entity.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small typo: s/doesnt/don't/

graph/src/schema/input_schema.rs Outdated

match field {
Some(field) => self
.inner
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can just call self.has_field at this point; avoids duplicating logic

runtime/wasm/src/host_exports.rs Outdated
let entity = state
.entity_cache
.make_entity(data.into_iter().map(|(key, value)| (key, value)))
.make_entity(filtered_entity_data.map(|(key, value)| (key, value)))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That map is a noop (was one before, too) and can just be removed

runtime/wasm/src/host_exports.rs Outdated
invalid_fields
)));
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could the block starting at line 203 be put into a check_invalid_fields helper function? It would make store_set more readable.

@lutter
Copy link
Collaborator

lutter commented Sep 29, 2023

@lutter this has conflicts after you merged the recent PR. If its ok to include that in the release I'll rebase ( i have rebased locally ). Else we can merge this into the branch incrypto32/v0.33.0 after the review.

That's unfortunate - I think you'll have to put this onto a release branch, and then rebase it to merge it into master.

@incrypto32 incrypto32 changed the base branch from master to incrypto32/v0.33.0 October 7, 2023 10:11
@incrypto32 incrypto32 force-pushed the krishna/fake-field-validation branch from 9514579 to d25c639 Compare October 7, 2023 11:23
@incrypto32 incrypto32 merged commit b1502a4 into incrypto32/v0.33.0 Oct 8, 2023
5 of 7 checks passed
incrypto32 added a commit that referenced this pull request Oct 16, 2023
@incrypto32 @dependabot
Add new API Version to validate when setting fields not defined in th…
a8e3b72 
…e schema (#4894)

* build(deps): bump chrono from 0.4.26 to 0.4.31 (#4876)

Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.26 to 0.4.31.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](chronotope/chrono@v0.4.26...v0.4.31)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump webpki from 0.22.0 to 0.22.1 (#4857)

Bumps [webpki](https://github.com/briansmith/webpki) from 0.22.0 to 0.22.1.
- [Commits](https://github.com/briansmith/webpki/commits)

---
updated-dependencies:
- dependency-name: webpki
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* runtime: only include valid fields in entity for store_set

* graph, runtime: add new apiVersion to validate fields not defined in the schema

* graph: update tests for setting invalid field

* tests: add runner tests for undefined field setting validation in apiVersion 0.0.8

* graph: add check_invalid_fields method to HostExports

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
incrypto32 added a commit that referenced this pull request Oct 18, 2023
@incrypto32 @dependabot
Add new API Version to validate when setting fields not defined in th…
e12c79c 
…e schema (#4894)

* build(deps): bump chrono from 0.4.26 to 0.4.31 (#4876)

Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.26 to 0.4.31.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](chronotope/chrono@v0.4.26...v0.4.31)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump webpki from 0.22.0 to 0.22.1 (#4857)

Bumps [webpki](https://github.com/briansmith/webpki) from 0.22.0 to 0.22.1.
- [Commits](https://github.com/briansmith/webpki/commits)

---
updated-dependencies:
- dependency-name: webpki
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* runtime: only include valid fields in entity for store_set

* graph, runtime: add new apiVersion to validate fields not defined in the schema

* graph: update tests for setting invalid field

* tests: add runner tests for undefined field setting validation in apiVersion 0.0.8

* graph: add check_invalid_fields method to HostExports

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
incrypto32 added a commit that referenced this pull request Oct 23, 2023
@incrypto32 @dependabot
Add new API Version to validate when setting fields not defined in th…
a963f19 
…e schema (#4894)

* build(deps): bump chrono from 0.4.26 to 0.4.31 (#4876)

Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.26 to 0.4.31.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](chronotope/chrono@v0.4.26...v0.4.31)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump webpki from 0.22.0 to 0.22.1 (#4857)

Bumps [webpki](https://github.com/briansmith/webpki) from 0.22.0 to 0.22.1.
- [Commits](https://github.com/briansmith/webpki/commits)

---
updated-dependencies:
- dependency-name: webpki
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* runtime: only include valid fields in entity for store_set

* graph, runtime: add new apiVersion to validate fields not defined in the schema

* graph: update tests for setting invalid field

* tests: add runner tests for undefined field setting validation in apiVersion 0.0.8

* graph: add check_invalid_fields method to HostExports

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
incrypto32 added a commit that referenced this pull request Oct 25, 2023
@incrypto32 @dependabot
v0.33.0 (#4886)
246cde6 
* cargo: update workspace crates' version to v0.33.0

* update NEWS.md for v0.33.0

* Add new API Version to validate when setting fields not defined in the schema  (#4894)

* build(deps): bump chrono from 0.4.26 to 0.4.31 (#4876)

Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.26 to 0.4.31.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](chronotope/chrono@v0.4.26...v0.4.31)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump webpki from 0.22.0 to 0.22.1 (#4857)

Bumps [webpki](https://github.com/briansmith/webpki) from 0.22.0 to 0.22.1.
- [Commits](https://github.com/briansmith/webpki/commits)

---
updated-dependencies:
- dependency-name: webpki
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* runtime: only include valid fields in entity for store_set

* graph, runtime: add new apiVersion to validate fields not defined in the schema

* graph: update tests for setting invalid field

* tests: add runner tests for undefined field setting validation in apiVersion 0.0.8

* graph: add check_invalid_fields method to HostExports

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update NEWS.md

* tests: add .gitignore for api-version test

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
zorancv pushed a commit that referenced this pull request Nov 13, 2023
@incrypto32 @dependabot @zorancv
v0.33.0 (#4886)
6c5ab00 
* cargo: update workspace crates' version to v0.33.0

* update NEWS.md for v0.33.0

* Add new API Version to validate when setting fields not defined in the schema  (#4894)

* build(deps): bump chrono from 0.4.26 to 0.4.31 (#4876)

Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.26 to 0.4.31.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](chronotope/chrono@v0.4.26...v0.4.31)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump webpki from 0.22.0 to 0.22.1 (#4857)

Bumps [webpki](https://github.com/briansmith/webpki) from 0.22.0 to 0.22.1.
- [Commits](https://github.com/briansmith/webpki/commits)

---
updated-dependencies:
- dependency-name: webpki
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* runtime: only include valid fields in entity for store_set

* graph, runtime: add new apiVersion to validate fields not defined in the schema

* graph: update tests for setting invalid field

* tests: add runner tests for undefined field setting validation in apiVersion 0.0.8

* graph: add check_invalid_fields method to HostExports

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update NEWS.md

* tests: add .gitignore for api-version test

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
zorancv added a commit that referenced this pull request Nov 13, 2023
@zorancv @incrypto32
@dependabot @saihaj @lutter @leoyvens @maoueh
Log the progress of substreams (#4935)
79703ba 
* Update protobuf structure

* Log progress message once every 30 sec.

* superfluous trace_id

* better formating

* spelling

* humane representation of time

* concise logs

* factor out the log string code

* move loging code to other file

* don't allow multiple trace_id in logs

* proper handling of trace_id

* v0.33.0 (#4886)

* cargo: update workspace crates' version to v0.33.0

* update NEWS.md for v0.33.0

* Add new API Version to validate when setting fields not defined in the schema  (#4894)

* build(deps): bump chrono from 0.4.26 to 0.4.31 (#4876)

Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.26 to 0.4.31.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](chronotope/chrono@v0.4.26...v0.4.31)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump webpki from 0.22.0 to 0.22.1 (#4857)

Bumps [webpki](https://github.com/briansmith/webpki) from 0.22.0 to 0.22.1.
- [Commits](https://github.com/briansmith/webpki/commits)

---
updated-dependencies:
- dependency-name: webpki
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* runtime: only include valid fields in entity for store_set

* graph, runtime: add new apiVersion to validate fields not defined in the schema

* graph: update tests for setting invalid field

* tests: add runner tests for undefined field setting validation in apiVersion 0.0.8

* graph: add check_invalid_fields method to HostExports

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update NEWS.md

* tests: add .gitignore for api-version test

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* docker: upgrade cloudbuild machineType

* fix(graphql): change CDN to JS Deliver for GraphiQL (#4941)

* fix(graphql): change CDN to JS Deliver for GraphiQL

* fix(graphql): add crossorigin prop

* build(deps): bump toml from 0.7.6 to 0.8.4

Bumps [toml](https://github.com/toml-rs/toml) from 0.7.6 to 0.8.4.
- [Commits](toml-rs/toml@toml-v0.7.6...toml-v0.8.4)

---
updated-dependencies:
- dependency-name: toml
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump semver from 1.0.18 to 1.0.20

Bumps [semver](https://github.com/dtolnay/semver) from 1.0.18 to 1.0.20.
- [Release notes](https://github.com/dtolnay/semver/releases)
- [Commits](dtolnay/semver@1.0.18...1.0.20)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* docs: Update information about `graphman config check`

* docs: Add a guide for sharding

* tests: Update runner-tests package.json

* fix(tests): yarn workspace to install all subfolders

* ci: update to node 20

* fix(graphql): add crossorigin prop (#4948)

* Fix issues in runner tests (#4962)

* tests: Don't compare WASM backtraces

While it would be nice to test them in isolation, here it is just
making these high-level tests too sensitive to unrelated changes.

* tests: Test PoI as hex and update to current value

This changed probably because graph-cli and -ts were updated.

* tests: Make `data_source_revert` more robust to code updates

Previously it would break whenever the graft base subgraph id would change,
now that id is updated automatically.

* Update tests/runner-tests/data-source-revert/package.json

Co-authored-by: Saihajpreet Singh <saihajpreet.singh@gmail.com>

---------

Co-authored-by: Saihajpreet Singh <saihajpreet.singh@gmail.com>

* store/test-store: test more types for DataSource context in manifest

* server/index-node : lazy load features from manifest when a subgraph is not deployed

* [Feature] Add support for `endBlock` in data sources (#4787)

* graph,chain,store/test-store : Allow new param `endBlock` in manifest

* core,graph,store: ignore end_block reached datasources in match_and_decode, include them in processed datasources

* tests : add runner tests for end-block

* core: move TriggerFilter construction into SubgraphRunner.run_inner

* core: filter out endBlock reached subgraphs when constructing TriggerFilter

* chain,core: refactor endBlock implementation

* refactor `SubgraphRunner.run_inner` to extract `build_filter`

* core : handle reverts for endBlock

* chain,graph: set min_spec_version requirements for endBlock

* core: refaction `build_filter`

* tests: runner test for endblock on reorg

* core: restart block stream in the next block for endblock reached ds

* graph: bump specVersion requirement for endBlock

* core: refactor build_filter logic

* core, tests, graph : make TriggerFilters testable

* chain/startknet: endBlock support for starknet

* chain,core,graph: refactor end_block implementation

* core: refactor build_filter

* Add comments for end-block runner tests

* graph, runtime, chain: Add GasMetrics for DIPS experiments

* graph: refactor GasCounter for trackiing gas metrics, add a new env for gas metrics

* tests: Revamp the integration tests

With these changes, we do not use truffler/ganache anymore. Tests are now
written as normal Rust tests. See `tests/README.md` for details

* runner tests: Do not clobber api-version tests

Two different tests are run in the same directory; make sure that the files
they generate and use do not clobber each other.

* workflows: Do not compile with --verbose for runner tests

* graph, store: Track connection wait time in the trace

There are other places during query execution where we might be waiting for
connections, but this at least records the times for getting the actual
data.

* graph, store: Track permit wait times in query traces

* fix: Increase RPC base backoff (#4984)

* fix: Increase RPC base backoff

* fix: Add 'stack underflow' to deterministic geth errors

* Make `SubstreamsBlockIngestor` start at chain's head if no cursor already exist (#4951)

Fixes #4942

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Zoran Cvetkov <zoran@edgeandnode.com>
Co-authored-by: Krishnanand V P <44740264+incrypto32@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: incrypto32 <vpkrishnanand@gmail.com>
Co-authored-by: Saihajpreet Singh <saihajpreet.singh@gmail.com>
Co-authored-by: David Lutterkort <lutter@watzmann.net>
Co-authored-by: Leonardo Yvens <leoyvens@gmail.com>
Co-authored-by: Matthieu Vachon <matt@streamingfast.io>
YaroShkvorets added a commit to YaroShkvorets/graph-node that referenced this pull request Nov 17, 2023
@YaroShkvorets
Reset substreams/firehose block ingestor backoff
7c15c72 
fixes graphprotocol#4976

graphql: Include introspection fields in error (graphprotocol#4968)

Include `__schema` and `__type` part of the result when there is an indexing error and the error policy is deny.

build(deps): bump proc-macro2 from 1.0.66 to 1.0.69

Bumps [proc-macro2](https://github.com/dtolnay/proc-macro2) from 1.0.66 to 1.0.69.
- [Release notes](https://github.com/dtolnay/proc-macro2/releases)
- [Commits](dtolnay/proc-macro2@1.0.66...1.0.69)

---
updated-dependencies:
- dependency-name: proc-macro2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

store/test-store: test more types for DataSource context in manifest

server/index-node : lazy load features from manifest when a subgraph is not deployed

[Feature] Add support for `endBlock` in data sources (graphprotocol#4787)

* graph,chain,store/test-store : Allow new param `endBlock` in manifest

* core,graph,store: ignore end_block reached datasources in match_and_decode, include them in processed datasources

* tests : add runner tests for end-block

* core: move TriggerFilter construction into SubgraphRunner.run_inner

* core: filter out endBlock reached subgraphs when constructing TriggerFilter

* chain,core: refactor endBlock implementation

* refactor `SubgraphRunner.run_inner` to extract `build_filter`

* core : handle reverts for endBlock

* chain,graph: set min_spec_version requirements for endBlock

* core: refaction `build_filter`

* tests: runner test for endblock on reorg

* core: restart block stream in the next block for endblock reached ds

* graph: bump specVersion requirement for endBlock

* core: refactor build_filter logic

* core, tests, graph : make TriggerFilters testable

* chain/startknet: endBlock support for starknet

* chain,core,graph: refactor end_block implementation

* core: refactor build_filter

* Add comments for end-block runner tests

graph, runtime, chain: Add GasMetrics for DIPS experiments

graph: refactor GasCounter for trackiing gas metrics, add a new env for gas metrics

tests: Revamp the integration tests

With these changes, we do not use truffler/ganache anymore. Tests are now
written as normal Rust tests. See `tests/README.md` for details

runner tests: Do not clobber api-version tests

Two different tests are run in the same directory; make sure that the files
they generate and use do not clobber each other.

workflows: Do not compile with --verbose for runner tests

graph, store: Track connection wait time in the trace

There are other places during query execution where we might be waiting for
connections, but this at least records the times for getting the actual
data.

graph, store: Track permit wait times in query traces

fix: Increase RPC base backoff (graphprotocol#4984)

* fix: Increase RPC base backoff

* fix: Add 'stack underflow' to deterministic geth errors

Make `SubstreamsBlockIngestor` start at chain's head if no cursor already exist (graphprotocol#4951)

Fixes graphprotocol#4942

Log the progress of substreams (graphprotocol#4935)

* Update protobuf structure

* Log progress message once every 30 sec.

* superfluous trace_id

* better formating

* spelling

* humane representation of time

* concise logs

* factor out the log string code

* move loging code to other file

* don't allow multiple trace_id in logs

* proper handling of trace_id

* v0.33.0 (graphprotocol#4886)

* cargo: update workspace crates' version to v0.33.0

* update NEWS.md for v0.33.0

* Add new API Version to validate when setting fields not defined in the schema  (graphprotocol#4894)

* build(deps): bump chrono from 0.4.26 to 0.4.31 (graphprotocol#4876)

Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.26 to 0.4.31.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](chronotope/chrono@v0.4.26...v0.4.31)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump webpki from 0.22.0 to 0.22.1 (graphprotocol#4857)

Bumps [webpki](https://github.com/briansmith/webpki) from 0.22.0 to 0.22.1.
- [Commits](https://github.com/briansmith/webpki/commits)

---
updated-dependencies:
- dependency-name: webpki
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* runtime: only include valid fields in entity for store_set

* graph, runtime: add new apiVersion to validate fields not defined in the schema

* graph: update tests for setting invalid field

* tests: add runner tests for undefined field setting validation in apiVersion 0.0.8

* graph: add check_invalid_fields method to HostExports

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update NEWS.md

* tests: add .gitignore for api-version test

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* docker: upgrade cloudbuild machineType

* fix(graphql): change CDN to JS Deliver for GraphiQL (graphprotocol#4941)

* fix(graphql): change CDN to JS Deliver for GraphiQL

* fix(graphql): add crossorigin prop

* build(deps): bump toml from 0.7.6 to 0.8.4

Bumps [toml](https://github.com/toml-rs/toml) from 0.7.6 to 0.8.4.
- [Commits](toml-rs/toml@toml-v0.7.6...toml-v0.8.4)

---
updated-dependencies:
- dependency-name: toml
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump semver from 1.0.18 to 1.0.20

Bumps [semver](https://github.com/dtolnay/semver) from 1.0.18 to 1.0.20.
- [Release notes](https://github.com/dtolnay/semver/releases)
- [Commits](dtolnay/semver@1.0.18...1.0.20)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* docs: Update information about `graphman config check`

* docs: Add a guide for sharding

* tests: Update runner-tests package.json

* fix(tests): yarn workspace to install all subfolders

* ci: update to node 20

* fix(graphql): add crossorigin prop (graphprotocol#4948)

* Fix issues in runner tests (graphprotocol#4962)

* tests: Don't compare WASM backtraces

While it would be nice to test them in isolation, here it is just
making these high-level tests too sensitive to unrelated changes.

* tests: Test PoI as hex and update to current value

This changed probably because graph-cli and -ts were updated.

* tests: Make `data_source_revert` more robust to code updates

Previously it would break whenever the graft base subgraph id would change,
now that id is updated automatically.

* Update tests/runner-tests/data-source-revert/package.json

Co-authored-by: Saihajpreet Singh <saihajpreet.singh@gmail.com>

---------

Co-authored-by: Saihajpreet Singh <saihajpreet.singh@gmail.com>

* store/test-store: test more types for DataSource context in manifest

* server/index-node : lazy load features from manifest when a subgraph is not deployed

* [Feature] Add support for `endBlock` in data sources (graphprotocol#4787)

* graph,chain,store/test-store : Allow new param `endBlock` in manifest

* core,graph,store: ignore end_block reached datasources in match_and_decode, include them in processed datasources

* tests : add runner tests for end-block

* core: move TriggerFilter construction into SubgraphRunner.run_inner

* core: filter out endBlock reached subgraphs when constructing TriggerFilter

* chain,core: refactor endBlock implementation

* refactor `SubgraphRunner.run_inner` to extract `build_filter`

* core : handle reverts for endBlock

* chain,graph: set min_spec_version requirements for endBlock

* core: refaction `build_filter`

* tests: runner test for endblock on reorg

* core: restart block stream in the next block for endblock reached ds

* graph: bump specVersion requirement for endBlock

* core: refactor build_filter logic

* core, tests, graph : make TriggerFilters testable

* chain/startknet: endBlock support for starknet

* chain,core,graph: refactor end_block implementation

* core: refactor build_filter

* Add comments for end-block runner tests

* graph, runtime, chain: Add GasMetrics for DIPS experiments

* graph: refactor GasCounter for trackiing gas metrics, add a new env for gas metrics

* tests: Revamp the integration tests

With these changes, we do not use truffler/ganache anymore. Tests are now
written as normal Rust tests. See `tests/README.md` for details

* runner tests: Do not clobber api-version tests

Two different tests are run in the same directory; make sure that the files
they generate and use do not clobber each other.

* workflows: Do not compile with --verbose for runner tests

* graph, store: Track connection wait time in the trace

There are other places during query execution where we might be waiting for
connections, but this at least records the times for getting the actual
data.

* graph, store: Track permit wait times in query traces

* fix: Increase RPC base backoff (graphprotocol#4984)

* fix: Increase RPC base backoff

* fix: Add 'stack underflow' to deterministic geth errors

* Make `SubstreamsBlockIngestor` start at chain's head if no cursor already exist (graphprotocol#4951)

Fixes graphprotocol#4942

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Zoran Cvetkov <zoran@edgeandnode.com>
Co-authored-by: Krishnanand V P <44740264+incrypto32@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: incrypto32 <vpkrishnanand@gmail.com>
Co-authored-by: Saihajpreet Singh <saihajpreet.singh@gmail.com>
Co-authored-by: David Lutterkort <lutter@watzmann.net>
Co-authored-by: Leonardo Yvens <leoyvens@gmail.com>
Co-authored-by: Matthieu Vachon <matt@streamingfast.io>

store: fix graphman not accepting table names

runner tests: Tag all log lines with the test name

runner tests: Wait longer for subgraphs to reach stop block

Also, make it clearer when a subgraph doesn't reach the stop block in the
allotted time.

graph: Qualify uses of graphql parser types with s::

graph: Move InputSchema validations over from Schema

These validations only makes sense for input schemas, not generic GraphQL
schemas

graph, store: Remove InputSchema::new

graph: Inline InputSchema::create into InputSchema::parse

all: Fix test schemas

Since validation of InputSchema isn't mandatory, many of the schemas we use
in our tests actually failed validation. This commit addresses that.

graph: Make it impossible to construct an unvalidated InputSchema

graph: Validate the id field of object types

This hadn't been done before, though in practice it was probably never
violated (presumably since graph-cli enforces such rules, and because
trying to deploy such a schema would end badly)

The `examples/validate.rs` is a tool that makes it possible to bulk
validate subgraph schemas.

Add more labels to `deployment_sync_secs` (graphprotocol#4965)

* initial try

* small fix

* add shart parameter in tests

* fmt

---------

Co-authored-by: Zoran Cvetkov <zoran@edgeandnode.com>

tests: Make how long runner tests wait for sync configurable

Set it to 10 minutes for github actions. Default is 1 minute

build(deps): bump protobuf-parse from 3.2.0 to 3.3.0

Bumps [protobuf-parse](https://github.com/stepancheg/rust-protobuf) from 3.2.0 to 3.3.0.
- [Changelog](https://github.com/stepancheg/rust-protobuf/blob/master/CHANGELOG.md)
- [Commits](stepancheg/rust-protobuf@v3.2.0...v3.3.0)

---
updated-dependencies:
- dependency-name: protobuf-parse
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

build(deps): bump sha2 from 0.10.7 to 0.10.8

Bumps [sha2](https://github.com/RustCrypto/hashes) from 0.10.7 to 0.10.8.
- [Commits](RustCrypto/hashes@sha2-v0.10.7...sha2-v0.10.8)

---
updated-dependencies:
- dependency-name: sha2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

docs: fix typos

build(deps): bump atomic_refcell from 0.1.12 to 0.1.13

Bumps [atomic_refcell](https://github.com/bholley/atomic_refcell) from 0.1.12 to 0.1.13.
- [Commits](https://github.com/bholley/atomic_refcell/commits)

---
updated-dependencies:
- dependency-name: atomic_refcell
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

build(deps): bump termcolor from 1.2.0 to 1.4.0

Bumps [termcolor](https://github.com/BurntSushi/termcolor) from 1.2.0 to 1.4.0.
- [Commits](BurntSushi/termcolor@1.2.0...1.4.0)

---
updated-dependencies:
- dependency-name: termcolor
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

docs: Fix typos

fix typos
@incrypto32 incrypto32 deleted the krishna/fake-field-validation branch March 12, 2024 08:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lutter lutter lutter approved these changes

Assignees

@incrypto32 incrypto32

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@incrypto32 @lutter