[EPIC] Implement loadInBlock and loadRelated and update dependencies

[dimitrovmaksim]

Progress:

Main scope

• Update graph-node dependencies - Implement loadRelated method #399
• implement loadInBlock - Implement loadInBlock method #398
• implement loadRelated - Implement loadRelated method #399 feat: Support derivedFrom with Interfaces #413
• Add function to log Entities with their derived relationships - Add logEntity function #405
• Update documentation - Update matchstick docs for 0.6.0 release graphprotocol/docs#519
  Chore/update readme demo-subgraph#47

Extended scope

• Implement File DataSource - Add ipfs data source #409
• Add custom message to asserts - feat: Add asserts with custom messages #408
• Test contract DataDources- feat: Make possible to test and log dynamic datasources #406

The Issue

Currently there are two new graph-node functionalities loadInBlock and loadRelated which are not implemented/mocked in Matchstick. This causes matchstick to panic and it prevents some users from properly testing their subgraphs.

loadInBlock

Description:

From my understanding loadInBlock tries to load an entity from some kind of a cache that persists for the time a specific block is being indexed. The entity will be present in the cache only if it was created or updated in that block.

For example event A occurs in block 123, in our handler we create an entity B with ID the address (0x0...1) of the emitter contract C. Later in that block another event A is emitted by contract C. If we try to load the entity using B.loadInBlock("0x0...1") we should get the entity we have created during the first occurrence of the event A.

But if in block 124, contract C has once again emitted event A and we try to load the entity using B.loadInBlock("0x0...1"), the function will return null because the entity is no more in the cache and instead we should try to load it from the database.

Matchstick implementation:

My proposed solution would be to:

1. Have the default loadInBlock behaviour to return null
2. Add a mock cashe store
3. Add a function that mocks the entity in the cache store with user defined
4. Add a function to remove/clean the cache store

loadRelated

Description:

loadRelated is a function that allows users to load derived entities into their handlers. This was previously not possible in the graph-node, but was possible with Matchstick due a bug in the graph-cli that generated getter (and setter) functions for those fields and the way the Matchstick's own entity store works. Derived fields are virtual fields and do not actually exists in the graph-node database. Instead they are created during query time and are only accessible when querying the subgraph. More info on derived fields https://thegraph.com/docs/en/developing/creating-a-subgraph/#reverse-lookups

Matchstick's own store

Matchstick does not work with a database, but implements its own key/value entity store. In the Matchstick store the derive fields actually exists and they keep the IDs of the child entities, hence why the getter/setter functions worked in Matchstick. While this was helpful for users to test their relations, it caused a lot of issues (especially with handling both String and Byte IDs) and made the code complex.
It will probably be wise to refactor the store and make it function similarly to the graph-node, i.e dynamically construct the derived fields when requested. We already keep the derived relations in a separate derived key/value store, but it may not be needed anymore, because the child entity's name, the parent id and the derivedFrom field should come from the loadRelated function itself, which should make it easier to find it in our entity store.

Matchstick implementation:

Entity Store:

Looking into the function loadRelated(entity: string, id: string, field: string): Array<Entity>; function we may be able to:

1. Simplify the matchstick store. - Currently we keep all relations into a separate derived mapping. But the function itself should provide us with the relation info we need in order to fetch the child entities from the entities store, so this mapping becomes obsolete as well as this whole module.
2. We will no longer need to track the relations into the store directly - the derived_fields module will probably become obsolete

logStore():

The potential change to the store will affect what the logStore() function prints to the console. There's several ways to approach it:

1. Dynamically construct the derived fields when executing logStore()
2. Add a logEntity(entityId) function that will only print the specified entity and it's relations
3. Add an optional boolean parameter to logStore() that if set to true will also construct and print the relations
4. Extend logStore() to accept a set of options {related: true, entityId: 123} that will combine 2. and 3.

Update Matchstick dependencies with the latest graph-node version:

Description:

Matchstick depends on some graph-node functionalities. It hasn't been updated for a while, so will need to do that in Cargo.toml. This may lead to issues that will need to be fixed

Resources:

Matchstick store and derived_fields implementation:

Entity store - https://github.com/LimeChain/matchstick/blob/fix-derived-with-byte-ids/src/context/mod.rs#L96
Derived relations store - https://github.com/LimeChain/matchstick/blob/fix-derived-with-byte-ids/src/context/mod.rs#L115
Initialising the relations store - https://github.com/LimeChain/matchstick/blob/fix-derived-with-byte-ids/src/context/derived_schema.rs
Save Entity to store - https://github.com/LimeChain/matchstick/blob/fix-derived-with-byte-ids/src/context/mod.rs#L503
Saving the derived fields 🤯 - https://github.com/LimeChain/matchstick/blob/fix-derived-with-byte-ids/src/context/derived_fields.rs
Get entity from store - https://github.com/LimeChain/matchstick/blob/fix-derived-with-byte-ids/src/context/mod.rs#L478
logStore - https://github.com/LimeChain/matchstick/blob/fix-derived-with-byte-ids/src/context/mod.rs#L181
clearStore - https://github.com/LimeChain/matchstick/blob/fix-derived-with-byte-ids/src/context/mod.rs#L190

Matchstick-as

https://github.com/LimeChain/matchstick-as

Matchstick docs

https://github.com/LimeChain/matchstick
https://thegraph.com/docs/en/developing/unit-testing-framework/

Demo-subgraph

https://github.com/LimeChain/demo-subgraph

[azf20]

@dimitrovmaksim thanks for this, I think Matchstick may also need updates to support File Data Sources?

[dimitrovmaksim]

@dimitrovmaksim thanks for this, I think Matchstick may also need updates to support File Data Sources?

Yeah, File Data Sources are not currently supported as well

[dimitrovmaksim]

Btw @azf20 just FYI, it seems some changes that were made to the graph-node in order to re-use some modules/functionalities in matchstick have been reverted/moved/removed, so updating the graph-node dependencies to the latest version in matchstick could require a PR to the graph-node as well.