A Modular & Extendable Subgraph Using Apollo Federations

[dOrgJelli]

Goals

1. Allow new DAOs to be launched without bloating the base subgraph (this repo).
2. Allow new Schemes & Constraints to be developed without burdening the base subgraph.
3. Be able to query data from multiple DAOs with a single graphql query.
4. Updates / fixes to base subgraph can easily be deployed, and all other dependents will updated / fixed too.

Required Reading

Apollo Federations
Gateways

Proposed Plan

1. Split the existing subgraph into two parts, base-subgraph and dao-subgraph.
   a. base-subgraph stores all entities that are universal to all DAOs. These would all have to come from predefined addresses (universal contracts).
   b. dao-subgraph stores all entities that are DAO specific. This project would take the form of a subgraph template, where the deployer would need to provide contract addresses for the contracts that should be watched (Avatar, Reputation, DAOToken, Controller, etc).
   c. Ideally base-subgraph & dao-subgraph have no overlapping contract addresses they're watching. Doing this would help satisfy Goal # 4. This will require refactoring the schema to further decouple things.
2. Each DAO will have its own subgraph. This subgraph will be an instantiation of the dao-base subgraph template talked about in 1b.
3. 3rd party schemes, constraints, and voting machines will run their own subgraphs.
4. All of these subgraphs will be queryable through a gateway. To learn more about how a gateway merges data from all of the graphql stores, please read the "Required Reading" above.
   a. use https://github.com/nautilus/gateway ?
   b. 3rd parties could ask to be added to DAOstack's gateway, or they could create their own.

[orenyodfat]

sound like a good direction.
can we use the "gateway" / Apollo Federation without support/update of the graph-node ?
any experience using the "gateway" ?

[jellegerbrandy]

i agree that we need to split up of the subgraph in "modules" along the lines of what you propose.

My worries are about the federation/gateway solution.

• I'm a bit worried about the https://github.com/nautilus/gateway code, seems to be just a single person that stopped working on it 4 months ago. Is there something more mature? How experimental is this stuff?
• worry about performance: If data is spread over many different databases, querying can be very slow. For a simple example, getting all DAOs with their names would require hitting all DAO databases seperately. Perhaps this can be somehow mitigated by caching, or clever definitions of grapql schemes in combination with clever distribution of data, but this is not trivial at all.
• Another worry is about possible difficulties ntegrating this with the graph. We now have graphql schemas that are generated automatically by the graph, and using federation means adding and/or changing those schemes.
• Let's be more clear about why we need this kind of solution. You talk about "bloating"and "burdening" in step 1 and 2, but I find this not very convincing. We are not talking here about large amounts of data. In the next year, at the very most a few 100 DAOs and maybe 10.000 users? A million vote/stake events would still be something that a postgres database can handle without much problem. Afaics, the main (only?) reason we might want to have independent subgraph deployments is because we want to be able to have anyone deploy a DAO easily, and this is hard to do in a single subgraph because it cannot index stuff "retroactively"...

[ben-kaufman]

According to the Graph team, it is not yet possible to integrate with Apollo Federation. I'm not sure what's the Gateway

This is a possible approach but I have another viewpoint I think might be good. So currently, the subgraph is basically an "Alchemy subgraph", it is very suited to the needs of Alchemy, but maybe not as much for other dapps. I think we should plan it as a Graph Node per dapp. What I mean by that is that we will have a basic subgraph which serves more like a library to be used by other subgraph implementations. So, for example, we will have an "Alchemy Subgraph" which uses a "DAOstack Subgraph" as a library and inherit from it. We will run this Alchemy subgraph on a graph node and manage it as needed. Other teams with different dapps will do basically the same.
I think the benefit of this is that it's more secure and consistent to have a graph per UI, and let the UI operator manage it.