Testing

Note: everyone's setup is unique. A problem's underlying cause may not be the same for one person as it is for another.

Phase 0: the basics

Do your subgraphs look healthy overall?

Find the IP address of your query node. If you did the k8 (Kubernetes) setup, use the IP address of the load balancer named query-node.

Then enter the following into your browser, replacing QUERY_NODE_IP with that IP address.

http://QUERY_NODE_IP/index-node/graphql/playground?query=%7B%0A%20%20indexingStatuses(subgraphs%3A%20%5B%22QmTXzATwNfgGVukV1fX2T6xw9f6LAYRVWpsdXyRWzUR2H9%22%2C%20%22Qme2hDXrkBpuXAYEuwGPAjr6zwiMZV4FHLLBa3BHzatBWx%22%2C%20%22QmXKwSEMirgWVn41nRzkT3hpUBw29cp619Gx58XW6mPhZP%22%2C%20%22QmbeDC4G8iPAUJ6tRBu99vwyYkaSiFwtXWKwwYkoNphV4X%22%5D)%20%7B%0A%20%20%20%20health%0A%20%20%20%20synced%0A%20%20%20%20subgraph%0A%20%20%20%20fatalError%20%7B%0A%20%20%20%20%20%20message%0A%20%20%20%20%7D%0A%20%20%20%20chains%20%7B%0A%20%20%20%20%20%20network%0A%20%20%20%20%20%20chainHeadBlock%20%7B%0A%20%20%20%20%20%20%20%20number%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20latestBlock%20%7B%0A%20%20%20%20%20%20%20%20number%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20lastHealthyBlock%20%7B%0A%20%20%20%20%20%20%20%20number%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D

You should see the following query loaded and ready to run:

{
  indexingStatuses(subgraphs: ["QmTXzATwNfgGVukV1fX2T6xw9f6LAYRVWpsdXyRWzUR2H9", "Qme2hDXrkBpuXAYEuwGPAjr6zwiMZV4FHLLBa3BHzatBWx", "QmXKwSEMirgWVn41nRzkT3hpUBw29cp619Gx58XW6mPhZP", "QmbeDC4G8iPAUJ6tRBu99vwyYkaSiFwtXWKwwYkoNphV4X"]) {
    health
    synced
    subgraph
    fatalError {
      message
    }
    chains {
      network
      chainHeadBlock {
        number
      }
      latestBlock {
        number
      }
      lastHealthyBlock {
        number
      }
    }
  }
}

Hit the Run button. For each subgraph, you should see output something like the following.

      {
        "chains": [
          {
            "chainHeadBlock": {
              "number": "10683953"
            },
            "lastHealthyBlock": null,
            "latestBlock": {
              "number": "10195912"
            },
            "network": "mainnet"
          }
        ],
        "fatalError": null,
        "health": "healthy",
        "subgraph": "Qme2hDXrkBpuXAYEuwGPAjr6zwiMZV4FHLLBa3BHzatBWx",
        "synced": false
      }

If healthy is false, fatalError should give you a clue about what the issue is. For example, if fatalError says "too many queries," it might mean that you've run out of quota with whichever Ethereum service you've been using.

If synced is false, that's ok for Phase 0. What's important is that the subgraph is progressing. I believe that you can confirm an unsynced subgraph is progressing by looking at the latestBlock number and rerunning the query. After some time - usually just 5-10 seconds in my case - that number should go up.

In the query result, there should be three sections like the one above - one section for each subgraph. If you don't see at least three, you are missing one or more of the subgraphs.

Do you have the correct subgraph versions deployed?

There exist different versions of each subgraph. You need to use not just the right subgraphs, but the right version of each subgraph.

uniswap/uniswap-v2: QmXKwSEMirgWVn41nRzkT3hpUBw29cp619Gx58XW6mPhZP
synthetixio-team/synthetix: Qme2hDXrkBpuXAYEuwGPAjr6zwiMZV4FHLLBa3BHzatBWx
molochventures/moloch: QmTXzATwNfgGVukV1fX2T6xw9f6LAYRVWpsdXyRWzUR2H9

You can answer this question using the GraphQL approach above, but here is another way.

http get <prometheus-url>/federate?match[]=subgraph_query_execution_time_count&match[]=subgraph_count&match[]=QmXKwSEMirgWVn41nRzkT3hpUBw29cp619Gx58XW6mPhZP_sync_total_secs&match[]=Qme2hDXrkBpuXAYEuwGPAjr6zwiMZV4FHLLBa3BHzatBWx_sync_total_secs&match[]=QmTXzATwNfgGVukV1fX2T6xw9f6LAYRVWpsdXyRWzUR2H9_sync_total_secs

The above query looks for the following metrics.

  // Query node metrics
  const queryNodeMetrics = ['subgraph_query_execution_time_count']

  // Index node metrics
  const indexNodeMetrics = [
    'subgraph_count',
    'QmXKwSEMirgWVn41nRzkT3hpUBw29cp619Gx58XW6mPhZP_sync_total_secs',
    'Qme2hDXrkBpuXAYEuwGPAjr6zwiMZV4FHLLBa3BHzatBWx_sync_total_secs',
    'QmTXzATwNfgGVukV1fX2T6xw9f6LAYRVWpsdXyRWzUR2H9_sync_total_secs',
  ]

Another way to test this out is via the Prometheus web UI. Navigate to the following URL in your browser.

<prometheus-url>/federate?match[]=subgraph_query_execution_time_count&match[]=subgraph_count&match[]=QmXKwSEMirgWVn41nRzkT3hpUBw29cp619Gx58XW6mPhZP_sync_total_secs&match[]=Qme2hDXrkBpuXAYEuwGPAjr6zwiMZV4FHLLBa3BHzatBWx_sync_total_secs&match[]=QmTXzATwNfgGVukV1fX2T6xw9f6LAYRVWpsdXyRWzUR2H9_sync_total_secs

If a metric is missing, you have the wrong version deployed of one or more subgraphs.

Has your subgraph made progress in the past 60 seconds?

Here is one example of a query you could use to confirm this from the command line.

http post <node>:8030/graphql query='{ indexingStatuses { subgraph node synced fatalError { message } health chains { ... on EthereumIndexingStatus { latestBlock { number } } } } }'

If you see no progress being made in the past 60 seconds, per @Jannis via Discord, there could be all kinds of reasons for it.

Here's what it means:

• There is a /subgraphs metadata endpoint that reports, among other things, all deployments, their data sources etc. It also (like the indexing status API that is by default served at :8030/graphql) reports the block that each subgraph has indexed up to.

• Even if there is no data for the subgraph in new blocks, the block number should advance with every block that is mined on chain. Some delay is expected, but generally, I'd expect this to happen roughly every seconds, so ~15s on mainnet.

• "The subgraph hasn't made progress in 60 seconds" means that during our test, this didn't happen. It suggests that the subgraph may have failed or be stuck at a specific block. You can check by sending e.g.