Added health monitor

[niklabh]

Closes: #684

nodewatcher job, get the blockIndex in chain-db, then wait 10s and get it again, it should increment. 

is not done and service addition in k8 is needed

[Tbaut]

I'd like to have additional deeper queries to assess the situation better:

• it's not because hasura and auth are green, that the connection between the 2 is fine (LB problems etc). Can we add a hasura->auth check, that would query the last 10 posts with their author and name?
• same for hasura->chain-db can we query the last 10 posts with the onchain_link.referendum_id !== null and the onchain_referendum.delay.
• same for the nodewatcher-server->node-watcher-deployment:

query{
  referendums(last:10){
    delay
  }
}

^ this is the link that breaks regularly, and that nothing monitors as of now. This is not a replacement for the nodewatcher check (below), but an additionnal "link" check I'd say.

is not done and service addition in k8 is needed
Right, we don't have direct access to it.

[niklabh]

I'd like to have additional deeper queries to assess the situation better:

• it's not because hasura and auth are green, that the connection between the 2 is fine (LB problems etc). Can we add a hasura->auth check, that would query the last 10 posts with their author and name?
• same for hasura->chain-db can we query the last 10 posts with the onchain_link.referendum_id !== null and the onchain_referendum.delay.
• same for the nodewatcher-server->node-watcher-deployment:

query{
  referendums(last:10){
    delay
  }
}

^ this is the link that breaks regularly, and that nothing monitors as of now. This is not a replacement for the nodewatcher check (below), but an additionnal "link" check I'd say.

is not done and service addition in k8 is needed
Right, we don't have direct access to it.

Done

[Tbaut]

lgtm for the code, just a couple comments, testing it now.

[Tbaut]

I had a play with and thing look good, I think it'd be important to check with @fevo1971 to know better what is possible to do for his health monitor.

I don't think we can do too many crazy things. maybe he would expect something like a 200 when everything is fine, and a 5XX with a message of what goes wrong. Because the prisma version for instance won't help him. Also I've made a test with a chain-db and no referenda, I get a 200 although:

onchainLinkReferendumDelays | false
referendumDelays | false

This shouldn't be the case.

[niklabh]

onchainLinkReferendumDelays

Throwing errors

[Tbaut]

• Fix the bug on onchain_links being an array that made the health monitor constantly failing.
• Fixed to actually check the chain-db server. It could have failed without the health-check noticing.
• Upgrade polkadot/api (kept showing errors in the console)
• Added better check to avoid failing with id === 0
• Changed the emoji (not showing in bash)
• Added some logs to show clearly the used urls

[Tbaut]

Looks good, please have a look at my commits before merging.

[niklabh]

working nicely. Just need k8s setup now