Outlining the testing scenarios

[alexauroradev]

Let's collect the bridge testing scenarios that are absent at the moment, but we would like to implement.

@mfornet @abacabadabacaba

[alexauroradev]

Comment from @nearmax on the similar issue #411

This issue outlines all failure points of a bridge running in production. Some of these issues might have been addressed in the past iterations of the code, however, since our code is constantly changing we need to be aware of them in case we accidentally re-enable them.

Issues that happened in the past

1. Relay is stuck in a failure loop. When relay fails to perform RPC operation it typically retries certain number of times before failing the process. When process fails pm2 restarts it guaranteeing that relay cannot be stuck in an inconsistent state. However, if we messed up the code it can get stuck in this state for a long time (e.g. the num of retries constant is too high) or indefinitely;

2. EthToNearRelay running out of NEAR tokens. It is possible that NEAR account that EthToNearRelay is using to pay for gas has ran out of tokens. In general it should take weeks before NEAR account with 500 tokens is depleted, because we relay only one Ethereum header every 14 seconds. However, if there is a bug in the code such that when EthToNearRelay fails to submit a transaction it retries immediately and gets stuck in the loop of constant failures, the balance can be depleted fairly fast. This is a variation of (1);

3. NearToEthRelay running out of Ethereum tokens. Similar to (2), but we are running out of Ethereum tokens;

4. Web3js stuck in a state where it gets nonce wrong all the time. It is possible for Web3js to get stuck in a state where it fails to submit a transaction because it guesses the next available nonce wrong, it does not seem to be able to recover from it. This can get triggered when the account used by NearToEthRelay was used concurrently by something else to submit a transaction. The process should be able to crash when this happens repeatedly which resets web3js state, see (1);

5. near-api-js gets connection broken and never restored. It is possible for near-api-js to lose connection with the node and not being able to re-establish it. When this happens we should be able to rely on the process getting killed, see (1);

6. Upon restart relay picks up a wrong account address. When we are starting a relay we are passing it a "master account" through the command line flags. This address is then used to sign transactions and pay for gas, it is also written into the config. If someone later issues another command that defines a different "master account" they will overwrite it in the config. If relay is manually restarted later it might pick up a wrong master account from the config;

7. Infura runs out of quota. It is technically possible for relays to use up all of the Infura quota, if we are using free Infura account, or if we are paid Infura account but relay got stuck in a cycle where it rapidly queries Infura and runs out of quota. Relevant: (1);

8. Infura produces bad data after fork. Sometimes when there are forks, especially large ones, Infura might be either returning an outdated data before the fork, or it might be returning empty response;

9. Combination of JS libraries on top of Eth RPC produce missing or semi-missing data. It seems to be the case that sometimes something is returned by ETH RPC, passed to Web3js, then passed to eth-object library, etc and at the end in our code base we have an Ethereum header or proof represented as JS object but with some fields missing. I was not able to debug this yet;

10. Using non-full NEAR node. Near2EthRelay and the code that performs token transfers might have issues if they are communicating with NEAR node that has GC turned on. Near2EthRelay might fail to extract light header, while transfer code might fail to extract proofs of the events;

11. Using wrong capitalization for Eth addresses and secret keys. It seems like some Ethereum libraries expect specific capitalization of hex addresses and secret keys (this might be related to checksum or just to the code not being robust). When they are passed with wrong capitalization to bridge code we might be getting all kinds of weird errors or no errors at all;

12. Rounding NEAR timestamp. NEAR timestamps are not JSON-friendly and might be accidentally rounded when passed through JSON (probably because of u53 limit in JSON), e.g. when returned NEAR RPC. E.g. timestamp in the light header is represented twice in RPC output -- once as integer and once as string, so if we use precise timestamp we should be using the one coming from the string. Using wrong timestamp might result in certain validation checks to fail, e.g. NEAR light header being rejected by the NearOnEthClient.

13. pm2 misunderstands if the process is running. In some places in our code we are signaling pm2 that the service has started, if this is done incorrectly pm2 will not think that the process has started and it will time it out;

14. pm2 needs warmup. It seems that sometimes pm2 commands whether issues from CLI or through the library might silently fail because pm2 service has not started yet. It is important to always check first whether pm2 is running or force it to startup by running pm2 ping command. Failure to do so might result only in a subset of services starting;

15. eth2near-relay does not recover when there is temporarily less than 2/3 of validators. When NEAR testnet has <2/3 validators the RPC commands of eth2near-relay start timing out. When Testnet regains 2/3 of validators eth2near-relay does not continue operation and is just getting hung.

16. First transfer command is transitively failing. It seems like, when we transfer ERC20 token the first time to NEAR the transfer command can fail. It might be related to token factory performing a different operation on first transfer because it needs to deploy the mirror NEP21 token. CC @SkidanovAlex

Issues that might hypothetically happen

a. Go tool computed DAG roots incorrectly. If we have computed DAG roots incorrectly and initialized the EthOnNearClient contract with them we won't know until the next Ethereum epoch switch;

b. Gas threshold attack on NearOnEthClient. It might be possible to submit such malicious block to NearOnEthClient such the submission transaction will be under Ethereum gas limit but the transaction to challenge such block will be over the gas limit, making it non-challengable.