EPIC: Post feature freeze testing

[evan-forbes]

We have been pushing many breaking changes lately to fit them in before feature freeze at the end of the month. After feature freeze, we will want to spend more time building out robust tests for mainnet. This includes but is not limited to:

• Test that we are only removing txs from the block at the max square size #1102
• Consolidate utility testing functions where possible #832
• Consolidate network testing code #829
• Fuzz tests in CI #695
• Integration test for IBC connections #1249
• Version compatibility testing #313
• Create testing handbook #1324
• Make timeouts dynamic celestia-core#939
• Add a testutil function that will fuzz random valid and invalid blocks #1516
• Integration-test re governance & feegrant module #1541

Stretch

• Testground network tests in CI #1258
• Implement end to end tests that attempt to propose invalid blocks #233
• testing: Add integration test for tx prioritization #513
• Unbounded blocksize network test celestia-core#945
• Recreate the manual testing of share encoding #630
• Add LastValsetBeforeNonce and other QGB RPC methods tests #1343
• Remove the requirement for e2e test users to run a local machine #1556

[MSevey]

Something that might be good to add to the stretch goals is thinking about a framework for testing version upgrades and compatibility.

This could start by just ensuring we have the ability to launch nodes on different versions, and upgrade or downgrade a node without restarting the test.

[evan-forbes]

yes! good idea @MSevey, I totally forgot about #313, added to stretch goals

[MSevey]

nice yea, honestly i think that test #313 might make sense to bump up to a required and tie it into release. Basically launch the tests with half the nodes on the current path version and half on the new patch version as a sanity check that nothing breaks.

Then a future test could be more involved in terms of more patch versions, minor versions, and major versions to really stress test the network and understand compatibility issues. This one would of course be more of a debugging test. There might be some specific upgrade scenarios that we would want to build into the pipeline, but this and those would definitely be stretchy stretch goals.

[Bidon15]

Thanks for the detail overview of the epic. It's Epic 😄
I would appreciate if we take our time to discuss how we approach this epic both from app and devops/testing teams

We can break down for example #1258 or compact blocks celestiaorg/celestia-core#883 to decide who does what so at the end the tests bring the most value and insights to core/app team
As well as we can learn how we can collaborate better for future joint issues

[cmwaters]

I think an overarching goal around our testing should be to put together some written artefact across the teams which can really act as a handbook making it clear for any team member or for anyone interested what we test, how we test it, when we test things and so forth. There is no monolithic one solution fits all kind of approach to testing. It really requires a myriad of different solutions that juggle the tradeoffs between different forms of coverage, "introspectability", time and resource costs. Thinking about testing the same systematic way we think about protocols and writing "specifications" will produce more reliable software. Having a place where people can read about everything will make it easier for us to cover our blindspots.

[cmwaters]

I also think @MSevey is totally spot on with his points. We should already be getting into the rhythm with upgrade and compatibility testing so it's like clockwork for us engineers by the time we hit our first mainnet upgrade. Previously at Tendermint, I had started working on extending the e2e framework for upgrade testing and perhaps we should endeavour to do the same here. We need to make sure this works alongside our versioning strategy. If we have long lived branches for our minor versions, we need to ensure that rolling upgrades happen smoothly for node operators. This is generally light-weight and thus could be tested on a per PR basis (in the same way you might detect protobuf breaking changes).

[MSevey]

I think an overarching goal around our testing should be to put together some written artefact across the teams which can really act as a handbook making it clear for any team member or for anyone interested what we test, how we test it, when we test things and so forth. There is no monolithic one solution fits all kind of approach to testing. It really requires a myriad of different solutions that juggle the tradeoffs between different forms of coverage, "introspectability", time and resource costs. Thinking about testing the same systematic way we think about protocols and writing "specifications" will produce more reliable software. Having a place where people can read about everything will make it easier for us to cover our blindspots.

One option to kick start this type of documentation could be a README in the test package that explains the different tooling and types of tests.

[evan-forbes]

we still have a dangling issues, but most have been ice-boxed