stricter schema validation: run all CAT tests locally

[erohmensing]

Assumptions/manual steps for this hacky version:

1. Give docker a THICK amount of storage space (potentially also RAM and CPU)

2. Build all images for connectors locally

3. Download all GSM credentials locally:
   Follow readme instructions, then

   VERSION=dev ci_credentials all write-to-storage
   

   works after bug fix

4. Build dev image of CAT

5. Hack acceptance-test-docker.sh to always use dev version of CAT and not build the dev image of the connector

6. Hack acceptance-test-docker.sh to always select the test I'm tesing
   ... run tests

7. Manually separate true positives, true negatives, false negatives, false positives

8. Check to make sure all connectors were tested (after running it a few times due to hanging tests)

• Only run on beta + GA connectors, since those are the ones we'll update anyway
  • ignore issues with alpha connectors such as missing credentials

Goals for current initial hacky version:

• Run acceptance-test.sh for all connectors
  • Handle tests that seem to hang forever (e.g. mixpanel takes forever)
  • Pick a level of concurrency that doesn't crash docker
  • skip alpha connectors
• Collect outputs and return codes
  • Save successes and failures separately
  • Separate true positives, true negatives, false negatives, false positives -> happening manually
    • selecting true negatives (fail due to additionalproperties) is automated
• Tests can run concurrently (async pool)
• Max concurrency can be set

[erohmensing]

Goals for actual version:

• Build dev version of CAT before running all tests
• Multiple options for running on all connectors. Both should be possible:
  • declare that connectors images are already built and to use them
  • ask the script to build the images before and delete them after (takes longer, but preserves precious disk space)
• Getting secrets automatically. One or many should be possible:
  • download all secrets before running all tests
  • download each secret before running each test
  • declare that secrets are already downloaded and don't need them
  • would we be interested in deleting them after, security-wise? LInk separate issue for ci_credentials improvements
• Ability to change which list of connectors should be tested (i.e. not ALL_DEFINITIONS)
  • a prelisted set of definitions (GA_DEFINITIONS)
  • a CLI list of connectors (maybe?)
  • a file that contains a list of connectors to test
• Ability to pass in a test to select for (to pass through to -k parameter of acceptance-test-docker.sh)

Deal with the various ways the test can fail:

See all test results categorized in this commit