Feature/able to retry all connections

[gregingenii]

resolves #193

Description

Adds a 'retry_all' configuration boolean to override the behaviour that only 'retryable' connection errors are attempted again.

Help with which tests need extending or how this can be tested is greatly appreciated.

Checklist

• [/] I have signed the CLA
• [/] I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• [/] I have updated the CHANGELOG.md and added information about my change to the "dbt next" section.

[jtcohen6]

Thanks for the PR @gregingenii! This looks like a clever extension of the workaround we put in place, long ago, to handle long cluster startup times over HTTP.

Could you say just a bit more about:

• The specific connection issue you see transiently
• Which connection method you're using when you see it
• How long/how well this fix has been working for you locally

I don't have a great sense of how we'd want to add a test for this, since I'm not sure we can reliably reproduce a transient connection issue. The code change itself looks quite straightforward, so I'm leaning toward merging it as is.

[gregingenii]

Thanks for coming back @jtcohen6 . Answers to your questions:

• The error we were seeing was the ever helpful and explanatory 'Database Error':

2021-07-19 14:57:47.225853 (MainThread): 
2021-07-19 14:57:47.225957 (MainThread): [Errno 0] Error
2021-07-19 14:57:47.226050 (MainThread): 
2021-07-19 14:57:47.226140 (MainThread): Runtime Error in test source_not_null_<test name> (models/<model>/schema.yml)
2021-07-19 14:57:47.226223 (MainThread):   Database Error
2021-07-19 14:57:47.226359 (MainThread):     failed to connect
2021-07-19 14:57:47.226446 (MainThread): 
2021-07-19 14:57:47.226534 (MainThread): Runtime Error in test source_not_null_<test name>_ (models/<model>/schema.yml)
2021-07-19 14:57:47.226614 (MainThread):   Database Error
2021-07-19 14:57:47.226686 (MainThread):     failed to connect
2021-07-19 14:57:47.226762 (MainThread): 
2021-07-19 14:57:47.226843 (MainThread): [Errno 0] Error

• We're connecting with http
• This fix is working well. We've been running with it for a couple of months now with no issues.

If there isn't already a testing framework for connection issues to add on to, I would like to duck implementing one if that's okay. I can also see that a couple of conflicts have arisen so I'll resolve those.

Thank you for your help.

[jtcohen6]

Appreciate that context @gregingenii. I just tried this out locally by setting host to 0.0.0.0, it worked great.

The only slight pause I have: For the traditional retry (because a cluster is spinning up), it made sense to have a connect_timeout of 30 seconds or so. In terms of handling query transience, I imagine you'd want this to be more like 5 seconds. I just wanted to mention—I don't think it's a big deal, and I'd much rather reuse the existing parameter that serves the same basic purpose.

Two tiny comments, then this is good to go.

[jtcohen6]

@gregingenii Thanks for this contribution!