Fix for "table dropped by concurrent query" on Redshift

[drewbanin]

• make integration tests run on Redshift
• create reproducible test case for failure
• update materialization logic

A test was added to reliably reproduce this error. Before the materialization logic was updated, the result looked like:

======================================================================
FAIL: test__redshift__concurrent_transaction_incremental (test_concurrent_transaction.TestConcurrentTransaction)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/dbt/test/integration/032_concurrent_transaction_test/test_concurrent_transaction.py", line 109, in test__redshift__concurrent_transaction_incremental
    self.run_test()
  File "/home/ubuntu/dbt/test/integration/032_concurrent_transaction_test/test_concurrent_transaction.py", line 85, in run_test
    self.assertEqual(self.query_state['model_1'], 'good')
AssertionError: 'error: table 3380474 dropped by concurrent transaction\n' != 'good'
-------------------- >> begin captured stdout << ---------------------

[drewbanin]

I had to make this change because dbt was previously adding a begin statement before the drop. Since the drop now happens after the transaction, there was no corresponding commit, and the transaction containing the drop was rolled back.

I checked the other places where dbt uses this method, and I believe this change should be safe, but I'm very open to improving how dbt handles transactions here

[cmcarthur]

there's one failure mode here that i'm concerned about, can you take a look at that?

looks like snowflake is passing 👍

[cmcarthur]

would this logic fail if the backup relation wasn't deleted on the previous run? i.e.:

run 1:

• dbt renames target to backup, commit succeeds, but then it fails to drop backup here
  run 2:
• ERROR: relation already exists?

i'd imagine you could fix this by also dropping the backup relation at the start of the run, if it exists.

[drewbanin]

really good point. We do something similar for the __dbt_tmp relation -- i will update!

[cmcarthur]

YESSSS

wrong status

[cmcarthur]

💯