fix: CTAS on multiple statements

[betodealmeida]

SUMMARY

The current behavior for running CTAS (create table as select) in SQL Lab is to check for each statement if it's a SELECT, and if so prepend the query with CREATE TABLE $table_name AS $query. For example, if we have this query:

SELECT 1 AS col;

And we run CTAS by passing the table name my_table we get this query:

CREATE TABLE my_table AS
SELECT 1 AS col;

The problem is that for a query with multiple statements we run CTAS for each one. For example:

SELECT 1 AS col;
SELECT 2 AS col;

When run with CTAS and my_table will produce these 2 queries:

CREATE TABLE my_table AS
SELECT 1 AS col;
CREATE TABLE my_table AS
SELECT 2 AS col;

The first query runs successfully, but the second fails because the table already exists.

This PR fixes the CTAS behavior, making it more consistent with how multiple statements work in SQL Lab. It changes SQL Lab so that:

1. A CTAS query can have multiple statements, as long as the last one is a SELECT. This allows user to pre-process the data before loading it into the table. A simple example in MySQL would be:

SET @value = 42;
SELECT @value AS foo;

When run in a CTAS, this will create a table with the column foo with a row with the value 42, as expected.

In Hive, a common pattern is to write queries like this:

INSERT OVERWRITE staging_table ...;
SELECT * FROM staging_table WHERE ...;

Which should also work.

2. A CVAS (create view as select) query, on the other hand, can have a single statement and it MUST be a SELECT. Before, there was no differentiation between CTAS and CVAS (other than the query generated).

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A

TEST PLAN

Tested manually, added unit tests.

ADDITIONAL INFORMATION

• Has associated issue:
• Changes UI
• Requires DB Migration.
• Confirm DB Migration upgrade and downgrade tested.
• Introduces new feature or API
• Removes existing feature or API

[codecov-io]

Codecov Report

Merging #12188 (06ebf9c) into master (a52031a) will decrease coverage by 3.10%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #12188      +/-   ##
==========================================
- Coverage   66.61%   63.51%   -3.11%     
==========================================
  Files         994      484     -510     
  Lines       49079    29771   -19308     
  Branches     4982        0    -4982     
==========================================
- Hits        32695    18908   -13787     
+ Misses      16254    10863    -5391     
+ Partials      130        0     -130     

Flag	Coverage Δ
cypress	?
javascript	?
python	63.51% <100.00%> (-0.72%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/sql_lab.py	80.37% <100.00%> (+0.75%)	⬆️
superset/sql_parse.py	99.36% <100.00%> (+0.02%)	⬆️
superset/sql_validators/postgres.py	50.00% <0.00%> (-50.00%)	⬇️
superset/views/database/views.py	62.50% <0.00%> (-25.07%)	⬇️
superset/db_engine_specs/mysql.py	79.59% <0.00%> (-12.25%)	⬇️
superset/databases/commands/create.py	83.67% <0.00%> (-8.17%)	⬇️
superset/databases/commands/update.py	85.71% <0.00%> (-8.17%)	⬇️
superset/db_engine_specs/presto.py	74.67% <0.00%> (-8.01%)	⬇️
superset/db_engine_specs/base.py	79.38% <0.00%> (-6.71%)	⬇️
superset/sql_validators/base.py	93.33% <0.00%> (-6.67%)	⬇️
... and 525 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a52031a...06ebf9c. Read the comment docs.

[villebro]

Thanks for a verbose description, made this very easy to review 👍 This will surely be appreciated by advanced SQL Lab users. Minor non-blocking comment, LGTM!

[villebro]

As the error messages imply a deep understanding of how is_valid_ctas() and is_valid_cvas() work, I wonder if these should be moved into ParsedQuery as something like assert_is_valid_ctas() and assert_is_valid_cvas() and have the error messages there?

[betodealmeida]

Ah, great point, @villebro! I'll move the exception message closer to the checks.

[amitmiran137]

What would be an example use case for using CTAS ?

[mistercrunch]

What would be an example use case for using CTAS ?

If you mean "why materialize a dataset?", the main reason is to create a summary and/or richer dataset to speed subsequent queries against this snapshot

If you mean "why using the Superset CTAS feature over simply writing your own CREATE TABLE {} AS" statement, it has to do with administrator/policy control. It's possible in Superset to NOT give DML access, but allow people to effectively do this in a controlled way, and even force a target location/naming scheme for the CTAS-generated table

[bkyryliuk]

+1 to this comment, we are actively leveraging it

What would be an example use case for using CTAS ?

If you mean "why materialize a dataset?", the main reason is to create a summary and/or richer dataset to speed subsequent queries against this snapshot

If you mean "why using the Superset CTAS feature over simply writing your own CREATE TABLE {} AS" statement, it has to do with administrator/policy control. It's possible in Superset to NOT give DML access, but allow people to effectively do this in a controlled way, and even force a target location/naming scheme for the CTAS-generated table