[ETL-676] Create data loading procedure for each parquet datatype

[philerooski]

For each of our Parquet data types, we create a stored procedure which can do a COPY INTO operation from the S3 location of that data type into a Snowflake table. Additionally, I've added an external stage for our development S3 bucket. We do not call these stored procedures as part of the deployment process. Instead, we will call the procedures using Tasks (future work).

• snowflake/objects/database/recover/schema/parquet/procedure/copy_into_table_from_stage.sql
  This is the stored procedure logic. This procedure will copy data from a specific stage (S3) location, but can copy into any Snowflake table (specified by the target_table parameter).
• snowflake/objects/database/recover/schema/parquet/procedure/deploy.sql
  This handles the logic of deploying the above procedure for each of our Parquet data types.
• snowflake/objects/database/recover/schema/parquet/deploy.sql
  This is our (relative) top-level deployment script which handles deployment for all the child objects under the Parquet schema. We've added some logic which will create the procedures. Depending on whether this is a dev or prod/staging (i.e., the branch is main) deployment, we will have the procedures reference the production or development stage.
• snowflake/objects/database/recover/schema/parquet/stage/*
  Everything under this directory is related to deploying the prod and dev stages.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[thomasyu888]

🔥 LGTM! The test seems to be failing for certain portions though.

[BryanFauble]

These changes look great!

Are there any instructions written up around how to run these items via CLI? I know during code spaces you were covering a few ways that running these were a bit different from passing arguments normally. I'm thinking about this from the perspective of "If I need to make a change, what is the best way for me to test the changes?"

[philerooski]

@BryanFauble

I know during code spaces you were covering a few ways that running these were a bit different from passing arguments normally. I'm thinking about this from the perspective of "If I need to make a change, what is the best way for me to test the changes?"

This is one area where I feel like Snowflake starts tripping over itself. The only way to invoke a SQL script locally (afaik) is with snow sql -f, but that requires Jinja variables to be formatted like &{ this }, whereas in an actual deployment you would use EXECUTE IMMEDIATE FROM statements to invoke the SQL, which uses {{ this }} format. And we can't just snow sql -q "EXECUTE IMMEDIATE FROM ..." to run local SQL files because Snowflake requires the top-level SQL script in your EXECUTE IMMEDIATE FROM chain to be in a stage -- which means we need to push the code to an internal Snowflake stage or push it to Git and use an external stage.

I've been developing without variables as much as possible, then amending my git commits whenever I'm ready to start using variables. This is pretty inconvenient. It might be easier to replace {{ this }} with &{ this } when developing and then switch the formatting when I'm ready to push to Git. I'm still trying to figure out the best way to go about development.

[philerooski]

@thomasyu888

The test seems to be failing for certain portions though.

Seems like it might have had something to do with Synapse being down for maintenance Sunday evening?

[BryanFauble]

@BryanFauble

I know during code spaces you were covering a few ways that running these were a bit different from passing arguments normally. I'm thinking about this from the perspective of "If I need to make a change, what is the best way for me to test the changes?"

This is one area where I feel like Snowflake starts tripping over itself. The only way to invoke a SQL script locally (afaik) is with snow sql -f, but that requires Jinja variables to be formatted like &{ this }, whereas in an actual deployment you would use EXECUTE IMMEDIATE FROM statements to invoke the SQL, which uses {{ this }} format. And we can't just snow sql -q "EXECUTE IMMEDIATE FROM ..." to run local SQL files because Snowflake requires the top-level SQL script in your EXECUTE IMMEDIATE FROM chain to be in a stage -- which means we need to push the code to an internal Snowflake stage or push it to Git and use an external stage.

I've been developing without variables as much as possible, then amending my git commits whenever I'm ready to start using variables. This is pretty inconvenient. It might be easier to replace {{ this }} with &{ this } when developing and then switch the formatting when I'm ready to push to Git. I'm still trying to figure out the best way to go about development.

@philerooski What I have been doing in the Terraform world is to just commit my changes and push to origin:
Sage-Bionetworks-Workflows/eks-stack@main...ibcdpe-935-vpc-updates

I'm then letting the CICD pipeline automatically run for me - And then I look at the results in the UI. I can do all of this locally, but this was more than sufficient for me.

Perhaps we would need to follow the same thing here where you push changes to origin and watch the github action that runs?

What this would prevent in your case is if a mistake is made when you "switch the formatting when I'm ready to push to Git"

[philerooski]

@BryanFauble That's the process I use now, letting the CI/CD do the deployment after I amend the commit and force push. A few (minor-ish) downsides:

1. My deployment method doesn't take into account unchanged "stacks". Everything needs to be deployed from scratch, which is a lot slower than you would hope if all you're trying to do is update a single object. So I end up manually commenting out time-consuming steps like snowflake/objects/.../table/deploy.sql.
2. RECOVER's CI/CD includes a lot of other time-consuming stuff like AWS deployments and tests -- which aren't blocking the Snowflake deployment in the CI/CD, but also aren't related and don't need to run.