feat: Add Resource for External Volumes #3106

jdoldis · 2024-09-27T19:46:35Z

This PR adds the resource for External Volumes. This is part of a series of PRs aimed at adding provider support for Iceberg tables.

References

pkg/resources/external_volume.go

pkg/resources/external_volume_acceptance_test.go

jdoldis · 2024-09-27T19:55:56Z

pkg/resources/external_volume.go

+			return diag.FromErr(err)
+		}
+
+		if withExternalChangesMarking {


As discussed on Slack, I’m not exactly sure how this works. It seems to be used for all Boolean values so I’ve copied it over and the tests are passing, would be good if you could check it.

it looks good, I don't know every detail, but this magic is mostly done to detect changes based on show_output field and seeing how it's changing and comparing it to the config/state you're currently having. It's a bit complicated, but Terraform SDKv2 is very limited when it comes to change detection. We tried a few solutions, and we ended up with this one (actually, we have a different approach for handling parameters, e.g. in the database, that utilizes Optional + Computed combo, you'll probably use it when implementing Iceberg Tables). Maybe @sfc-gh-asawicki would give more insight here.

jdoldis · 2024-09-27T19:57:42Z

pkg/resources/external_volume.go

+					Optional:    true,
+					Description: "Specifies the case-sensitive Amazon Resource Name (ARN) of the AWS identity and access management (IAM) role that grants privileges on the S3 bucket containing your data files.",
+				},
+				"storage_aws_external_id": {


I’ve followed the example of Storage Integration, which leaves this as a read only parameter. However, for both the Storage Integration and the External Volume I can see use cases in the future where users would want to set it. We should keep that in mind for the next iteration.

Hmm, I'm not sure if that parameter was settable when we were working on storage integration. If it's possible in Snowflake, then it should be possible here. But maybe as you say, we can leave it for the next iteration. cc: @sfc-gh-asawicki

pkg/acceptance/helpers/external_volume_client.go

jdoldis · 2024-09-27T20:03:41Z

Hey @sfc-gh-asawicki / @sfc-gh-jcieslak / @sfc-gh-jmichalak 🙂

I’ll be on holidays for the next two weeks so won’t be able to respond to the review until I’m back. If you have the time you are welcome to take the PR over yourself to make changes / get it merged, otherwise I’ll take a look once I’m back. I’ve checked Allow edits by maintainers, so you should be able to push here if you like.

Cheers,
James

pkg/resources/external_volume.go

jdoldis · 2024-09-29T12:11:25Z

pkg/resources/external_volume.go

+				}
+			}
+
+			temp_storage_location, err := CopyStorageLocationWithTempName(removedLocations[0])


It shouldn’t happen often but if deleting the temp resources fails in one apply, adding them on the next apply could fail due to duplicate names. If we want to avoid this, we could first run a check to see if the temp storage location already exists and run a remove operation.

Hmm, that's annoying. cc: @sfc-gh-asawicki is d.Partial essential in this case ?

sfc-gh-jcieslak · 2024-10-04T11:55:37Z

docs/resources/external_volume.md

+description: |-
+  Resource used to manage external volume objects. For more information, check external volume documentation https://docs.snowflake.com/en/sql-reference/commands-data-loading#external-volume.
+---
+


V1 candidate note is missing, a new template has to be created in the templates/resources directory for external volume (you can copy-paste the schema.md.tmpl one, but change the migration guide link header to the correct one).

Have added that without a migration guide link for now as I'm not sure when we'll get it released, will leave the comment as unresolved so we don't forget.

Actually, we don't want this note here. This object is not officially supported and requires a follow-up on our side. After this follow-up (also with an entry in the migration guide), we can add a note about the release candidate.

We can merge it now, but this note shouldn't be present in the next release.

pkg/acceptance/bettertestspoc/assert/resourceassert/external_volume_resource_ext.go

pkg/acceptance/helpers/external_volume_client.go

pkg/helpers/helpers.go

pkg/resources/external_volume_acceptance_test.go

pkg/resources/external_volume.go

sfc-gh-jmichalak

Thanks for your contribution 🚀

pkg/sdk/external_volumes_def.go

pkg/schemas/gen/sdk_show_result_structs.go

pkg/helpers/helpers.go

pkg/resources/testdata/TestAcc_ExternalVolume/azure/with-storage-aws-role-arn/test.tf

sfc-gh-jmichalak · 2024-10-14T13:26:11Z

pkg/sdk/testint/external_volumes_gen_integration_test.go

-				),
+		parsedExternalVolumeDescribed, err := helpers.ParseExternalVolumeDescribed(props)
+		require.NoError(t, err)
+		expectedParsedExternalVolumeDescribed := helpers.ParsedExternalVolumeDescribed{


We now have such assertions in the objectassert package, but I think we can move this on our side.

sfc-gh-jmichalak · 2024-10-14T13:30:22Z

pkg/resources/external_volume_acceptance_test.go

+	"github.com/hashicorp/terraform-plugin-testing/tfversion"
+)
+
+var (


I'm not sure which functions you mean here. If you mean getS3StorageLocation et al., then they can be left for now. I'd just change the names to sth like getS3StorageLocationConfigVariables.

pkg/resources/external_volume_acceptance_test.go

update acceptance tests to use this

…up alter volume calls

sfc-gh-jmichalak

Thanks for the follow-up! Please mark this PR as a ready for review, instead of a draft.

sfc-gh-jmichalak · 2024-10-22T08:33:31Z

pkg/acceptance/bettertestspoc/assert/resourceassert/external_volume_resource_ext.go

@@ -10,3 +11,23 @@ func (e *ExternalVolumeResourceAssert) HasStorageLocationLength(len int) *Extern
 	e.AddAssertion(assert.ValueSet("storage_location.#", strconv.FormatInt(int64(len), 10)))
 	return e
 }
+
+func (e *ExternalVolumeResourceAssert) HasStorageLocationAtIndex(


Nit: we can pass here a slice of a custom struct with these fields. But it's okay like this.

pkg/resources/external_volume.go

sfc-gh-jmichalak · 2024-10-22T09:03:30Z

pkg/resources/external_volume.go

-					}
-				}
-			}
-
 			temp_storage_location, err := CopyStorageLocationWithTempName(removedLocations[0])
 			if err != nil {
 				return diag.FromErr(err)


I think that CopyStorageLocationWithTempName is not suitable here - we can create an arbitrary storage location with our special name. We can leave it as is, but rename the function to copySentinelStorageLocation or something similar.

That's also an option - we could have a storage location with placeholder values for the base url, role arn etc. I think this would work as it works in the tests, so it seems Snowflake doesn't try to verify access to the underlying data.

I stuck with this though as not having storage locations with placeholder values seemed better long term, for example if Snowflake does implement verifications here in the future. I don't have a strong opinion though - happy to go with whatever you think is best.

I'll update the name of the function to CopySentinelStorageLocation for now

pkg/resources/helpers.go

sfc-gh-jmichalak · 2024-10-22T09:25:13Z

Examples are missing - please add them to examples/resources/external_volume (import.sh and resource.tf, check other resources like masking policy). After this, please run make pre-push which should regenerate docs.

…lumes_gen.go

jdoldis · 2024-10-23T09:42:19Z

pkg/helpers/helpers.go

+			parsedExternalVolumeDescribed.Active = p.Value
+		case p.Name == "ALLOW_WRITES":
+			parsedExternalVolumeDescribed.AllowWrites = p.Value
+		case strings.Contains(p.Name, "STORAGE_LOCATION_"):


One thing to check here - for azure storage locations the encryption_type will be set, although it's not set in the config. It's a special case where it's not a valid parameter, but is set on Snowflake side. This shouldn't cause any issues, as otherwise I think the tests would fail with unstable plans. But worth checking before merging.

sfc-gh-jmichalak · 2024-10-25T10:07:57Z

/ok-to-test sha=c4af02d4ee3a3b2213a95cd4946d3ad42dd747e9

github-actions · 2024-10-25T12:20:10Z

Integration tests failure for c4af02d4ee3a3b2213a95cd4946d3ad42dd747e9

sfc-gh-jmichalak

LGTM! The unresolved topics will be fixed on our side.

jdoldis · 2024-10-25T12:47:19Z

Awesome, thanks @sfc-gh-jmichalak !

## [0.98.0](v0.97.0...v0.98.0) (2024-11-08) Feature scope readiness for V1: [link](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/v1-preparations/ESSENTIAL_GA_OBJECTS.MD) ([Roadmap reference](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/ROADMAP.md#wrap-up-the-functional-scope)). :exclamation: Migration guide: [v0.97.0 -> v0.98.0](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/MIGRATION_GUIDE.md#v0970--v0980) ### 🎉 What's new - New resources: - authentication_policy ([#3098](#3098)), references [#2880](#2880) - external_volume ([#3106](#3106)), partially references [#2980](#2980) - stream_on_directory_table ([#3129](#3129)) - stream_on_view ([#3150](#3150)) - primary_connection, secondary_connection ([#3162](#3162)) - secret_with_basic_authentication, secret_with_generic_string, secret_with_oauth_authorization_code_grant, secret_with_oauth_client_credentials ([#3110](#3110)), ([#3141](#3141)) - New data sources: - connections ([#3155](#3155)), ([#3173](#3173)) - secrets ([#3131](#3131)) - Reworked: - provider configuration hierarchy ([#3166](#3166)), references [#1881](#1881), [#2145](#2145), [#2925](#2925), [#2983](#2983), [#3104](#3104) - provider configuration fields ([#3152](#3152)) streams data source ([#3151](#3151)) - SDK upgrades: - Upgrade tag SDK ([#3126](#3126)) - Recreate streams when they are stale ([#3129](#3129)) ### 🔧 Misc - Add object renaming research summary ([#3172](#3172)) - Test support for object renaming ([#3130](#3130)), ([#3147](#3147)), ([#3154](#3154)) - Add tests to issue [#3117](#3117) ([#3133](#3133)) - New roadmap entry ([#3158](#3158)) - Test more authentication methods ([#3178](#3178)) - Minor fixes ([#3174](#3174)) ### 🐛 Bug fixes - Apply various fixes ([#3176](#3176)), this addresses BCR 2024_08, references [#2717](#2717), [#3005](#3005), [#3125](#3125), [#3127](#3127), [#3153](#3153) - Connection and secret data sources tests ([#3177](#3177)) - Fix grant import docs ([#3183](#3183)), resolves [#3179](https://github.com/Snowflake-Labs/terraform-provider-snowflake/discussions/3179) - Fix user resource import ([#3181](#3181)) - Handle external type changes in stream resources ([#3164](#3164)) - Do not use OR REPLACE on initial creation in resources with copy_grants ([#3129](#3129)) - Address issue [#2201](#2201) by introducing new stream resources Co-authored-by: snowflake-release-please[bot] <105954990+snowflake-release-please[bot]@users.noreply.github.com>

jdoldis commented Sep 27, 2024

View reviewed changes

pkg/resources/external_volume.go Show resolved Hide resolved

jdoldis commented Sep 27, 2024

View reviewed changes

pkg/resources/external_volume_acceptance_test.go Show resolved Hide resolved

jdoldis commented Sep 27, 2024

View reviewed changes

pkg/resources/external_volume_acceptance_test.go Outdated Show resolved Hide resolved

jdoldis commented Sep 27, 2024

View reviewed changes

pkg/acceptance/helpers/external_volume_client.go Show resolved Hide resolved

feat: Add External Volume Resource

1704a50

jdoldis force-pushed the main branch from 31e7e95 to 1704a50 Compare September 29, 2024 09:46

jdoldis marked this pull request as draft September 29, 2024 11:18

Clean up and call out existing issue

afbd4a0

jdoldis force-pushed the main branch from c3417a2 to afbd4a0 Compare September 29, 2024 12:04

jdoldis commented Sep 29, 2024

View reviewed changes

pkg/resources/external_volume.go Outdated Show resolved Hide resolved

jdoldis commented Sep 29, 2024

View reviewed changes

sfc-gh-fbudzynski marked this pull request as ready for review October 4, 2024 10:27

sfc-gh-fbudzynski marked this pull request as draft October 4, 2024 10:30

sfc-gh-jcieslak requested changes Oct 7, 2024

View reviewed changes

sfc-gh-jmichalak requested changes Oct 14, 2024

View reviewed changes

jdoldis added 12 commits October 14, 2024 17:10

Review feedback part 1

fd4062e

Generate show output assertions and update acceptance tests to use these

f143a26

Add V1 candidate note

afe0235

Add custom external volumes assert for storage locations at index, and

2304ae4

update acceptance tests to use this

Update helpers file with changes suggested in review comments

1f190e6

Reduce number of test files and move list of valid storage providers

68c104d

Remove schema version, add defer for temp location cleanup and clean …

337f545

…up alter volume calls

Rename longestCommonPrefix and use DeepEqual for StorageLocationsEqual

4f89a9d

Order schema gen list alphabetically

2c09c7d

Use pre defined temp storage location name

15b8738

Remove defer as it's not currently working

90f2c88

Finish revert of defer change

6029e87

sfc-gh-jmichalak reviewed Oct 22, 2024

View reviewed changes

pkg/resources/helpers.go Outdated Show resolved Hide resolved

jdoldis added 2 commits October 22, 2024 14:21

Bring back SNOW-999142 TODO, rename function and fix typo

ff43f8a

Set FullyQualifiedNameAttributeName and move functions to external_vo…

8ac4af1

…lumes_gen.go

jdoldis commented Oct 23, 2024

View reviewed changes

sfc-gh-jmichalak added 3 commits October 25, 2024 11:51

Merge remote-tracking branch 'upstream/main'

184b06c

Resolve conflicts

b2836c7

Merge remote-tracking branch 'upstream/main'

c4af02d

sfc-gh-jmichalak marked this pull request as ready for review October 25, 2024 10:07

sfc-gh-jmichalak approved these changes Oct 25, 2024

View reviewed changes

sfc-gh-jmichalak enabled auto-merge (squash) October 25, 2024 12:42

sfc-gh-jmichalak disabled auto-merge October 25, 2024 12:42

sfc-gh-jmichalak merged commit 64ba674 into Snowflake-Labs:main Oct 25, 2024
7 of 8 checks passed

snowflake-release-please bot mentioned this pull request Oct 25, 2024

chore(main): release 0.98.0 #3137

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add Resource for External Volumes #3106

feat: Add Resource for External Volumes #3106

jdoldis commented Sep 27, 2024

jdoldis Sep 27, 2024

sfc-gh-jcieslak Oct 7, 2024

jdoldis Sep 27, 2024 •

edited

Loading

sfc-gh-jcieslak Oct 7, 2024

jdoldis commented Sep 27, 2024 •

edited

Loading

jdoldis Sep 29, 2024 •

edited

Loading

sfc-gh-jcieslak Oct 22, 2024

sfc-gh-jcieslak Oct 4, 2024

jdoldis Oct 14, 2024

sfc-gh-jmichalak Oct 25, 2024

sfc-gh-jmichalak left a comment

sfc-gh-jmichalak Oct 14, 2024

sfc-gh-jmichalak Oct 14, 2024

sfc-gh-jmichalak left a comment

sfc-gh-jmichalak Oct 22, 2024

sfc-gh-jmichalak Oct 22, 2024 •

edited

Loading

jdoldis Oct 22, 2024 •

edited

Loading

jdoldis Oct 22, 2024

sfc-gh-jmichalak commented Oct 22, 2024 •

edited

Loading

jdoldis Oct 23, 2024 •

edited

Loading

sfc-gh-jmichalak commented Oct 25, 2024

github-actions bot commented Oct 25, 2024

sfc-gh-jmichalak left a comment

jdoldis commented Oct 25, 2024

feat: Add Resource for External Volumes #3106

feat: Add Resource for External Volumes #3106

Conversation

jdoldis commented Sep 27, 2024

References

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdoldis Sep 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdoldis commented Sep 27, 2024 • edited Loading

jdoldis Sep 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-jmichalak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-jmichalak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-jmichalak Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

jdoldis Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-jmichalak commented Oct 22, 2024 • edited Loading

jdoldis Oct 23, 2024 • edited Loading

Choose a reason for hiding this comment

sfc-gh-jmichalak commented Oct 25, 2024

github-actions bot commented Oct 25, 2024

sfc-gh-jmichalak left a comment

Choose a reason for hiding this comment

jdoldis commented Oct 25, 2024

jdoldis Sep 27, 2024 •

edited

Loading

jdoldis commented Sep 27, 2024 •

edited

Loading

jdoldis Sep 29, 2024 •

edited

Loading

sfc-gh-jmichalak Oct 22, 2024 •

edited

Loading

jdoldis Oct 22, 2024 •

edited

Loading

sfc-gh-jmichalak commented Oct 22, 2024 •

edited

Loading

jdoldis Oct 23, 2024 •

edited

Loading