Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Destination Snowflake: Verify hostname via regex in spec #24615

Merged
merged 5 commits into from
Mar 28, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -355,7 +355,7 @@
- name: Snowflake
destinationDefinitionId: 424892c4-daac-4491-b35d-c6688ba547ba
dockerRepository: airbyte/destination-snowflake
dockerImageTag: 0.4.57
dockerImageTag: 0.4.58
documentationUrl: https://docs.airbyte.com/integrations/destinations/snowflake
icon: snowflake.svg
normalizationConfig:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6189,7 +6189,7 @@
supported_destination_sync_modes:
- "overwrite"
- "append"
- dockerImage: "airbyte/destination-snowflake:0.4.57"
- dockerImage: "airbyte/destination-snowflake:0.4.58"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/destinations/snowflake"
connectionSpecification:
Expand All @@ -6213,6 +6213,9 @@
- "accountname.snowflakecomputing.com"
type: "string"
title: "Host"
pattern: "^(http(s)?:\\/\\/)?([^./?#]+\\.)?([^./?#]+\\.)?([^./?#]+\\.)?([^./?#]+\\\
.snowflakecomputing\\.com)$"
pattern_descriptor: "{account_name}.snowflakecomputing.com or {accountname}.{aws_location}.aws.snowflakecomputing.com"
order: 0
role:
description: "Enter the <a href=\"https://docs.snowflake.com/en/user-guide/security-access-control-overview.html#roles\"\
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,5 +20,5 @@ RUN tar xf ${APPLICATION}.tar --strip-components=1

ENV ENABLE_SENTRY true

LABEL io.airbyte.version=0.4.57
LABEL io.airbyte.version=0.4.58
LABEL io.airbyte.name=airbyte/destination-snowflake
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
],
"type": "string",
"title": "Host",
"pattern": "^(http(s)?:\\/\\/)?([^./?#]+\\.)?([^./?#]+\\.)?([^./?#]+\\.)?([^./?#]+\\.snowflakecomputing\\.com)$",
"pattern_descriptor": "{account_name}.snowflakecomputing.com or {accountname}.{aws_location}.aws.snowflakecomputing.com",
"order": 0
},
"role": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@
import io.airbyte.commons.json.Jsons;
import io.airbyte.commons.resources.MoreResources;
import io.airbyte.integrations.destination.snowflake.SnowflakeDestination.DestinationType;
import io.airbyte.protocol.models.v0.ConnectorSpecification;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Stream;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
Expand All @@ -26,6 +29,47 @@ public class SnowflakeDestinationTest {

private static final ObjectMapper mapper = MoreMappers.initMapper();

private static Stream<Arguments> urlsDataProvider() {
return Stream.of(
// See https://docs.snowflake.com/en/user-guide/admin-account-identifier for specific requirements
// "Account name in organization" style
arguments("https://acme-marketing-test-account.snowflakecomputing.com", true),
arguments("https://acme-marketing_test_account.snowflakecomputing.com", true),
arguments("https://acme-marketing.test-account.snowflakecomputing.com", true),

// Legacy style (account locator in a region)
// Some examples taken from https://docs.snowflake.com/en/user-guide/admin-account-identifier#non-vps-account-locator-formats-by-cloud-platform-and-region
arguments("xy12345.snowflakecomputing.com", true),
arguments("xy12345.us-gov-west-1.aws.snowflakecomputing.com", true),
arguments("xy12345.us-east-1.aws.snowflakecomputing.com", true),
// And some other formats which are, de facto, valid
arguments("xy12345.foo.us-west-2.aws.snowflakecomputing.com", true),
arguments("https://xy12345.snowflakecomputing.com", true),
arguments("https://xy12345.us-east-1.snowflakecomputing.com", true),
arguments("https://xy12345.us-east-1.aws.snowflakecomputing.com", true),
arguments("https://xy12345.foo.us-west-2.aws.snowflakecomputing.com", true),

// Invalid formats
arguments("example.snowflakecomputing.com/path/to/resource", false),
arguments("example.snowflakecomputing.com:8080", false),
arguments("example.snowflakecomputing.com:12345", false),
arguments("example.snowflakecomputing.com//path/to/resource", false),
arguments("example.snowflakecomputing.com/path?query=string", false),
arguments("example.snowflakecomputing.com/#fragment", false),
arguments("ab12345.us-east-2.aws.snowflakecomputing. com", false),
arguments("ab12345.us-east-2.aws.snowflakecomputing..com", false));
}

@ParameterizedTest
@MethodSource({"urlsDataProvider"})
void testUrlPattern(final String url, final boolean isMatch) throws Exception {
final ConnectorSpecification spec = new SnowflakeDestination(OssCloudEnvVarConsts.AIRBYTE_OSS).spec();
final Pattern pattern = Pattern.compile(spec.getConnectionSpecification().get("properties").get("host").get("pattern").asText());

Matcher matcher = pattern.matcher(url);
assertEquals(isMatch, matcher.find());
}

@Test
@DisplayName("When given S3 credentials should use COPY")
public void useS3CopyStrategyTest() {
Expand Down
2 changes: 1 addition & 1 deletion connectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -333,7 +333,7 @@
| **S3 Glue** | <img alt="S3 Glue icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/s3-glue.svg" height="30" height="30"/> | Destination | airbyte/destination-s3-glue:0.1.3 | alpha | [link](https://docs.airbyte.com/integrations/destinations/s3-glue) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-s3-glue) | <small>`471e5cab-8ed1-49f3-ba11-79c687784737`</small> |
| **SFTP-JSON** | <img alt="SFTP-JSON icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/sftp.svg" height="30" height="30"/> | Destination | airbyte/destination-sftp-json:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/destinations/sftp-json) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-sftp-json) | <small>`e9810f61-4bab-46d2-bb22-edfc902e0644`</small> |
| **Scylla** | <img alt="Scylla icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/scylla.svg" height="30" height="30"/> | Destination | airbyte/destination-scylla:0.1.3 | alpha | [link](https://docs.airbyte.com/integrations/destinations/scylla) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-scylla) | <small>`3dc6f384-cd6b-4be3-ad16-a41450899bf0`</small> |
| **Snowflake** | <img alt="Snowflake icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/snowflake.svg" height="30" height="30"/> | Destination | airbyte/destination-snowflake:0.4.57 | generally_available | [link](https://docs.airbyte.com/integrations/destinations/snowflake) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-snowflake) | <small>`424892c4-daac-4491-b35d-c6688ba547ba`</small> |
| **Snowflake** | <img alt="Snowflake icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/snowflake.svg" height="30" height="30"/> | Destination | airbyte/destination-snowflake:0.4.58 | generally_available | [link](https://docs.airbyte.com/integrations/destinations/snowflake) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-snowflake) | <small>`424892c4-daac-4491-b35d-c6688ba547ba`</small> |
| **Streamr** | <img alt="Streamr icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/streamr.svg" height="30" height="30"/> | Destination | ghcr.io/devmate-cloud/streamr-airbyte-connectors:0.0.1 | alpha | [link](https://docs.airbyte.com/integrations/destinations/streamr) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/devmate-cloud) | <small>`eebd85cf-60b2-4af6-9ba0-edeca01437b0`</small> |
| **Teradata Vantage** | <img alt="Teradata Vantage icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/teradata.svg" height="30" height="30"/> | Destination | airbyte/destination-teradata:0.1.1 | alpha | [link](https://docs.airbyte.io/integrations/destinations/teradata) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-teradata) | <small>`58e6f9da-904e-11ed-a1eb-0242ac120002`</small> |
| **TiDB** | <img alt="TiDB icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/tidb.svg" height="30" height="30"/> | Destination | airbyte/destination-tidb:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/destinations/tidb) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-tidb) | <small>`06ec60c7-7468-45c0-91ac-174f6e1a788b`</small> |
Expand Down
6 changes: 3 additions & 3 deletions docs/integrations/destinations/snowflake.md
Original file line number Diff line number Diff line change
Expand Up @@ -275,7 +275,7 @@ Now that you have set up the Snowflake destination connector, check out the foll
## Troubleshooting

### 'Current role does not have permissions on the target schema'
If you receive an error stating `Current role does not have permissions on the target schema` make sure that the
If you receive an error stating `Current role does not have permissions on the target schema` make sure that the
Snowflake destination `SCHEMA` is one that the role you've provided has permissions on. When creating a connection,
it may allow you to select `Mirror source structure` for the `Destination namespace`, which if you have followed
some of our default examples and tutorials may result in the connection trying to write to a `PUBLIC` schema.
Expand All @@ -287,8 +287,8 @@ Otherwise, make sure to grant the role the required permissions in the desired n

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:-----------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------|
| 0.4.57 | 2023-03-23 | [#23904](https://github.com/airbytehq/airbyte/pull/24405) | Revert host validation change |
| 0.4.56 | 2023-03-22 | [#23904](https://github.com/airbytehq/airbyte/pull/23904) | Added host validation by pattern on UI |
| 0.4.58 | 2023-03-27 | [#24615](https://github.com/airbytehq/airbyte/pull/24615) | Fixed host validation by pattern on UI |
| 0.4.56 (broken) | 2023-03-22 | [#23904](https://github.com/airbytehq/airbyte/pull/23904) | Added host validation by pattern on UI |
| 0.4.54 | 2023-03-17 | [#23788](https://github.com/airbytehq/airbyte/pull/23788) | S3-Parquet: added handler to process null values in arrays |
| 0.4.53 | 2023-03-15 | [\#24058](https://github.com/airbytehq/airbyte/pull/24058) | added write attempt to internal staging Check method |
| 0.4.52 | 2023-03-10 | [\#23931](https://github.com/airbytehq/airbyte/pull/23931) | Added support for periodic buffer flush |
Expand Down