Add namespace test in destination acceptance test #10793

tuliren · 2022-03-02T12:59:29Z

What

Close Create a well defined charset for namespaces and test them in DAT #9351.
The new versions are published in Publish new snowflake and bigquery versions #11274.

How

Add a new test in DAT.
Run a simple data sync to a variety of namespaces on the destination.
The test is disabled by default. It is currently only activated for Snowflake and BigQuery.

🚨 User Impact 🚨

None expected.

tuliren · 2022-03-02T13:31:32Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1922475141
❌ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1922475141
🐛

tuliren · 2022-03-02T13:31:39Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1922475558
❌ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1922475558
🐛

edgao

edgao · 2022-03-02T21:16:25Z

...te-integrations/bases/standard-destination-test/src/main/resources/namespace_test_cases.json

@@ -0,0 +1,51 @@
+[
+  {
+    "id": "s1-1",


nitpick: can these IDs be a little more descriptive (e.g. all_caps) to make them easier to cross-reference? right now if a test fails, it'll just log that case s1-1 failed, and then the developer would need to find this file to understand why.

Done. I also updated the exception message to ask people to look into the namespace_test_case.json for details.

tuliren · 2022-03-02T23:27:57Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1925195221
❌ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1925195221
🐛

tuliren · 2022-03-02T23:28:05Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1925195831
❌ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1925195831
🐛

We don't want to introduce changes that will affect existing connections for now.

tuliren · 2022-03-18T09:48:34Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2003407941
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2003407941
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     30    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1394    519    63%

tuliren · 2022-03-18T09:53:34Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2003428337
❌ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2003428337
🐛

tuliren · 2022-03-18T15:13:24Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2004904499
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2004904499
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     30    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1394    519    63%

tuliren · 2022-03-18T15:13:30Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2004904984
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2004904984
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     30    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1394    519    63%

tuliren · 2022-03-18T15:13:33Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2004905343
❌ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2004905343
🐛

tuliren · 2022-03-19T00:33:39Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2007062177
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2007062177
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     30    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1394    519    63%

tuliren · 2022-03-19T08:26:23Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2008100587
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2008100587
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     30    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1394    519    63%

tuliren · 2022-03-19T08:26:28Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2008100740
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2008100740
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     30    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1394    519    63%

tuliren · 2022-03-19T08:26:34Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2008101018
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2008101018
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     30    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1394    519    63%

tuliren added 2 commits March 2, 2022 04:54

Add namespace test for snowflake

2075116

Enable namespace test for bigquery

53e035b

github-actions bot added the area/connectors Connector related issues label Mar 2, 2022

Format code

72cbd76

tuliren temporarily deployed to more-secrets March 2, 2022 13:02 Inactive

octavia-squidington-iii temporarily deployed to more-secrets March 2, 2022 13:34 Inactive

tuliren requested a review from edgao March 2, 2022 17:17

edgao approved these changes Mar 2, 2022

View reviewed changes

tuliren added 2 commits March 2, 2022 15:26

Capitalize test case id

2292000

Update exception message to point to test case file

7c6fdc2

tuliren temporarily deployed to more-secrets March 2, 2022 23:28 Inactive

octavia-squidington-iii temporarily deployed to more-secrets March 2, 2022 23:30 Inactive

tuliren added 10 commits March 17, 2022 19:04

Merge branch 'master' into liren/destination-namespace-charset

668e0fe

Update snowflake name transformer to prepend underscore

82b4f58

Override convertStreamName instead of getIdentifier

e2d3506

Add missing state message

e46e0f2

Remove unused import

1a5e73d

Disable more namespace test cases

d7eda39

We don't want to introduce changes that will affect existing connections for now.

Dry method that mutates namespace

1435264

Pass through null

e698272

Normalize namespace

241455a

Fix test case

cc7da57

tuliren added 3 commits March 18, 2022 02:26

Bump version for snowflake and bigquery

019d64a

Enable namespace test for bigquery denormalized

c14e92d

Dry bigquery denormalized acceptance test

3f9111b

github-actions bot added the area/documentation Improvements or additions to documentation label Mar 18, 2022

Revert some of the variable scope change

e4be015

tuliren added 6 commits March 18, 2022 03:06

Fix unit test

6412b7b

Bump version

dcb8f98

Introduce getNamespace method

89f9f35

Implement getNamespace method for bigquery

bf159c0

Switch to getNamespace methods

b758324

Update comments

7b8708a

tuliren added 2 commits March 18, 2022 14:36

Fix bigquery denormalized acceptance test

7ca043d

Format code

9e93a4a

tuliren added 4 commits March 18, 2022 21:48

Dry bigquery destination test

c226dfc

Skip partition test for gcs mode

b9caead

Merge branch 'master' into liren/destination-namespace-charset

b4d789c

Bump version

c359284

tuliren merged commit 21ec23c into master Mar 20, 2022

tuliren deleted the liren/destination-namespace-charset branch March 20, 2022 00:47

tuliren mentioned this pull request Mar 20, 2022

Publish new snowflake and bigquery versions #11274

Merged

octavia-squidington-iii mentioned this pull request Mar 21, 2022

Bump Airbyte version from 0.35.58-alpha to 0.35.59-alpha #11298

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add namespace test in destination acceptance test #10793

Add namespace test in destination acceptance test #10793

tuliren commented Mar 2, 2022 •

edited

Loading

tuliren commented Mar 2, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 2, 2022 •

edited by github-actions bot

Loading

edgao left a comment

edgao Mar 2, 2022

tuliren Mar 2, 2022

tuliren commented Mar 2, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 2, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 18, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 18, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 18, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 18, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 18, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 19, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 19, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 19, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 19, 2022 •

edited by github-actions bot

Loading

Add namespace test in destination acceptance test #10793

Add namespace test in destination acceptance test #10793

Conversation

tuliren commented Mar 2, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

tuliren commented Mar 2, 2022 • edited by github-actions bot Loading

tuliren commented Mar 2, 2022 • edited by github-actions bot Loading

edgao left a comment

Choose a reason for hiding this comment

edgao Mar 2, 2022

Choose a reason for hiding this comment

tuliren Mar 2, 2022

Choose a reason for hiding this comment

tuliren commented Mar 2, 2022 • edited by github-actions bot Loading

tuliren commented Mar 2, 2022 • edited by github-actions bot Loading

tuliren commented Mar 18, 2022 • edited by github-actions bot Loading

tuliren commented Mar 18, 2022 • edited by github-actions bot Loading

tuliren commented Mar 18, 2022 • edited by github-actions bot Loading

tuliren commented Mar 18, 2022 • edited by github-actions bot Loading

tuliren commented Mar 18, 2022 • edited by github-actions bot Loading

tuliren commented Mar 19, 2022 • edited by github-actions bot Loading

tuliren commented Mar 19, 2022 • edited by github-actions bot Loading

tuliren commented Mar 19, 2022 • edited by github-actions bot Loading

tuliren commented Mar 19, 2022 • edited by github-actions bot Loading

tuliren commented Mar 2, 2022 •

edited

Loading

tuliren commented Mar 2, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 2, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 2, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 2, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 18, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 18, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 18, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 18, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 18, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 19, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 19, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 19, 2022 •

edited by github-actions bot

Loading

tuliren commented Mar 19, 2022 •

edited by github-actions bot

Loading