🎉 Base-normalization: Implement `normalization` for `MSSQL-destination` #6079

bazarnov · 2021-09-15T12:30:02Z

What

#3430 - Implement normalization for the MSSql Destination.

How

NORMALIZATION MSSQL

edited macros for dbt_utils: array.sql, concat.sql, datatypes.sql, json_operations.sql, surrogate_key.sql, type_conversions.sql
added hash.sql macros for dbt_utils, for MS SQL Server specifically
added MS SQL Server compatibility for modules:
- dbt_integration_test.py, test_ephemeral.py, test_normalization.py
- /transform_catalog/destination_name_transformer.py
- /transform_catalog/reserver_keywords.py
- /transform_catalog/stream_processor.py
- /transform_catalog/destination_type.py
- /transform_config/transform.py
added unit_tests for MS SQL Server compatibility

AIRBYTE-WORKERS

added MSSQL type to DestinationType enum in DefaultNormalizationRunner.java
bumped the version of the default normalization image in DefaultNormalizationRunner.java to 0.1.50
added .put("airbyte/destination-mssql", DefaultNormalizationRunner.DestinationType.MSSQL) to mapping inside of NormalizationRunnerFactory.java

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions
Connector version bumped like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

…nfig_mssql, edited dockerfile

…-normalization

ChristopheDuong · 2021-09-22T14:52:05Z

To fix the test with column_with_quoted_names, you can have a look to see if you need this PR #6380

…-normalization

bazarnov · 2021-10-06T15:57:11Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1312655907
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1312655907
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              200     94    53%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     24    41%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  860    419    51%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 357    207    42%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         149     39    74%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1058    402    62%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            83     59    29%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     19    42%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        660    393    40%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 357    207    42%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         149     39    74%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1058    402    62%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     12    92%
	 normalization/transform_catalog/destination_name_transformer.py     120      4    97%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 357     32    91%
	 normalization/transform_catalog/table_name_registry.py              174     51    71%
	 normalization/transform_catalog/transform.py                         45     30    33%
	 normalization/transform_catalog/utils.py                             33      0   100%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         149     47    68%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1058    182    83%

…-normalization

bazarnov · 2021-10-07T09:05:54Z

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1315490217
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1315490217

…-normalization

…n` (airbytehq#6079) See the attached PR (airbytehq#6079)

bazarnov added 3 commits September 14, 2021 16:04

added destynation_type = mssql, added reserved_keywords, transform_co…

fa77e73

…nfig_mssql, edited dockerfile

edited destination-mssql spec.json to accept normalization = true

73cb90c

continue testing...

ac994c3

bazarnov self-assigned this Sep 15, 2021

github-actions bot added area/connectors Connector related issues area/worker Related to worker normalization labels Sep 15, 2021

bazarnov added 2 commits September 15, 2021 19:18

managed to connecto to dbt-sqlserver adapter

f232836

almost there with dbt

889f7ec

bazarnov linked an issue Sep 17, 2021 that may be closed by this pull request

Implement normalization for the MSSql Destination. #3430

Closed

bazarnov added 4 commits September 18, 2021 12:14

array corrections, json_operations corrections

d46b101

Merge remote-tracking branch 'origin/master' into bazarnov/3430-mssql…

16574b3

…-normalization

fixed dbt_integration_test for sql server

f3614cf

Merge remote-tracking branch 'origin/master' into bazarnov/3430-mssql…

d1fc7d6

…-normalization

bazarnov temporarily deployed to more-secrets September 19, 2021 13:06 Inactive

Merge remote-tracking branch 'origin/master' into bazarnov/3430-mssql…

d497baa

…-normalization

bazarnov added 7 commits September 23, 2021 14:52

passed test_normalization.py

979a529

need to fix concat

de48e53

implemented batches for contact function

7124249

fix unit_tests

449b598

formated

9eb4080

Merge remote-tracking branch 'origin/master' into bazarnov/3430-mssql…

0f766d6

…-normalization

added threads for faster dbt run

acfa50c

bazarnov temporarily deployed to more-secrets September 24, 2021 19:38 Inactive

added append+dedup mode to MSSQL Destination

fc9c6aa

bazarnov temporarily deployed to more-secrets September 24, 2021 19:47 Inactive

bazarnov added 2 commits September 24, 2021 22:48

bumped the version of the destination-mssql

c971dbc

edited destination-mssql documentation

f1b87bd

github-actions bot added the area/documentation Improvements or additions to documentation label Sep 24, 2021

bazarnov temporarily deployed to more-secrets October 6, 2021 15:58 Inactive

jrhizor temporarily deployed to more-secrets October 6, 2021 15:59 Inactive

bazarnov temporarily deployed to more-secrets October 6, 2021 17:23 Inactive

annalvova05 mentioned this pull request Oct 6, 2021

🐛 fix: Normalization date-time should handle empty strings "" #6379

Merged

15 tasks

jrhizor temporarily deployed to more-secrets October 6, 2021 18:50 Inactive

bumped version of base-normalization

afbe9e0

bazarnov temporarily deployed to more-secrets October 6, 2021 19:35 Inactive

merge with master

0f2d864

bazarnov temporarily deployed to more-secrets October 6, 2021 19:37 Inactive

jrhizor temporarily deployed to more-secrets October 6, 2021 19:39 Inactive

jrhizor temporarily deployed to more-secrets October 6, 2021 20:38 Inactive

Merge remote-tracking branch 'origin/master' into bazarnov/3430-mssql…

07097b4

…-normalization

jrhizor temporarily deployed to more-secrets October 7, 2021 08:35 Inactive

Merge remote-tracking branch 'origin/master' into bazarnov/3430-mssql…

67faeba

…-normalization

bazarnov temporarily deployed to more-secrets October 7, 2021 09:06 Inactive

jrhizor temporarily deployed to more-secrets October 7, 2021 09:08 Inactive

jrhizor temporarily deployed to more-secrets October 7, 2021 09:59 Inactive

bazarnov added 3 commits October 7, 2021 13:48

Merge remote-tracking branch 'origin/master' into bazarnov/3430-mssql…

e32efff

…-normalization

revert changes to destination-mssql connector

50059d9

Merge remote-tracking branch 'origin/master' into bazarnov/3430-mssql…

0021f06

…-normalization

bazarnov temporarily deployed to more-secrets October 7, 2021 15:27 Inactive

bazarnov merged commit e5abaec into master Oct 7, 2021

bazarnov deleted the bazarnov/3430-mssql-normalization branch October 7, 2021 15:46

bazarnov mentioned this pull request Oct 7, 2021

🎉 Destination MSSQL: support normalization on destination-mssql destination connector #6876

Closed

jrhizor mentioned this pull request Oct 7, 2021

Bump Airbyte version from 0.30.15-alpha to 0.30.16-alpha #6880

Merged

schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022

🎉 Base-normalization: Implement normalization for `MSSQL-destinatio…

c7d853c

…n` (airbytehq#6079) See the attached PR (airbytehq#6079)

rx007 mentioned this pull request Dec 27, 2023

[Snyk] Security upgrade axios from 0.21.1 to 1.6.3 rx007/airbyte#2209

Open

rx007 mentioned this pull request Jan 6, 2024

[Snyk] Security upgrade axios from 0.21.1 to 1.6.4 rx007/airbyte#2210

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Base-normalization: Implement `normalization` for `MSSQL-destination` #6079

🎉 Base-normalization: Implement `normalization` for `MSSQL-destination` #6079

bazarnov commented Sep 15, 2021 •

edited

Loading

ChristopheDuong commented Sep 22, 2021 •

edited

Loading

bazarnov commented Oct 6, 2021 •

edited by github-actions bot

Loading

bazarnov commented Oct 7, 2021 •

edited by github-actions bot

Loading

🎉 Base-normalization: Implement normalization for MSSQL-destination #6079

🎉 Base-normalization: Implement normalization for MSSQL-destination #6079

Conversation

bazarnov commented Sep 15, 2021 • edited Loading

What

How

NORMALIZATION MSSQL

AIRBYTE-WORKERS

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

ChristopheDuong commented Sep 22, 2021 • edited Loading

bazarnov commented Oct 6, 2021 • edited by github-actions bot Loading

bazarnov commented Oct 7, 2021 • edited by github-actions bot Loading

🎉 Base-normalization: Implement `normalization` for `MSSQL-destination` #6079

🎉 Base-normalization: Implement `normalization` for `MSSQL-destination` #6079

bazarnov commented Sep 15, 2021 •

edited

Loading

ChristopheDuong commented Sep 22, 2021 •

edited

Loading

bazarnov commented Oct 6, 2021 •

edited by github-actions bot

Loading

bazarnov commented Oct 7, 2021 •

edited by github-actions bot

Loading