🎉 New Destination: Teradata Vantage

[sc250072]

What

Describe what the change is solving
Teradata Vantage is the complete cloud analytics and data platform with next-generation, cloud-native deployment and expanded analytics capabilities. This connector helps to load data to teradata from other sources and apply transformations through airbyte.
It helps to add screenshots if it affects the frontend.
doesn't effect the frontend

How

Describe the solution
The current version of the connector provides importing data to teradata through airbyte. The implemented version of the connector will allow for overwrite and append sync modes. Tried to implement basic and dbt normalization but facing issues on normalization process. Currently, this connector able to load data to teradata into raw tables.

Recommended reading order

1. TeradataDestination.java
2. TeradataSqlOperations.java and rest

🚨 User Impact 🚨

No breaking changes. User can load data from any airbyte supported source to teradata through airbyte.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • docs/integrations/README.md
  • airbyte-integrations/builds.md
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub by running the /publish command described here
• After the connector is published, connector added to connector index as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

• Issue acceptance criteria met
• PR name follows PR naming conventions
• If adding a new generator, add it to the list of scaffold modules being tested
• The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• Documentation which references the generator is updated as needed

Tests

Unit

Integration

Acceptance

Put your acceptance tests output here.

Integration/AcceptanceTests requires teradata vantage instance and able to access from machine, where these tests are running. Please follow Prerequisites section of https://github.com/SatishChGit/airbyte/blob/teradata_java_dest_conn/docs/integrations/destinations/teradata.md.
To run Integration/AcceptanceTests, this connector requires config.json and failureconfig.json json files under airbyte-integrations/connectors/destination-teradata/secrets
format of config.json and failureconfig.json is
{ "host" : "teradatainstancehostname", "username": "username", "password": "password", "schema" : "schemaname" }

Below three test cases are failing and need airbyte help to resolve these.
TeradataDestinationAcceptanceTest. [1] exchange_rate_messages.txt, exchange_rate_catalog.json
TeradataDestinationAcceptanceTest. [2] edge_case_messages.txt, edge_case_catalog.json
TeradataDestinationAcceptanceTest. [2] edge_case_messages.txt, edge_case_catalog.json

[luistorrealbaCinfo]

Hi,

I was trying to use this connector. Can you please tell me where should I look into in order to try this connector out?

[sc250072]

Hi,

I was trying to use this connector. Can you please tell me where should I look into in order to try this connector out?

Hi, please refer to https://github.com/SatishChGit/airbyte/blob/teradata_java_dest_conn/docs/integrations/destinations/teradata.md

[natalyjazzviolin]

Hi @SatishChGit ! Thanks so much for your contribution. Currently we're blocked from testing/merging this PR because we don't have a sandbox account. I've opened a request here:
#19656

[sc250072]

Hi @SatishChGit ! Thanks so much for your contribution. Currently we're blocked from testing/merging this PR because we don't have a sandbox account. I've opened a request here: #19656

natalyjazzviolin Thank you for your response. Can you check the possibility of setting up free version of Vantage Express on machines your local machines and can be used same machine for testing. Please check this possibility and let us know if you need any help on this.

If you need a new instance of Vantage, you can install a free version called Vantage Express in the cloud on Google Cloud, Azure, and AWS. You can also run Vantage Express on your local machine using VMware, VirtualBox, or UTM.

[marcosmarxm]

Hello 👋, first thank you for this amazing contribution.

We really appreciate the effort you've made to improve the project.
We ask you patience for the code review. Last month our team was focused on Hacktoberfest event and that probably left some PR without the proper feedback. And this week, due to the Thanksgiving US Holiday, most our team is out of office with their families. Another important piece of information why code won't be merge this week is: as a safety measure the core team has decided to freeze merging code to main branch to keep the release stable. Next week we'll return to you with the proper code review and update the status of your contribution.

If you have any questions feel free to send me a message in Slack!
Thanks!

[sc250072]

As per suggestion from marcosmarxm, creating two PR requests for vantage connector. One for Base connector without normalization and other one for normalization support.