destination-s3-glue: Add TableType and fix race condition #22220

blarghmatey · 2023-02-01T17:45:25Z

The TableType attribute is not getting populated in the Glue catalog which has started to lead to errors when trying to query the table from e.g. Trino. The error message observed is Cannot invoke "String.equals(Object)" because "tableType" is null
There is a race condition in initializing the destination connector where a failure occurs if multiple connections attempt to initialize at the same time, because the test table that gets created is statically named. This adds a random suffix to the table to avoid that race condition.

What

Describe what the change is solving
It helps to add screenshots if it affects the frontend.

How

Describe the solution

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

blarghmatey · 2023-02-01T17:45:45Z

cc @itaseskii

itaseskii · 2023-02-01T18:19:51Z

/test connector=connectors/destination-s3-glue

🕑 connectors/destination-s3-glue https://github.com/airbytehq/airbyte/actions/runs/4067458942
✅ connectors/destination-s3-glue https://github.com/airbytehq/airbyte/actions/runs/4067458942
No Python unittests run

Build Passed

Test summary info:

All Passed

itaseskii · 2023-02-01T19:09:49Z

/test connector=connectors/destination-s3-glue

itaseskii · 2023-02-01T19:54:05Z

/test connector=connectors/destination-s3-glue

itaseskii · 2023-02-01T20:01:54Z

/test connector=connectors/destination-s3-glue

sajarin · 2023-02-03T16:39:21Z

/test connector=connectors/destination-s3-glue

🕑 connectors/destination-s3-glue https://github.com/airbytehq/airbyte/actions/runs/4086083786
✅ connectors/destination-s3-glue https://github.com/airbytehq/airbyte/actions/runs/4086083786
No Python unittests run

Build Passed

Test summary info:

All Passed

blarghmatey · 2023-02-03T17:16:48Z

Please hold on merging this. I have found a potential bug and am working on fixing it.

blarghmatey · 2023-02-03T17:30:45Z

Nevermind, false alarm. Please proceed with merging as you are ready and able 🙂

blarghmatey · 2023-02-03T18:41:01Z

@sajarin @itaseskii what are the next steps here? Let me know if there's anything I can help with.

sajarin · 2023-02-03T18:56:35Z

@blarghmatey this looks good, the next steps here are to publish the connector and then merge it into master. I would run the action for the publishing but we have a code freeze in place until Monday so I won't be able to merge until then.

blarghmatey · 2023-02-03T19:31:52Z

@sajarin if you are able to publish the connector that would be perfect. I'm happy to wait for the code to merge as long as I can get the connector updated. Is that possible

blarghmatey · 2023-02-07T13:41:22Z

@sajarin can we get this released and merged today? Thanks!

sajarin · 2023-02-07T16:11:31Z

/test connector=connectors/destination-s3-glue

🕑 connectors/destination-s3-glue https://github.com/airbytehq/airbyte/actions/runs/4115896229
✅ connectors/destination-s3-glue https://github.com/airbytehq/airbyte/actions/runs/4115896229
No Python unittests run

Build Passed

Test summary info:

All Passed

sajarin · 2023-02-07T16:34:01Z

/publish connector=connectors/destination-s3-glue

🕑 Publishing the following connectors:
connectors/destination-s3-glue
https://github.com/airbytehq/airbyte/actions/runs/4116100405

Connector	Did it publish?	Were definitions generated?
connectors/destination-s3-glue	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

sajarin · 2023-02-07T17:24:35Z

@blarghmatey looks like I can't push to your branch can you follow step 4 from the instructions linked above and generate the definitions by editing the destination.yaml file and running the gradle command?

We can merge once the changes are committed and pushed. Thanks for being patient.

- The TableType attribute is not getting populated in the Glue catalog which has started to lead to errors when trying to query the table from e.g. Trino. The error message observed is `Cannot invoke "String.equals(Object)" because "tableType" is null` - There is a race condition in initializing the destination connector where a failure occurs if multiple connections attempt to initialize at the same time, because the test table that gets created is statically named. This adds a random suffix to the table to avoid that race condition. - The Airbyte sync ID and emitted_at fields are useful for deduplicating data. This adds those fields to the table definition since they are already included in the records as written.

sajarin · 2023-02-08T16:39:29Z

Thanks for the PR @blarghmatey and thanks for the review @itaseskii

…22220) - The TableType attribute is not getting populated in the Glue catalog which has started to lead to errors when trying to query the table from e.g. Trino. The error message observed is `Cannot invoke "String.equals(Object)" because "tableType" is null` - There is a race condition in initializing the destination connector where a failure occurs if multiple connections attempt to initialize at the same time, because the test table that gets created is statically named. This adds a random suffix to the table to avoid that race condition. - The Airbyte sync ID and emitted_at fields are useful for deduplicating data. This adds those fields to the table definition since they are already included in the records as written.

octavia-squidington-iv added area/connectors Connector related issues bounty community connectors/destination/s3-glue labels Feb 1, 2023

itaseskii approved these changes Feb 1, 2023

View reviewed changes

blarghmatey force-pushed the master branch 2 times, most recently from 0ccee34 to 8cc1063 Compare February 1, 2023 19:07

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Feb 1, 2023

blarghmatey force-pushed the master branch 2 times, most recently from d2dd1d9 to 455f856 Compare February 1, 2023 19:53

blarghmatey force-pushed the master branch from 455f856 to 7ec87cb Compare February 1, 2023 20:01

itaseskii closed this Feb 1, 2023

itaseskii reopened this Feb 1, 2023

blarghmatey force-pushed the master branch from 7ec87cb to 445e214 Compare February 1, 2023 21:41

rachellougee mentioned this pull request Feb 3, 2023

add edx user course activities model mitodl/ol-data-platform#550

Merged

7 tasks

blarghmatey force-pushed the master branch from 445e214 to 9affce1 Compare February 3, 2023 15:54

blarghmatey force-pushed the master branch from 9affce1 to 55e7df0 Compare February 3, 2023 20:36

sajarin self-requested a review February 7, 2023 16:34

sajarin approved these changes Feb 7, 2023

View reviewed changes

blarghmatey force-pushed the master branch from 55e7df0 to 45d2a83 Compare February 8, 2023 16:11

blarghmatey force-pushed the master branch from 45d2a83 to 97f6a97 Compare February 8, 2023 16:25

sajarin merged commit b0a42bf into airbytehq:master Feb 8, 2023

github-actions bot mentioned this pull request Feb 9, 2023

Bump helm chart version reference to 0.43.32 #22620

Closed

This was referenced Feb 10, 2023

Bump Airbyte version from 0.40.32 to 0.40.33 #22689

Closed

Bump Airbyte version from 0.40.32 to 0.40.33 #22839

Closed

blarghmatey mentioned this pull request Feb 13, 2023

Expose Airbyte sync ID and emitted at timestamp in table definitions mitodl/ol-data-platform#547

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

destination-s3-glue: Add TableType and fix race condition #22220

destination-s3-glue: Add TableType and fix race condition #22220

blarghmatey commented Feb 1, 2023 •

edited

Loading

blarghmatey commented Feb 1, 2023

itaseskii commented Feb 1, 2023 •

edited by github-actions bot

Loading

itaseskii commented Feb 1, 2023

itaseskii commented Feb 1, 2023

itaseskii commented Feb 1, 2023

sajarin commented Feb 3, 2023 •

edited by github-actions bot

Loading

blarghmatey commented Feb 3, 2023

blarghmatey commented Feb 3, 2023

blarghmatey commented Feb 3, 2023

sajarin commented Feb 3, 2023

blarghmatey commented Feb 3, 2023

blarghmatey commented Feb 7, 2023

sajarin commented Feb 7, 2023 •

edited by github-actions bot

Loading

sajarin commented Feb 7, 2023 •

edited by github-actions bot

Loading

sajarin commented Feb 7, 2023

sajarin commented Feb 8, 2023

destination-s3-glue: Add TableType and fix race condition #22220

destination-s3-glue: Add TableType and fix race condition #22220

Conversation

blarghmatey commented Feb 1, 2023 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

blarghmatey commented Feb 1, 2023

itaseskii commented Feb 1, 2023 • edited by github-actions bot Loading

Build Passed

itaseskii commented Feb 1, 2023

itaseskii commented Feb 1, 2023

itaseskii commented Feb 1, 2023

sajarin commented Feb 3, 2023 • edited by github-actions bot Loading

Build Passed

blarghmatey commented Feb 3, 2023

blarghmatey commented Feb 3, 2023

blarghmatey commented Feb 3, 2023

sajarin commented Feb 3, 2023

blarghmatey commented Feb 3, 2023

blarghmatey commented Feb 7, 2023

sajarin commented Feb 7, 2023 • edited by github-actions bot Loading

Build Passed

sajarin commented Feb 7, 2023 • edited by github-actions bot Loading

sajarin commented Feb 7, 2023

sajarin commented Feb 8, 2023

blarghmatey commented Feb 1, 2023 •

edited

Loading

itaseskii commented Feb 1, 2023 •

edited by github-actions bot

Loading

sajarin commented Feb 3, 2023 •

edited by github-actions bot

Loading

sajarin commented Feb 7, 2023 •

edited by github-actions bot

Loading

sajarin commented Feb 7, 2023 •

edited by github-actions bot

Loading