Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source Zendesk Support: Add undeclared columns to spec #24509

Closed
erohmensing opened this issue Mar 24, 2023 · 1 comment · Fixed by #29940
Closed

Source Zendesk Support: Add undeclared columns to spec #24509

erohmensing opened this issue Mar 24, 2023 · 1 comment · Fixed by #29940
Assignees

Comments

@erohmensing
Copy link
Contributor

What

In this PR, BasicReadTest.test_read test in the Connector Acceptance Tests was updated to fail if the connector produces
stream records which contain columns that haven't been declared in the spec.

Zendesk Support currently fails the updated test. Its acceptance-test-config.yml was edited with the
fail_on_extra_columns: false parameter in order to avoid this change from making the connector fail CAT.

We want to add the undeclared columns to the spec.

How this will help:

  • Column selection is currently blocked for connectors that don't declare all columns that they pass. This
    is because turning on column selection will stop these undeclared columns from being sent, when they were
    previously being sent. Doing this allows us to enable column selection for Zendesk Support!
  • Users will better understand the shape of the data they'll receive, since the columns will match the spec.

How

The following descriptions of streams that pass undeclared columns come from results of the failed connector acceptance test:

columns.txt:./source-zendesk-support.txt-85-The groups stream has the following schema errors:
columns.txt:./source-zendesk-support.txt:86:Additional properties are not allowed ('default', 'description', 'is_public' were unexpected)
columns.txt:./source-zendesk-support.txt-111-The macros stream has the following schema errors:
columns.txt:./source-zendesk-support.txt:112:Additional properties are not allowed ('default', 'raw_title' were unexpected)
columns.txt:./source-zendesk-support.txt-161-The ticket_comments stream has the following schema errors:
columns.txt:./source-zendesk-support.txt:162:Additional properties are not allowed ('event_type' was unexpected)
columns.txt:./source-zendesk-support.txt-332-The ticket_fields stream has the following schema errors:
columns.txt:./source-zendesk-support.txt:333:Additional properties are not allowed ('key', 'custom_statuses' were unexpected)
columns.txt:./source-zendesk-support.txt-466-The tickets stream has the following schema errors:
columns.txt:./source-zendesk-support.txt:467:Additional properties are not allowed ('fields', 'from_messaging_channel' were unexpected)
columns.txt:./source-zendesk-support.txt-654-The users stream has the following schema errors:
columns.txt:./source-zendesk-support.txt:655:Additional properties are not allowed ('iana_time_zone' was unexpected)
columns.txt:./source-zendesk-support.txt-789-The ticket_forms stream has the following schema errors:
columns.txt:./source-zendesk-support.txt:790:Additional properties are not allowed ('agent_conditions', 'end_user_conditions' were unexpected)

  1. Add the missing properties indicated by the Additional properties are not allowed ('<column>', 'column' were unexpected) logs to the connector's spec.
  2. Remove fail_on_extra_columns: false from the acceptance-test-config.yml file.
  3. Commit changes to spec and acceptance-test-config.yml and open a PR.
  4. Run tests on the connector, either automatically via CI or manually via the /test command
  5. Profit!

Definition of done: Zendesk Support passes CAT without declaring fail_on_extra_columns: false. If the
API starts sending over extra columns in the future, we will catch and fix them as part of the #connector-health
movement.

@evantahler
Copy link
Contributor

When this issue is picked up, please also consult our DataDog Dashboard for Record Unexpected Fields for this connector. The API might be returning even more record properties that need to be added to the catalog.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants