🐛 Destination Weaviate: Multi Tenancy Support

[Marcus0086]

What

Currently there is no way to enable multi-tenancy in weaviate through the configuration. Which disallows certain use cases like sharding and data isolation in db level.

it fixes #33992 (comment)

How

The solution is relatively simple, added an optional field in config object which can take user input as tenant_id
If tenand_id is empty then the schema would be without multitenancy enabled and vice-versa

More on this topic: https://weaviate.io/developers/weaviate/api/rest/schema#enable-multi-tenancy

🚨 User Impact 🚨

No, there is no as such merge conflicts and breaking changes. Just a new field is added as optional field

For connector PRs, use this section to explain which type of semantic versioning bump occurs as a result of the changes. Refer to our Semantic Versioning for Connectors guidelines for more information. Breaking changes to connectors must be documented by an Airbyte engineer (PR author, or reviewer for community PRs) by using the Breaking Change Release Playbook.

If there are breaking changes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Actions

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Connector version is set to 0.0.1
  • Dockerfile has version 0.0.1
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
  • docs/integrations/README.md

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.

Connector Generator

• Issue acceptance criteria met
• PR name follows PR naming conventions
• If adding a new generator, add it to the list of scaffold modules being tested
• The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:generateScaffolds then checking in your changes
• Documentation which references the generator is updated as needed

Updating the Python CDK

Airbyter

Before merging:

• Pull Request description explains what problem it is solving
• Code change is unit tested
• Build and my-py check pass
• Smoke test the change on at least one affected connector
  • On Github: Run this workflow, passing --use-local-cdk --name=source-<connector> as options
  • Locally: airbyte-ci connectors --use-local-cdk --name=source-<connector> test
• PR is reviewed and approved

After merging:

• Publish the CDK
  • The CDK does not follow proper semantic versioning. Choose minor if this the change has significant user impact or is a breaking change. Choose patch otherwise.
  • Write a thoughtful changelog message so we know what was updated.
• Merge the platform PR that was auto-created for updating the Connector Builder's CDK version
  • This step is optional if the change does not affect the connector builder or declarative connectors.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 17, 2024 10:33am

[CLAassistant]

All committers have signed the CLA.

[github-actions]

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

• PR name follows PR naming conventions
• Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
• Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
• You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
• Secrets in the connector's spec are annotated with airbyte_secret
• All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
• Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
• Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
• If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

1. Check for hidden checklists in your PR description

2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

[flash1293]

Hi @Marcus0086 , this looks mostly good to me, only thing missing seems to be handling the tenant id to the delete method. Could you add that and a test for it? Then it should be ready to merge.

I bumped the version number and added the changelog entry.

Thanks for the contribution

[flash1293]

CI PR here: #34269

[Marcus0086]

Sure adding that

[flash1293]

@Marcus0086 let me know once it's ready for another look

[Marcus0086]

Hey @flash1293 , can you review the changes?
I added the condition to check if tenant_id is not associated to the class for multiple connectors.
Thanks

[Marcus0086]

Getting this error when trying to do sync on another connection with different tenant_id

Delete in batch! Unexpected status code: 422, with response body: {'error': [{'message': 'batch delete objects: cannot find objects: tenant not found: "45678"'}]}.

[Marcus0086]

Should I escape the deleting if a tenant is not in class tenants? This would be for multitenancy.
i guess some process is checking if a stream exists then delete it?
if a tenant is not there then the data is not indexed

[flash1293]

@Marcus0086 What do you mean by "escape"? I would have expected that prior to deleting you add the currently configured tenant id to all the classes.

Basically execute https://github.com/airbytehq/airbyte/pull/34229/files?diff=unified&w=1#diff-605751b53927fd85aee9d14cf7cc427ddc74aad7c986c31bffc1b06e528a5923R105 even if the class exists already - because we always need to make sure the tenant is added to the classes

[Marcus0086]

@Marcus0086 What do you mean by "escape"? I would have expected that prior to deleting you add the currently configured tenant id to all the classes.

Basically execute https://github.com/airbytehq/airbyte/pull/34229/files?diff=unified&w=1#diff-605751b53927fd85aee9d14cf7cc427ddc74aad7c986c31bffc1b06e528a5923R105 even if the class exists already - because we always need to make sure the tenant is added to the classes

By ‘escape’ I wanted to mean that I would be skipping the whole deletion part if tenant is not in the class tenants list.
But we can add it before deletion too that way we are making sure tenant is associated with the class.

[flash1293]

Adjusted the unit test but LGTM now. Merging...

[Marcus0086]

Thanks for all the help