Fix creation of custom connector definitions

[josephkmh]

When creating a custom source/destination connector, the frontend should use createCustomSourceDefinition() and createCustomDestinationDefinition() respectively. The currently used methods (createSourceDefinition() and createDestinationDefinition()) result in orphaned rows in the actor_catalog.

Context on what led to this issue:

PR 19221 updated the following API methods in the webapp globally:

1. createSourceDefinition() => createCustomSourceDefinition()
2. createDestinationDefinition() => createCustomDestinationDefinition()
3. updateDestinationDefinition() => updateCustomDestinationDefinition()
4. updateSourceDefinition() => updateCustomSourceDefinition()

While 1. and 2. were correct, 3. and 4. were causing OSS users to no longer be able to update connectors that are not custom.

This resulted in an oncall issue. A targeted revert PR was introduced (PR 19627) that reverted all 4 methods above. This means:

1. Non-custom connectors can now be updated again (good!)
2. New custom connectors are now being created with the old methods again (impact unclear, probably a new migration similar to the first one will be needed to update these)
3. Custom connectors created before the revert commit are now being updated with the old methods (unclear whether this works, needs to be tested)

Relates to #9652

[octavia-squidington-iv]

cc @airbytehq/frontend

[josephkmh]

After some local testing: creating custom connectors on master currently does not work. The connectors are created in the database, but are not returned by the server and consequently not shown in the UI. This is because the webapp is now using the old createSourceDefinition() endpoint, and the corresponding row in actor_definition_workspace_grant is not created.

[timroes]

Synced offline about this. Our outcome and action items

• createCustomSource/DestinationDefinition should be used (with workspaceId)
• Once in UI, we want to remove the old (non workspace scoped) create APIs
• updateSource/DestinationDefinition (does not need workspaceId)
  • Cloud will check if the passed connector ID, is actually assigned to a workspace that the user has access to, if not throws 404 in cloud if connector ID
• Same logic as for update should apply for the delete endpoint
• No migration will be needed, we accept orphaned rows in the actor table for the rare cases users ran into that case

[xiaohansong]

On backend we need to implement following tickets to address these action items:

#19669
#19670

[josephkmh]

Thanks @xiaohansong - could we also add the removal of the old creation endpoints to #19669 (or a new issue)?

• POST /v1/source_definitions/create
• POST /v1/destination_definitions/create

Just want to make sure we get rid of those two endpoints so we don't have another accidental usage of them in the FE again 🙂

[xiaohansong]

@josephkmh Just did that!