[Low-Code CDK] Generate Pydantic models from the handwritten component manifest schema

[brianjlai]

Low Code Schema Refactor Phase 2

Tell us about the problem you're trying to solve

As part of processing an incoming connector manifest, we want to be able to parse a manifest into related Pydantic models. In order to be able to do that, we first need to have the Pydantic models defined.

Describe the solution you’d like

Instead of handwriting every Pydantic model by hand, we should leverage our existing tooling that can generate Pydantic models based on an input schema. Using the schema written before as input, there should be a gradle command defined that will generate all the Pydantic models and store them within airbyte-cdk/python/airbyte_cdk/sources/declarative/models.

Implementation Details

We already have some prior art that we can work off because we are doing a similar process for transforming the airbyte protocol into Pydantic models in airbyte-cdk/python/bin/generate-protocol-files.sh.

We should be trying to leverage the existing airbyte/code-generator which under the hood is installing and using the datamodel-code-generator library.

In doing a spike on the level of effort, one option while doing object generation we may want to leverage is --enum-field-as-literal one. This would allow us to enforce a Literal on the type name when a component definition is ambiguous. For example, for OffsetIncrement and PageIncrement who have equivalent fields, Pydantic will choose the first one that matches while parsing. However, we can use Literal['OffsetIncrement'] to ensure the correct component gets resolved.

Acceptance Criteria

• Pydantic models are defined within the airbyte-cdk declarative package and should be accessible at runtime
• Pydantic models are to be derived from the component manifest schema
• A gradle command exists to easily regenerate models when the schema is changed

[brianjlai]

grooming notes:

• run it in the build pipeline and fail if a diff is detected
• any time we mess with the CI pipeline, theres always some unknowns