feat: iceberg schema airbyte metadata

[jdpgrailsdev]

What

• Add Airbyte metadata columns to schema
• Actually detect insert/update/delete operation

How

• Add Airbyte metadata columns to schema
• Detect insert/update/delete when converting to Iceberg record

Review guide

1. AirbyteTypeToIcebergSchema.kt
2. IcebergUtil.kt

Can this PR be safely reverted and rolled back?

• YES 💚
• NO ❌

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Nov 21, 2024 8:32pm

[edgao]

this looks a lot like https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/bulk/core/load/src/main/kotlin/io/airbyte/cdk/load/data/AirbyteTypeToAirbyteTypeWithMeta.kt ? which we can maybe reuse in some way

[subodh1810]

Most of these columns are being added to the schema using the mapper architecture. if you take a look at

val schema = pipeline.finalSchema.withAirbyteMeta(true).toIcebergSchema(primaryKeys)

In IcebergV2Writer. The withAirbyteMeta method is adding a bunch of em. If we want to add more like the generation id , etc. we should use the same design.

[jdpgrailsdev]

this looks a lot like https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/bulk/core/load/src/main/kotlin/io/airbyte/cdk/load/data/AirbyteTypeToAirbyteTypeWithMeta.kt ? which we can maybe reuse in some way

@edgao It does and I wasn't sure how to reuse that given that it returns Airbyte types and not Iceberg types. I suspect what is really being pointed out here is some refactor to this in the CDK to make the common parts more common.