feat(source): Avro with AWS Glue Schema Registry

[xiangjinwu]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Before this PR, there is a boolean use_schema_registry that differentiates usage of schema.location vs (confluent) schema.registry.

In this PR, we introduce AWS Glue schema registry with aws.glue.schema_arn option, and the boolean is deprecated in favor of enums.

• Hint
  • Review with hide whitespace because if-else are updated to match.
  • The 2 largest change glue.slt and majority of glue_resolver.rs are for tests.
• In catalog.proto, StreamSourceInfo still has use_schema_registry for backward compatibility.
• In src/frontend/src/handler/create_source.rs and src/connector/src/parser/mod.rs, the boolean has been replaced by enum SchemaLocation.
• In src/connector/src/parser/avro/parser.rs, the Option<Arc<ConfluentSchemaCache>> has been replaced by enum WriterSchemaCache.

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added test labels as necessary. See details.
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

Added support for AWS Glue Schema Registry when using encode avro for sources:

encode avro (
  aws.glue.schema_arn = 'arn:aws:glue:ap-southeast-1:123456123456:schema/default-registry/MyEvent',
  // other aws auth options same as s3 / kinesis
  aws.region = '...',
  aws.credentials.access_key_id = '...',
  ...
)

When using IAM permission policies, the following actions shall be allowed:

• glue:GetSchemaVersion

[xxchan]

Generally LGTM! The shape is good. Didn't check business logic carefully.

[xxchan]

Perhaps we need to add logic to check AWS_GLUE_SCHEMA_ARN_KEY is not specified together with other 2.

[xiangjinwu]

Just tested and when they are specified together:

NOTICE:  Get unknown format_encode_options for Plain Avro: schema.location

So it is already covered by the unknown options check, just not fatal. After the config parsing refactor #17675 we can try to make unknown options a fatal error.

[xxchan]

This isn't a good style to me: the trait seems a little unnecessary to me, but anyway

[xiangjinwu]

Unnecessary as a tool (generic) but it is a doc to make sure RealGlueSchemaCache and MockGlueSchemaCache have the same interface.

The unnecessary part is the enum, as we can simply Box<dyn GlueSchemaCache>. This aligns more with mocked interfaces in Java or Go, but in rust we always prefer static dispatch with enum rather than dynamic dispatch (?)