estuary-cdk and new connectors #1293

The Estuary CDK differs from the earlier flow-sdk in fundamental ways: * It leans heavily into Pydantic V2 (with a polyfill for V1), which is used for validation and schema generation, married with Flow's schema inference capabilities. * It has a framework -> connector -> library structure. The framework is maximally unopinionated as to how a connector is built. But, the CDK offers library routines (the `common` module) which encapsulates the common patterns for fetch snapshot or incremental resources. * It's async at it's core. All work proceeds concurrently across all bindings.

Still missing a bunch of entities, but functional.

…nector

…wocuments or a LogCursor This avoids implementations from having to know the maximum LogCursor until they're done reading documents, and avoids nested async generators. Also add a BasicAuth credential type.

request_stream() is an AsyncGenerator over arbitrary stream chunks. Then, request() and a new request_lines() are implemented in terms of request_stream(). This allows callers to efficiently process unbounded responses.

…ming Rework the contract to allow implementations to yield checkpoints at times of their choosing. This allows for more ergonomic handling of long-lived push streams of documents.

Begin to factor out common setup steps into reuseable composite actions. Add a common estuary-cdk Dockerfile

…ation

The runtime currently uses a non-UUID placeholder internally, which causes spurious schema violations

Establish a convention that `log: Logger` is the first parameter. We're going to be threading these through everywhere -- which is desireable, because it gives us a tightly-scoped structured log context that tells us as much as possible about the surrounding task -- so let's standardize how it should be passed so we don't have to think hard about it. Also refactor `http` module to clarify APIs which are stable, vs portions that are very likely to be refactored. A few other code-review cleanups as well.

We don't need it yet, so let's not have it.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

estuary-cdk and new connectors #1293

estuary-cdk and new connectors #1293

Commits on Mar 2, 2024