storage: Design doc for Source Versioning / Tables from Sources [ENG-TASK-15]

[rjobanp]

Motivation

Tips for reviewer

Rendered

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
• This PR includes the following user-facing behavior changes:

[benesch]

Thank you for writing this up, @rjobanp! The first five sections (problem, solution, success criteria, out of scope, solution proposal) are really clear and well done.

I'm not the expert on the source ingestion pipeline so I only lightly skimmed the implementation plan and don't have useful feedback to offer. I focused my review in that section on the UX considerations (i.e., the specific SQL syntax, and the migration path for the _progress relations).

[bosconi]

Looks great! The phased approach looks well worth the extra steps to reduce risk.

[morsapaes]

Migrating a private discussion from Slack around backfilling behavior for source tables:

Something I was mulling over after that user thread:
I expect users to ask us for a solution that allows them to not discard all state of the v0 source table in v1. For example, if they have a 7-day retention period in Kafka and create a source table v0, then after 7 days need to evolve the schema and create a source table v1, the blue/green process would cause all the green downstream objects to hydrate based on the new data only.
Naively, I'd say this looks like a UNION ($?) between v0 and v1, but...we might need to think about how to cleanly approach this (probably via dbt).
We've discussed keeping historical state around for in-place schema changes (i.e. adding new columns, but possibly keeping around the data for old columns that were previously ingested even if they're dropped), but here we're just assuming users are okay bootstrapping their source and its dependencies from scratch.

cc @sdht0, who separately also brought this up in conversation with @rjobanp.

[rjobanp]

I expect users to ask us for a solution that allows them to not discard all state of the v0 source table in v1.

Thanks for including that here @morsapaes ! It's a great question and will certainly be important to make this feature usable in real-world scenarios.

My initial thought is that we should try to backfill the new v1 source table using the usual snapshot functionality present in our sources, though this would only be able to make data available at timestamps that are still preserved in the upstream system's replication log (e.g. Postgres WAL, MySQL binlog, kafka topic). Hopefully this will be acceptable for most usecases, but it's worth thinking about whether there are better options (such as the union idea you've proposed).

[notion-workspace]

Design Doc

[github-actions]

All contributors have signed the CLA.
_{Posted by the CLA Assistant Lite bot.}

[petrosagg]

thanks for writing this up!

[morsapaes]

We'll move this to within scope and get help from Adapter with porting webhook sources into the new model. Added a placeholder task to #28653.

[benesch]

"data source" isn't a term we've used before. My preference would be to stick with a plain ol' "source ID." Are you worried that that would be too vague?

[rjobanp]

My thought was that we may also have tables who's data comes from other 'data sources' in the future -- e.g. if we supported CREATE TABLE .. FROM WEBHOOK or CREATE TABLE .. FROM FIVETRAN, etc, such that I didn't want to tie this column directly to our existing concept of sources. If we feel that other stuff isn't likely to happen I can switch it to just source_id

[benesch]

Ah! Ok. Will respond in the other thread above (#27907 (comment)) to avoid forking the discussion, since they've converged.

[benesch]

In other tables we use simply updated_at (see e.g. mz_cluster_replica_statuses) for the "last update" time.

[morsapaes]

+1

[benesch]

Not at all married to this, but in mz_source_statistics we use "known (as in offset_known) to represent the idea of things that are known to Materialize but may not be perfectly up to date.

Suggested change

	Introduce a `mz_catalog.mz_available_source_references` with the following columns:
	Introduce a `mz_catalog.mz_known_source_references` with the following columns:

[morsapaes]

Also for consistency, all our source-related tables start with mz_source, so we could roll with mz_source_references. Do we need an additional qualifier?

[benesch]

We do have precedent for a word before mz_source, like mz_kafka_sources, mz_postgres_sources, etc.

[benesch]

(I don't mind mz_source_references at all though.)

[morsapaes]

Also for consistency, all our source-related tables start with mz_source, so we could roll with mz_source_references. Do we need an additional qualifier?

[morsapaes]

+1

[benesch]

I think I understand the commonality we're trying to extract from webhook sources here, but I worry it won't be that valuable in practice. It's hard for me to imagine a real world use case for changing the body format, for example.

Retroactively adding new headers seems more valuable, but I'm worried it's actually quite surprising if we store the full raw HTTP requests for all time, even though that raw collection isn't visible to users. For example we document that you can exclude headers to omit sensitive information: https://materialize.com/docs/sql/create-source/webhook/#excluding-header-fields

But with the proposed design, that sensitive information will sit secretly inside Materialize until the source is dropped—unless someone reveals it with an ALTER SOURCE command.

The migration here for existing webhook sources seems inviable, too? Since we don't have the raw HTTP bytes saved for existing webhook sources.

Is there something lighter weight we could do here instead? Something that moves the ball forward without fully aligning with the new subsources model while also not painting ourselves into a corner? I think the most pressing concern is separating out the webhook source object from the webhook table object. But I think you could reasonably say that as a special case 1) envelopes and encodings are a property of the webhook source, not the subsource and 2) webhook subsources cannot be dropped/added.

Backing up a level, I think what folks actually want with webhook sources is the ability to "write at" webhook tables. Much more so than with other sources. Usually with webhooks you're totally happy to tear down the existing webhook source and make a new one with the new configuration (new headers, new CHECK, new body format, whatever) ... if only you could backfill the new webhook source with the fixed up data from the old webhook source!

CREATE SOURCE FROM WEBHOOK og_webhook ... HEADER 'foo';

-- Shoot, I need another header going forward!

CREATE SOURCE FROM WEBHOOK webhook2 ... HEADER foo, HEADER 'bar';

-- One time backfill

INSERT INTO webhook2 SELECT body, foo, 'unknown' AS bar FROM og_webhook;

[benesch]

I'm short on time, so that might not be entirely coherent. Happy to chat more about this live next time we're both in the office, @rjobanp, or during this week's S&S sync.

[rjobanp]

Those are all valid concerns and I am definitely worried about the migration of existing webhook sources to the model proposed here.
While we could do something simpler, I worry that we will continue to have special-casing for webhook sources and will need additional special casing for CREATE TABLE .. FROM SOURCE statements that refer to webhook sources unless we do the work now to align them to the new model. This isn't just confusing for us in the codebase, but also for users and our documentation.

Following up to our live discussion -- if the main concern is that there might be sensitive data in the raw request bytes we write to the first persist collection, I wonder if we instead can just have the 'Persist Source' ask persist to truncate the 'raw' collection as it ingests data into the downstream 'tables', such that we remove data just after it's been decoded. This would still operate with the new model but would just mean that any new 'table' added to an existing webhook source wouldn't be able to re-ingest previous requests, but all future requests would get decoded individually by all tables from that point forward.

[benesch]

New webhook design LGTM! 🙇🏽

[rjobanp]

This PR has gotten fairly big and since we've agreed on all the open discussion points I'm going to merge and apply any future updates to the doc in separate PRs