From 1f967fcf93b0b9348580dcfde7fd265f18c35a43 Mon Sep 17 00:00:00 2001 From: cgardens Date: Tue, 17 Nov 2020 13:44:25 -0800 Subject: [PATCH] remove docs --- docs/architecture/incremental.md | 22 ---------------------- 1 file changed, 22 deletions(-) delete mode 100644 docs/architecture/incremental.md diff --git a/docs/architecture/incremental.md b/docs/architecture/incremental.md deleted file mode 100644 index 0d24e7cee562..000000000000 --- a/docs/architecture/incremental.md +++ /dev/null @@ -1,22 +0,0 @@ -# Configuration: Streams & Incremental Syncs - -## AirbyteStream -* The `AirbyteStream` has two sections: - * The `discovered` section describes information and metadata about the data in the source. This section is generated by the source connector. Outside of the source, this section should be treated as read only. - * The `configuration` section describes functionality that an end user would like applied on the sync. This section is configured by the end-user, and is never touched by the source. - -## Discovered - -Currently this section contains two fields: `json_schema` and `metadata`. -* `json_schema` describes the structure of the data in a source. -* `metadata` describes other information about the data in the source. For example what features are supported for the stream. Currently the `metadata` object contains two fields. - * `supported_sync_modes` which describes which sync modes are supported for a stream. - * `default_comparable` which describes what field should be used to determine if a record should be included in a sync. If this is set, the end user does not need to specify their own comparable. If it is not set, then the end user does need to specify it. If it is set and the user sets their own, the user's value overrides the default value. - -## Configuration -* This section currently has two fields: `sync_mode` and `comparable`. -* `sync_mode` which describes how the data will be synced (e.g. `full_refresh` or `incremental`). -* `comparable` which is the path to the field that will be used to determine if a record is new or modified since the last sync. This field is REQUIRED if `sync_mode` is `incremental`. Otherwise it is ignored. - -## Source -A source advertises that it can do an incremental sync for a given stream, by specifying `incremental` as one of the sync modes. It can specify a `default_comparable` so that the user does not need to determine which field is used to determine which records are new or modified. If it does not specify this field, then the user _must_ set `comparable` field themselves if they want to use `incremental` syncing.