Incremental Docs and Data Model Update #1021

cgardens · 2020-11-19T06:16:05Z

What

As I've started documenting the data model and working with it, I think there was a pretty big weakness in the original implementation.
I have restructured it so that an ConfiguredAirbyteStream contains an AirbyteStream. This fixes a few things
- first copying the json_schema into ConfiguredAirbyteStream but not the other fields meant that the ConfiguredAirbyteStream didn't have enough information to configure an AirbyteStream (i.e. it didn't know what the default_cursor_field) was supposed to be.
- Copying some of the data and not others was also just confusing. In trying to document it, I couldn't write docs that gave good motivation for what was and wasn't copied. So if I was confused, the user was going to be flummoxed.
- Without copying the whole AirbyteStream handling the distinction between source-defined and user-defined cursors was going to be very confusing and involve copying more specific parts of the stream.

How

Describe the solution

Checklist

I have an branch that implements this restructuring. Still need to run integration tests and push images, but should be relatively fast.

Recommended reading order

airbyte_protocol.yaml
incremental.md
catalog.md

cgardens · 2020-11-19T06:18:19Z

airbyte-protocol/models/src/main/resources/airbyte_protocol/airbyte_protocol.yaml

@@ -134,6 +134,11 @@ definitions:
        type: array
        items:
          "$ref": "#/definitions/SyncMode"
+      cursor_field:


we could replace this with a boolean. with a source-defined cursors, it's not technically necessary to expose them outside the source for the actual replication. i exposed it in this iteration so that the UI would be able to display what field would be used (even if it's not configurable). i can go either way on this one.

making this not a boolean blurs the distinction between catalog and configuredcatalog because this feels like configuration. can't the user be expected to know that if the cursor_field_configurable==false then the cursor is the default_cursor_field?

cool. going with the boolean as you suggest. called source_defined_cursor_field

sherifnada · 2020-11-19T17:32:30Z

airbyte-protocol/models/src/main/resources/airbyte_protocol/airbyte_protocol.yaml

@@ -134,6 +134,11 @@ definitions:
        type: array
        items:
          "$ref": "#/definitions/SyncMode"
+      cursor_field:


making this not a boolean blurs the distinction between catalog and configuredcatalog because this feels like configuration. can't the user be expected to know that if the cursor_field_configurable==false then the cursor is the default_cursor_field?

docs/architecture/catalog.md

sherifnada · 2020-11-19T17:39:54Z

docs/architecture/airbyte-specification.md

@@ -168,7 +168,7 @@ read(Config, AirbyteCatalog, State) -> Stream<AirbyteMessage>

 * Input:
  1. `config` - A configuration JSON object that has been validated using the `ConnectorSpecification`.
-  2. `catalog` - An `AirbyteCatalog`. This `catalog` should be a subset of the `catalog` returned by the `discover` command. It is what will be used in the `read` command to select what data to transfer.
+  2. `catalog` - An `ConfiguredAirbyteCatalog`. This `catalog` should be constructed from the `catalog` returned by the `discover` command. This is done by copying the streams that you want to sync from the `AirbyteCatalog` into the `ConfiguredAirbyteCatalog`. To convert an `AirbyteStream` to a `ConfiguredAirbyteStream` copy the `json_schema` and `name` fields. struct It is what will be used in the `read` command to select what data to transfer.


Suggested change

2. `catalog` - An `ConfiguredAirbyteCatalog`. This `catalog` should be constructed from the `catalog` returned by the `discover` command. This is done by copying the streams that you want to sync from the `AirbyteCatalog` into the `ConfiguredAirbyteCatalog`. To convert an `AirbyteStream` to a `ConfiguredAirbyteStream` copy the `json_schema` and `name` fields. struct It is what will be used in the `read` command to select what data to transfer.

2. `catalog` - A `ConfiguredAirbyteCatalog`. This `catalog` should be constructed from the `catalog` returned by the `discover` command. This is done by copying the streams that you want to sync from the `AirbyteCatalog` into the `ConfiguredAirbyteCatalog`. To convert an `AirbyteStream` to a `ConfiguredAirbyteStream` copy the `json_schema` and `name` fields. struct It is what will be used in the `read` command to select what data to transfer.

sherifnada · 2020-11-19T17:40:33Z

docs/architecture/airbyte-specification.md

@@ -168,7 +168,7 @@ read(Config, AirbyteCatalog, State) -> Stream<AirbyteMessage>

 * Input:
  1. `config` - A configuration JSON object that has been validated using the `ConnectorSpecification`.
-  2. `catalog` - An `AirbyteCatalog`. This `catalog` should be a subset of the `catalog` returned by the `discover` command. It is what will be used in the `read` command to select what data to transfer.
+  2. `catalog` - An `ConfiguredAirbyteCatalog`. This `catalog` should be constructed from the `catalog` returned by the `discover` command. This is done by copying the streams that you want to sync from the `AirbyteCatalog` into the `ConfiguredAirbyteCatalog`. To convert an `AirbyteStream` to a `ConfiguredAirbyteStream` copy the `json_schema` and `name` fields. struct It is what will be used in the `read` command to select what data to transfer.


this should be updated to indicate you copy the whole stream right?

yup! fixed.

sherifnada · 2020-11-19T18:05:11Z

docs/architecture/incremental.md

+For a source to do incremental sync is must be able to keep track of new and updated records. This can take a couple different forms. Before we jump into them, we are going to use the word cursor or cursor field to describe the field or column in the data that Airbyte uses as a comparable to determine if any given record is new or has been updated since the last sync.
+
+## Source-Defined Cursor.
+Some sources are able to determine the cursor that the use without any user input. For example, in the exchange rates api source, the source itself can determine that date field is in fact the field to be used to determine the last record that was synced. In these cases, the source will set the `cursor_field` attribute in the `AirbyteStream`.


Suggested change

Some sources are able to determine the cursor that the use without any user input. For example, in the exchange rates api source, the source itself can determine that date field is in fact the field to be used to determine the last record that was synced. In these cases, the source will set the `cursor_field` attribute in the `AirbyteStream`.

Some sources are able to determine the cursor that they use without any user input. For example, in the exchange rates api source, the source itself can determine that date field is in fact the field to be used to determine the last record that was synced. In these cases, the source will set the `cursor_field` attribute in the `AirbyteStream`.

docs/architecture/incremental.md

sherifnada · 2020-11-19T18:07:06Z

docs/architecture/incremental.md

+Let's assume that our warehouse contains all of the data that it did at the end of the previous section. Now unfortunately the king and queen lose their heads. Let's see that delta:
+```json
+[
+    { "name": "Louis XVI", "deceased": true, "updated_at": 1793 },


this is such a grim example 🤣

if you couldn't tell, i was very please with myself when i came up with this. 😛

docs/architecture/catalog.md

jrhizor · 2020-11-19T18:29:38Z

docs/architecture/incremental.md

+
+## Overview
+
+Incremental syncs in Airbyte allow sources to replicate only new or modified data. This prevents re-sending data that has already been sent to your warehouse. We will call this set of new or updated records the delta going forward.


Should emphasize that this doesn't pull the full data from the source either.

cgardens commented Nov 19, 2020

View reviewed changes

cgardens requested review from sherifnada, jrhizor, michel-tricot and ChristopheDuong November 19, 2020 06:18

sherifnada reviewed Nov 19, 2020

View reviewed changes

jrhizor approved these changes Nov 19, 2020

View reviewed changes

cgardens force-pushed the cgardens/incremental_docs branch from 5f6a33c to 2b72d97 Compare November 19, 2020 23:00

cgardens added 3 commits November 19, 2020 15:17

new new catalog

19abe85

done?

c998b98

maybe

3ec5253

cgardens force-pushed the cgardens/incremental_docs branch from 2b72d97 to 3ec5253 Compare November 20, 2020 02:38

michel-tricot approved these changes Nov 20, 2020

View reviewed changes

more

30f124f

cgardens merged commit 02819a4 into master Nov 20, 2020

cgardens deleted the cgardens/incremental_docs branch November 20, 2020 06:07

sherifnada mentioned this pull request Nov 23, 2020

update sendgrid with new models #1065

Merged

snyk-bot mentioned this pull request Mar 11, 2022

[Snyk] Security upgrade yup from 0.28.5 to 0.30.0 yeedster/airbyte#12

Open

snyk-bot mentioned this pull request Sep 26, 2022

[Snyk] Fix for 5 vulnerabilities yeedster/airbyte#83

Open

yeedster mentioned this pull request Sep 30, 2022

[Snyk] Fix for 5 vulnerabilities yeedster/airbyte#84

Open

snyk-bot mentioned this pull request Oct 4, 2022

[Snyk] Fix for 5 vulnerabilities yeedster/airbyte#85

Open

This was referenced Oct 5, 2022

[Snyk] Fix for 5 vulnerabilities yeedster/airbyte#86

Open

[Snyk] Fix for 5 vulnerabilities yeedster/airbyte#87

Open

[Snyk] Fix for 5 vulnerabilities yeedster/airbyte#88

Open

[Snyk] Fix for 5 vulnerabilities yeedster/airbyte#89

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incremental Docs and Data Model Update #1021

Incremental Docs and Data Model Update #1021

cgardens commented Nov 19, 2020

cgardens Nov 19, 2020

sherifnada Nov 19, 2020

cgardens Nov 19, 2020

sherifnada Nov 19, 2020

sherifnada Nov 19, 2020

sherifnada Nov 19, 2020

cgardens Nov 19, 2020

sherifnada Nov 19, 2020

sherifnada Nov 19, 2020

cgardens Nov 19, 2020

jrhizor Nov 19, 2020

cgardens Nov 19, 2020

	Some sources are able to determine the cursor that the use without any user input. For example, in the exchange rates api source, the source itself can determine that date field is in fact the field to be used to determine the last record that was synced. In these cases, the source will set the `cursor_field` attribute in the `AirbyteStream`.
	Some sources are able to determine the cursor that they use without any user input. For example, in the exchange rates api source, the source itself can determine that date field is in fact the field to be used to determine the last record that was synced. In these cases, the source will set the `cursor_field` attribute in the `AirbyteStream`.


		## Overview

		Incremental syncs in Airbyte allow sources to replicate only new or modified data. This prevents re-sending data that has already been sent to your warehouse. We will call this set of new or updated records the delta going forward.

Incremental Docs and Data Model Update #1021

Incremental Docs and Data Model Update #1021

Conversation

cgardens commented Nov 19, 2020

What

How

Checklist

Recommended reading order

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment