Code

Airbyte data source [CU-15xp9xt]#493

Code

Merged

mildbyte merged 31 commits intomastersplitgraph/sgr:masterfrom feature/airbyte-data-source-CU-15xp9xtsplitgraph/sgr:feature/airbyte-data-source-CU-15xp9xtCopy head branch name to clipboardJul 21, 2021

+1,794-182

Commits

Commits on Jul 12, 2021

• Move some common Singer ingestion routines into separate functions that can be referenced by non-Singer data sources.

  mildbyte

  committed
  e210ea7

  Copy full SHA for e210ea7

• Allow copy_to_container to take in the actual data rather than just the source path.

  mildbyte

  committed
  60227bc

  Copy full SHA for 60227bc

• Fix circular import: move types into _utils and rename _utils into common

  mildbyte

  committed
  4e97d12

  Copy full SHA for 4e97d12

• Allow empty data.

  mildbyte

  committed
  03f52b8

  Copy full SHA for 03f52b8

Commits on Jul 14, 2021

• Make Image._lq_checkout public, allow it to take in a list of tables to check out.

  mildbyte

  committed
  1c6c8da

  Copy full SHA for 1c6c8da

Commits on Jul 19, 2021

• Initial implementation of an Airbyte <> Splitgraph data source shim

  mildbyte

  committed
  e16fdc2

  Copy full SHA for e16fdc2

• Add Airbyte to the requirements as an extra and install it in tests.

  mildbyte

  committed
  5be1cc2

  Copy full SHA for 5be1cc2

• Break Docker network mode / host detection into a separate routine and use it for source containers too (in test we want to hit the MySQL Docker container from the host).

  mildbyte

  committed
  a6039c6

  Copy full SHA for a6039c6

• Airbyte log parsing fixes: decode the logs in case of an error and emit them; ignore log lines that aren't Airbyte messages.

  mildbyte

  committed
  7f80065

  Copy full SHA for 7f80065

• Add initial test suite for Airbyte (introspection/catalog manip)

  mildbyte

  committed
  7a7725c

  Copy full SHA for 7a7725c

• Use two different configurations for the source and the destination (for the destination, we want to override the namespace since otherwise PG will write out to the wrong schema).

  mildbyte

  committed
  514aa84

  Copy full SHA for 514aa84

• Always stub out the namespace in the messages between the source and the destination.

  mildbyte

  committed
  e82f596

  Copy full SHA for e82f596

• Use a custom image comment for Airbyte-generated images.

  mildbyte

  committed
  7604b6c

  Copy full SHA for 7604b6c

• Misc fixes to incremental loads and errors.

  mildbyte

  committed
  dfe0128

  Copy full SHA for dfe0128

• Add more unit and end-to-end tests for Airbyte loads (uses the MySQL loader as a test).

  mildbyte

  committed
  15598a2

  Copy full SHA for 15598a2

• Add missing __init__.py to Singer

  mildbyte

  committed
  a15d7af

  Copy full SHA for a15d7af

• Delete airbyte-cdk from the deps and instead copy the Pydantic models from it into the codebase, since that's all we need. Update the Poetry lockfile.

  mildbyte

  committed
  439987c

  Copy full SHA for 439987c

• Fix imports in tests and remove the pytest import guard (don't need airbyte-cdk).

  mildbyte

  committed
  d9c4dd8

  Copy full SHA for d9c4dd8

• Add chardet as a dependency (went away after requests was upgraded?)

  mildbyte

  committed
  43bf9f9

  Copy full SHA for 43bf9f9

• [CU-15xp9xt] Delete the airbyte extra from the installation script (not needed)

  mildbyte

  committed
  2511981

  Copy full SHA for 2511981

Commits on Jul 20, 2021

• Use the Airbyte message reader iterator in introspection to avoid issues with Docker log batch size (16364).

  mildbyte

  committed
  578c2f9

  Copy full SHA for 578c2f9

• Move Docker utilities out of splitgraph.commandline to avoid circular imports (since plugins are loaded in the commandline module).

  mildbyte

  committed
  5302bef

  Copy full SHA for 5302bef

• Merge branch 'master' into feature/airbyte-data-source-CU-15xp9xt

  mildbyte

  committed
  9fefa06

  Copy full SHA for 9fefa06

Commits on Jul 21, 2021

• Factor the DEFAULT_CHUNK_SIZE out into a variable and use it throughout various image creation routines instead of ad hoc numbers.

  mildbyte

  committed
  2ef8567

  Copy full SHA for 2ef8567

• Allow overriding the Docker environment in Airbyte containers.

  mildbyte

  committed
  62d4e71

  Copy full SHA for 62d4e71

• Add support for overriding the cursor field / PK in Airbyte-backed data sources (through airbyte_cursor_field and airbyte_primary_key table params). Also, report the plugin's default cursor/PK …

  Show description for 0b091f9
  mildbyte

  committed
  0b091f9

  Copy full SHA for 0b091f9

• Fix table param override flow in Singer too.

  mildbyte

  committed
  6cd734a

  Copy full SHA for 6cd734a

• Add the PK/cursor to the Airbyte stream even when in load mode.

  mildbyte

  committed
  ed4d422

  Copy full SHA for ed4d422

• Change table_schema_params_to_dict to support non-string parameters (table params can be anything that dumps to JSON).

  mildbyte

  committed
  55bdb38

  Copy full SHA for 55bdb38

• Validate table params against the JSONSchema if they're passed into a data source through a function rather than at instantiation time.

  mildbyte

  committed
  cb1b295

  Copy full SHA for cb1b295

• Fix a test (we set the cursor field from the connector's default in load mode too)

  mildbyte

  committed
  8e5c387

  Copy full SHA for 8e5c387