Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Basic normalization doesn't extract nested objects using $ref/$defs #8241

Closed
monai opened this issue Nov 24, 2021 · 4 comments
Closed

Basic normalization doesn't extract nested objects using $ref/$defs #8241

monai opened this issue Nov 24, 2021 · 4 comments
Labels
community connectors/source/linnworks connectors/sources-api normalization type/bug Something isn't working

Comments

@monai
Copy link
Contributor

monai commented Nov 24, 2021

Enviroment

  • Airbyte version: 0.32.6-alpha
  • OS Version / Instance: macOS
  • Deployment: Docker
  • Source Connector and version: Linnworks 0.1.2
  • Destination Connector and version: Postgres 0.3.12
  • Severity: Normal
  • Step where error happened: Sync job## Current Behavior

Current Behavior

Basic normalization doesn't extract nested objects when they're referenced using $ref and defined using $defs in the same schema. The current workaround is to inline all, usually duplicated, object definitions.

Example schema that uses $ref/$defs: 5303b0b

Expected Behavior

I expect to deduplicate schemas and have all the benefits of basic normalization nested object extraction.

Logs

Failed nested object extraction:

�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - WARN: Unknown type for column Address at processed_order_details/CustomerInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - WARN: Unknown type for column BillingAddress at processed_order_details/CustomerInfo
LOG 
2021-11-22 13:40:15 INFO () WorkerRun(call):49 - Executing worker wrapper. Airbyte version: 0.32.0-alpha
2021-11-22 13:40:16 INFO () TemporalAttemptExecution(get):116 - Executing worker wrapper. Airbyte version: 0.32.0-alpha
2021-11-22 13:40:16 WARN () Databases(createPostgresDatabaseWithRetry):41 - Waiting for database to become available...
2021-11-22 13:40:16 INFO () JobsDatabaseInstance(lambda$static$2):25 - Testing if jobs database is ready...
2021-11-22 13:40:16 INFO () Databases(createPostgresDatabaseWithRetry):58 - Database available!
2021-11-22 13:40:16 WARN () JsonMetaSchema(newValidator):338 - Unknown keyword example - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2021-11-22 13:40:16 INFO () DefaultReplicationWorker(run):82 - start sync worker. job id: 232 attempt id: 0
2021-11-22 13:40:16 INFO () DefaultReplicationWorker(run):91 - configured sync modes: {null.stock_items=full_refresh - append, null.processed_order_details=full_refresh - append, null.processed_orders=full_refresh - append, null.stock_locations=full_refresh - append}
2021-11-22 13:40:16 INFO () DefaultAirbyteDestination(start):64 - Running destination...
2021-11-22 13:40:16 INFO () LineGobbler(voidCall):82 - Checking if airbyte/destination-postgres:0.3.12 exists...
2021-11-22 13:40:16 INFO () LineGobbler(voidCall):82 - airbyte/destination-postgres:0.3.12 not found locally. Attempting to pull the image...
2021-11-22 13:40:36 INFO () LineGobbler(voidCall):82 - Pulled airbyte/destination-postgres:0.3.12 from remote.
2021-11-22 13:40:36 INFO () DockerProcessFactory(create):127 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/232/0 --network host --log-driver none airbyte/destination-postgres:0.3.12 write --config destination_config.json --catalog destination_catalog.json
2021-11-22 13:40:36 INFO () LineGobbler(voidCall):82 - Checking if airbyte/source-linnworks:dev exists...
2021-11-22 13:40:36 INFO () LineGobbler(voidCall):82 - airbyte/source-linnworks:dev was found locally.
2021-11-22 13:40:36 INFO () DockerProcessFactory(create):127 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/232/0 --network host --log-driver none airbyte/source-linnworks:dev read --config source_config.json --catalog source_catalog.json
2021-11-22 13:40:36 INFO () DefaultReplicationWorker(lambda$getDestinationOutputRunnable$3):226 - Destination output thread started.
2021-11-22 13:40:36 INFO () DefaultReplicationWorker(run):119 - Waiting for source thread to join.
2021-11-22 13:40:36 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):190 - Replication thread started.
�[35mdestination�[0m - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 �[32mINFO�[m i.a.i.d.p.PostgresDestination(main):69 - {} - starting destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
�[35mdestination�[0m - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):76 - {} - Running integration: io.airbyte.integrations.base.ssh.SshWrappedDestination
�[35mdestination�[0m - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 �[32mINFO�[m i.a.i.b.IntegrationCliParser(parseOptions):118 - {} - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json}
�[35mdestination�[0m - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):80 - {} - Command: WRITE
�[35mdestination�[0m - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):81 - {} - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'}
�[35mdestination�[0m - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
�[35mdestination�[0m - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
�[35mdestination�[0m - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
�[35mdestination�[0m - 2021-11-22 13:40:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:42 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
�[35mdestination�[0m - 2021-11-22 13:40:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:43 �[32mINFO�[m i.a.i.b.s.SshTunnel(getInstance):170 - {} - Starting connection with method: NO_TUNNEL
�[35mdestination�[0m - 2021-11-22 13:40:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:43 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$toWriteConfig$0):96 - {} - Write config: WriteConfig{streamName=processed_order_details, namespace=null, outputSchemaName=linnworks, tmpTableName=_airbyte_tmp_ctb_processed_order_details, outputTableName=_airbyte_raw_processed_order_details, syncMode=append}
�[35mdestination�[0m - 2021-11-22 13:40:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:43 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$toWriteConfig$0):96 - {} - Write config: WriteConfig{streamName=processed_orders, namespace=null, outputSchemaName=linnworks, tmpTableName=_airbyte_tmp_dok_processed_orders, outputTableName=_airbyte_raw_processed_orders, syncMode=append}
�[35mdestination�[0m - 2021-11-22 13:40:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:43 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$toWriteConfig$0):96 - {} - Write config: WriteConfig{streamName=stock_items, namespace=null, outputSchemaName=linnworks, tmpTableName=_airbyte_tmp_yan_stock_items, outputTableName=_airbyte_raw_stock_items, syncMode=append}
�[35mdestination�[0m - 2021-11-22 13:40:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:43 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$toWriteConfig$0):96 - {} - Write config: WriteConfig{streamName=stock_locations, namespace=null, outputSchemaName=linnworks, tmpTableName=_airbyte_tmp_trv_stock_locations, outputTableName=_airbyte_raw_stock_locations, syncMode=append}
�[35mdestination�[0m - 2021-11-22 13:40:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:43 �[32mINFO�[m i.a.i.d.b.BufferedStreamConsumer(startTracked):124 - {} - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started.
�[35mdestination�[0m - 2021-11-22 13:40:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:43 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):121 - {} - Preparing tmp tables in destination started for 4 streams
�[35mdestination�[0m - 2021-11-22 13:40:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:43 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):125 - {} - Preparing tmp table in destination started for stream processed_order_details. schema: linnworks, tmp table name: _airbyte_tmp_ctb_processed_order_details
�[35mdestination�[0m - 2021-11-22 13:40:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:44 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):125 - {} - Preparing tmp table in destination started for stream processed_orders. schema: linnworks, tmp table name: _airbyte_tmp_dok_processed_orders
�[35mdestination�[0m - 2021-11-22 13:40:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:44 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):125 - {} - Preparing tmp table in destination started for stream stock_items. schema: linnworks, tmp table name: _airbyte_tmp_yan_stock_items
�[35mdestination�[0m - 2021-11-22 13:40:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:44 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):125 - {} - Preparing tmp table in destination started for stream stock_locations. schema: linnworks, tmp table name: _airbyte_tmp_trv_stock_locations
�[35mdestination�[0m - 2021-11-22 13:40:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:40:44 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):131 - {} - Preparing tables in destination completed.
2021-11-22 13:40:51 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 1000
2021-11-22 13:41:03 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 2000
2021-11-22 13:41:18 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 3000
2021-11-22 13:41:30 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 4000
2021-11-22 13:41:53 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 5000
2021-11-22 13:42:08 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 6000
2021-11-22 13:42:32 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 7000
2021-11-22 13:42:57 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 8000
2021-11-22 13:43:27 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 9000
2021-11-22 13:43:49 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 10000
2021-11-22 13:44:23 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 11000
2021-11-22 13:45:08 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 12000
2021-11-22 13:45:45 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 13000
2021-11-22 13:46:14 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 14000
2021-11-22 13:46:26 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 15000
2021-11-22 13:46:48 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 16000
2021-11-22 13:47:10 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 17000
2021-11-22 13:47:23 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 18000
2021-11-22 13:47:41 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 19000
2021-11-22 13:48:01 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 20000
2021-11-22 13:48:20 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 21000
2021-11-22 13:48:36 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 22000
2021-11-22 13:48:47 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 23000
2021-11-22 13:49:04 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 24000
2021-11-22 13:49:23 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 25000
2021-11-22 13:49:47 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 26000
2021-11-22 13:50:06 INFO () DefaultReplicationWorker(run):121 - Source thread complete.
2021-11-22 13:50:06 INFO () DefaultReplicationWorker(run):122 - Waiting for destination thread to join.
�[35mdestination�[0m - 2021-11-22 13:50:06 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:06 �[32mINFO�[m i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):60 - {} - Airbyte message consumer: succeeded.
�[35mdestination�[0m - 2021-11-22 13:50:06 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:06 �[32mINFO�[m i.a.i.d.b.BufferedStreamConsumer(close):199 - {} - executing on success close procedure.
�[35mdestination�[0m - 2021-11-22 13:50:23 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:23 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):160 - {} - Finalizing tables in destination started for 4 streams
�[35mdestination�[0m - 2021-11-22 13:50:23 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:23 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):165 - {} - Finalizing stream processed_order_details. schema linnworks, tmp table _airbyte_tmp_ctb_processed_order_details, final table _airbyte_raw_processed_order_details
�[35mdestination�[0m - 2021-11-22 13:50:23 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:23 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):165 - {} - Finalizing stream processed_orders. schema linnworks, tmp table _airbyte_tmp_dok_processed_orders, final table _airbyte_raw_processed_orders
�[35mdestination�[0m - 2021-11-22 13:50:23 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:23 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):165 - {} - Finalizing stream stock_items. schema linnworks, tmp table _airbyte_tmp_yan_stock_items, final table _airbyte_raw_stock_items
�[35mdestination�[0m - 2021-11-22 13:50:23 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:23 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):165 - {} - Finalizing stream stock_locations. schema linnworks, tmp table _airbyte_tmp_trv_stock_locations, final table _airbyte_raw_stock_locations
�[35mdestination�[0m - 2021-11-22 13:50:23 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:23 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):178 - {} - Executing finalization of tables.
�[35mdestination�[0m - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):180 - {} - Finalizing tables in destination completed.
�[35mdestination�[0m - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):183 - {} - Cleaning tmp tables in destination started for 4 streams
�[35mdestination�[0m - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):187 - {} - Cleaning tmp table in destination started for stream processed_order_details. schema linnworks, tmp table name: _airbyte_tmp_ctb_processed_order_details
�[35mdestination�[0m - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):187 - {} - Cleaning tmp table in destination started for stream processed_orders. schema linnworks, tmp table name: _airbyte_tmp_dok_processed_orders
�[35mdestination�[0m - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):187 - {} - Cleaning tmp table in destination started for stream stock_items. schema linnworks, tmp table name: _airbyte_tmp_yan_stock_items
�[35mdestination�[0m - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):187 - {} - Cleaning tmp table in destination started for stream stock_locations. schema linnworks, tmp table name: _airbyte_tmp_trv_stock_locations
�[35mdestination�[0m - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):192 - {} - Cleaning tmp tables in destination completed.
�[35mdestination�[0m - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):133 - {} - Completed integration: io.airbyte.integrations.base.ssh.SshWrappedDestination
�[35mdestination�[0m - 2021-11-22 13:50:27 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 13:50:27 �[32mINFO�[m i.a.i.d.p.PostgresDestination(main):71 - {} - completed destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
2021-11-22 13:50:28 INFO () DefaultReplicationWorker(run):124 - Destination thread complete.
2021-11-22 13:50:28 INFO () DefaultReplicationWorker(run):152 - sync summary: io.airbyte.config.ReplicationAttemptSummary@52540356[status=completed,recordsSynced=26782,bytesSynced=66088631,startTime=1637588416788,endTime=1637589028283]
2021-11-22 13:50:28 INFO () DefaultReplicationWorker(run):161 - Source did not output any state messages
2021-11-22 13:50:28 WARN () DefaultReplicationWorker(run):172 - State capture: No state retained.
2021-11-22 13:50:28 INFO () TemporalAttemptExecution(get):137 - Stopping cancellation check scheduling...
2021-11-22 13:50:28 INFO () ReplicationActivityImpl(replicate):121 - sync summary: io.airbyte.config.StandardSyncOutput@59e5da9f[standardSyncSummary=io.airbyte.config.StandardSyncSummary@15220a54[status=completed,recordsSynced=26782,bytesSynced=66088631,startTime=1637588416788,endTime=1637589028283],state=<null>,outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@6d3f01ea[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@3d2c4068[stream=io.airbyte.protocol.models.AirbyteStream@4a6a5470[name=processed_order_details,jsonSchema={"type":"object","$defs":{"customer_address":{"type":"object","properties":{"Town":{"type":"string","description":"Customer's town."},"Region":{"type":"string","description":"Customer's region."},"Company":{"type":"string","description":"Customer's company name."},"Country":{"type":"string","description":"Customer's country."},"Address1":{"type":"string","description":"First line of customer address."},"Address2":{"type":"string","description":"Second line of customer address."},"Address3":{"type":"string","description":"Third line of customer address."},"FullName":{"type":"string","description":"Customer's first and second name."},"PostCode":{"type":"string","description":"Customer's postcode."},"Continent":{"type":"string","description":"Customer's continent"},"CountryId":{"type":"string"},"PhoneNumber":{"type":"string","description":"Customer's telephone number."},"EmailAddress":{"type":"string","description":"Customer's email address."}},"additionalProperties":false}},"$schema":"http://json-schema.org/draft-07/schema#","properties":{"Items":{"type":"array","items":{"type":"object","properties":{"SKU":{"type":"string","description":"Product SKU"},"Tax":{"type":"number","description":"Actual tax value on an item"},"Cost":{"type":"number","description":"Total item cost (exc tax)"},"Boxes":{"type":"array","items":{"type":"object","properties":{"BoxId":{"type":"integer","description":"Unique box id."},"Width":{"type":"number","description":"Width of the box"},"Height":{"type":"number","description":"Height of the box"},"Length":{"type":"number","description":"Depth of the box"},"Weight":{"type":"number","description":"Total weight of the box."},"Barcode":{"type":"string","description":"Box barcode, max 64 characters."},"BoxName":{"type":"string","description":"Box name max 16 characters"},"LogicalDelete":{"type":"boolean","description":"IsDeleted flag."},"StockItemIntId":{"type":"integer"},"PackagingTypeId":{"type":"string","description":"Packaging type id"},"ValuePercentage":{"type":"number","description":"Value break down percentage"}},"additionalProperties":false}},"Level":{"type":"integer","description":"Current stock level"},"RowId":{"type":"string","description":"Record row ID"},"Title":{"type":"string","description":"Item title"},"ItemId":{"type":"string","description":"Stock Item ID"},"Market":{"type":"integer","description":"Market"},"Weight":{"type":"number","description":"Order weight"},"BinRack":{"type":"string","description":"Binrack location"},"ImageId":{"type":["null","string"],"description":"Image ID"},"OnOrder":{"type":"integer","description":"Level due in purchase orders"},"OrderId":{"type":"string","description":"Order ID (pkOrderID)"},"StockId":{"type":"string"},"TaxRate":{"type":"number","description":"Tax rate"},"BinRacks":{"type":"array","items":{"type":"object","properties":{"BatchId":{"type":["null","integer"],"description":"If the item is batched, identifies the batch number"},"BinRack":{"type":"string","description":"BinRack"},"Location":{"type":"string","description":"LocationId of the BinRack"},"Quantity":{"type":"integer","description":"Quantity for BinRack per Location"},"OrderItemBatchId":{"type":["null","integer"],"description":"If the item is batched, identifies the unique order item batch row"}},"additionalProperties":false},"description":"List of BinRacks used for OrderItem"},"Discount":{"type":"number","description":"Percentage (0%, 10%, 20%, etc...)"},"HasImage":{"type":"boolean","description":"If item got an image"},"ItemName":{"type":"string","description":"PartShippedQty"},"Quantity":{"type":"integer","description":"Quantity"},"SalesTax":{"type":"number","description":"Sales Tax"},"UnitCost":{"type":"number","description":"Unit cost"},"IsService":{"type":"boolean","description":"if item is a service"},"CategoryId":{"type":"string"},"ChannelSKU":{"type":"string","description":"Channel product SKU"},"CostIncTax":{"type":"number","description":"Total item cost (inc tax)"},"IsUnlinked":{"type":"boolean"},"ItemNumber":{"type":"string","description":"Item number as on channel"},"ItemSource":{"type":"string","description":"Item source / channel name"},"InOrderBook":{"type":["null","integer"],"description":"Quantity currently in open orders"},"PartShipped":{"type":"boolean","description":"If order is partly shipped"},"StockItemId":{"type":"string","description":"Stock Item ID"},"CategoryName":{"type":"string","description":"Product category"},"ChannelTitle":{"type":"string","description":"Channel product title"},"MinimumLevel":{"type":["null","integer"],"description":"Minimum level"},"ParentItemId":{"type":"string"},"PricePerUnit":{"type":"number","description":"Unit price"},"ShippingCost":{"type":"number","description":"If batch number scan required"},"BarcodeNumber":{"type":"string","description":"Product barcode"},"DiscountValue":{"type":"number"},"AdditionalInfo":{"type":"array","items":{"type":"object","properties":{"Value":{"type":"string","description":"Value of the option"},"Property":{"type":"string","description":"Option property"},"pkOptionId":{"type":"string","description":"Option ID"}},"additionalProperties":false},"description":"List of order item options"},"AvailableStock":{"type":"integer","description":"Currently available stock level (Level-InOrderBook)"},"PartShippedQty":{"type":"integer","description":"ShippingCost"},"StockItemIntId":{"type":"integer"},"OnPurchaseOrder":{"type":"object","properties":{"Rowid":{"type":"string"},"SupplierName":{"type":"string"},"fkLocationId":{"type":"string"},"fkSupplierId":{"type":"string"},"pkPurchaseId":{"type":"string"},"DateOfDelivery":{"type":"string","format":"date-time"},"pkPurchaseItemId":{"type":"string","description":"Primary key of the bound"},"QuotedDeliveryDate":{"type":"string","format":"date-time"},"ExternalInvoiceNumber":{"type":"string"}},"description":"Purchase order bound to this item","additionalProperties":false},"TaxCostInclusive":{"type":"boolean","description":"If tax is included in a cost"},"CompositeSubItems":{"type":"array","items":{"type":"object"},"$comment":"It should be \"$ref\": \"#/properties/Items\" but Airbyte doesn't support recursive $refs."},"IsWarehouseManaged":{"type":"boolean"},"isBatchedStockItem":{"type":"boolean","description":"If item has batches"},"StockLevelIndicator":{"type":"integer","description":"Stock level indicator"},"CompositeAvailablity":{"type":["null","integer"],"description":"Composite availability"},"StockLevelsSpecified":{"type":"boolean","description":"If stock level specified"},"DespatchStockUnitCost":{"type":"number","description":"Despatch stock unit cost"},"InventoryTrackingType":{"type":"integer","description":"Identifies whether the item has a sell by date or other defined order in which inventory is to be sold"},"BatchNumberScanRequired":{"type":"boolean","description":"ItemName"},"SerialNumberScanRequired":{"type":"boolean","description":"If serial number scan required"}},"additionalProperties":false},"description":"List of order items"},"Notes":{"type":"array","items":{"type":"object","properties":{"Note":{"type":"string","description":"Note's text"},"OrderId":{"type":"string","description":"Order Id"},"Internal":{"type":"boolean","description":"order note type (Internal or External)"},"NoteDate":{"type":"string","format":"date-time","description":"Date and time when note was added"},"CreatedBy":{"type":"string","description":"User that created note"},"NoteTypeId":{"type":["null","string"]},"OrderNoteId":{"type":"string","description":"Order note ID"}},"additionalProperties":false},"description":"List of order notes"},"TaxId":{"type":"string","description":"Buyer's tax number."},"OrderId":{"type":"string","description":"Order ID (pkOrderId)"},"Processed":{"type":"boolean","description":"If order is processed"},"FolderName":{"type":"array","items":{"type":"string"},"description":"Folder names assigned to an order"},"NumOrderId":{"type":"integer","description":"Linnworks order number"},"TotalsInfo":{"type":"object","properties":{"Tax":{"type":"number","description":"Tax"},"Currency":{"type":"string","description":"Order currency"},"Subtotal":{"type":"number","description":"Order subtotal"},"pkOrderId":{"type":"string","description":"Order Id"},"PostageCost":{"type":"number","description":"Order postage cost"},"TotalCharge":{"type":"number","description":"Total charge"},"ProfitMargin":{"type":"number","description":"Profit margin"},"PaymentMethod":{"type":"string","description":"Payment method"},"TotalDiscount":{"type":"number","description":"Total discount applied to the order"},"ConversionRate":{"type":"number","description":"Currency conversion rate. Set at point of save by the currency"},"CountryTaxRate":{"type":"number","description":"Country tax rate"},"PaymentMethodId":{"type":"string","description":"Payment method ID"},"PostageCostExTax":{"type":"number","description":"Order postage cost ex. tax"}},"description":"Order totals information","additionalProperties":false},"GeneralInfo":{"type":"object","properties":{"Notes":{"type":"integer","description":"Quantity of order notes"},"Marker":{"type":["null","integer"],"description":"Order marker (0 = NOT TAG, 1 = Tag 1, 2 = Tag 2, 3 = Tag 3, 4 = Tag 4, 5 = Tag 5, 6 = Tag 6, 7 = Parked)"},"Source":{"type":"string","description":"Order ChannelName/Source (e.g. EBAY)"},"Status":{"type":"integer","description":"Order Status (0 = UNPAID, 1 = PAID, 2 = RETURN, 3 = PENDING, 4 = RESEND)"},"IsParked":{"type":"boolean","description":"Is the order parked?"},"Location":{"type":"string","description":"Order location ID"},"NumItems":{"type":"integer","description":"Quantity of order items"},"SiteCode":{"type":"string","description":"SiteCode used to differentiate between different sites from a single channel (eg. Amazon UK, Amazon US, Amazon FR...)"},"IsRuleRun":{"type":"boolean","description":"If rules engine rule ran on an order"},"SubSource":{"type":"string","description":"Order Subsource (e.g. EBAY1)"},"LabelError":{"type":"string","description":"Is there a label error"},"Identifiers":{"type":"array","items":{"type":"object","properties":{"Tag":{"type":"string","description":"Internal tag for identification purposes"},"Name":{"type":"string","description":"Name displayed where the tag is used"},"ImageId":{"type":"string"},"ImageUrl":{"type":"string"},"IsCustom":{"type":"boolean","description":"Is the tag user or system defined?"},"IdentifierId":{"type":"integer","description":"Internal identifier id. Use to update image and name."}},"additionalProperties":false},"description":"Order identifiers. [Prime | Scheduled]"},"PartShipped":{"type":"boolean","description":"If order partly shipped"},"PickwaveIds":{"type":"array","items":{"type":"integer"},"description":"All related Pickwave Ids"},"HoldOrCancel":{"type":"boolean","description":"This shows whether the order has been marked as on hold, for processed orders if the order has been cancelled OnHold = 1"},"LabelPrinted":{"type":"boolean","description":"Is label printed"},"ReceivedDate":{"type":"string","format":"date-time","description":"The date and time at which the order was placed on the sales channel"},"ReferenceNum":{"type":"string","description":"Order reference number (Channel defined)"},"DespatchByDate":{"type":"string","format":"date-time","description":"Despatch by Date"},"InvoicePrinted":{"type":"boolean","description":"Is invoice printed"},"PickListPrinted":{"type":"boolean","description":"Is pick list printed"},"ScheduledDelivery":{"type":"object","properties":{"To":{"type":"string","format":"date-time"},"From":{"type":"string","format":"date-time"}},"description":"Scheduled delivery dates. Take priority over despatch by date","additionalProperties":false},"SecondaryReference":{"type":"string","description":"An additional reference number for the orderr (Used by some channels)"},"StockAllocationType":{"type":["null","string"]},"ExternalReferenceNum":{"type":"string","description":"This is an additional reference number from the sales channel, typically used by eBay"},"HasScheduledDelivery":{"type":"boolean"}},"description":"General information about order","additionalProperties":false},"CustomerInfo":{"type":"object","properties":{"Address":{"$ref":"#/definitions/#/$defs/customer_address"},"BillingAddress":{"$ref":"#/definitions/#/$defs/customer_address"},"ChannelBuyerName":{"type":"string","description":"Username of customer (Comes from channel)"}},"description":"Order Customer information (Name, email etc)","additionalProperties":false},"PaidDateTime":{"type":["null","string"],"format":"date-time","description":"Date and time when the order was marked as paid"},"ShippingInfo":{"type":"object","properties":{"Vendor":{"type":"string","description":"Courier name (e.g. Royal Mail)"},"ItemWeight":{"type":"number","description":"If order is processed"},"PackageType":{"type":"string","description":"Package type name"},"PostageCost":{"type":"number","description":"Order postage cost"},"TotalWeight":{"type":"number","description":"Order total weight"},"ManualAdjust":{"type":"boolean","description":"If there is an adjustment to shipping cost was made"},"PackageTypeId":{"type":["null","string"],"description":"Package type ID"},"TrackingNumber":{"type":"string","description":"Order tracking number provided by courier"},"PackageCategory":{"type":"string","description":"Package category name"},"PostalServiceId":{"type":"string","description":"Postal service ID"},"PostageCostExTax":{"type":"number","description":"Order postage cost excluding tax"},"PackageCategoryId":{"type":"string","description":"Package category ID"},"PostalServiceName":{"type":"string","description":"Postal service name (e.g. Next day delivery)"}},"description":"Order shipping information","additionalProperties":false},"ProcessedDateTime":{"type":["null","string"],"format":"date-time","description":"Date and time when order was processed"},"ExtendedProperties":{"type":"array","items":{"type":"object","properties":{"Name":{"type":"string","description":"Extended property name"},"Type":{"type":"string","description":"Extended property type"},"RowId":{"type":"string","description":"Record row ID"},"Value":{"type":"string","description":"Extended property value"}},"additionalProperties":false},"description":"Extended properties of an order"},"FulfilmentLocationId":{"type":"string","description":"Location ID"}},"definitions":{"#/$defs/customer_address":{"type":"object","properties":{"Town":{"type":"string","description":"Customer's town."},"Region":{"type":"string","description":"Customer's region."},"Company":{"type":"string","description":"Customer's company name."},"Country":{"type":"string","description":"Customer's country."},"Address1":{"type":"string","description":"First line of customer address."},"Address2":{"type":"string","description":"Second line of customer address."},"Address3":{"type":"string","description":"Third line of customer address."},"FullName":{"type":"string","description":"Customer's first and second name."},"PostCode":{"type":"string","description":"Customer's postcode."},"Continent":{"type":"string","description":"Customer's continent"},"CountryId":{"type":"string"},"PhoneNumber":{"type":"string","description":"Customer's telephone number."},"EmailAddress":{"type":"string","description":"Customer's email address."}},"additionalProperties":false}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=<null>,defaultCursorField=[],sourceDefinedPrimaryKey=[[NumOrderId]],namespace=<null>,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=append,primaryKey=[[NumOrderId]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@520b2f31[stream=io.airbyte.protocol.models.AirbyteStream@3ab55142[name=processed_orders,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"Town":{"type":"string","description":"Town"},"fTax":{"type":"number","description":"Order tax"},"TaxId":{"type":"string","description":"Order tax id"},"Region":{"type":"string","description":"Region, county, area"},"Source":{"type":"string","description":"ChannelName/Source (e.g. EBAY)"},"Vendor":{"type":"string","description":"Courier name (e.g. DPD)"},"Company":{"type":"string","description":"Company"},"dPaidOn":{"type":"string","format":"date-time","description":"When order was marked as PAID"},"nStatus":{"type":"integer","description":"Order status"},"Address1":{"type":"string","description":"Order first line of address"},"Address2":{"type":"string","description":"Order second line of address"},"Address3":{"type":"string","description":"Order third line of address"},"IsResend":{"type":"boolean","description":"If order was created from a resend"},"Subtotal":{"type":"number","description":"Order subtotal"},"cCountry":{"type":"string","description":"Country"},"nOrderId":{"type":"integer","description":"Linnworks order ID"},"timeDiff":{"type":"number","description":"Days elapsed between order received and order processed"},"SubSource":{"type":"string","description":"Order subsource (e.g. EBAY1)"},"cCurrency":{"type":"string","description":"Order currency"},"cFullName":{"type":"string","description":"Customer full name"},"cPostCode":{"type":"string","description":"Post Code"},"pkOrderID":{"type":"string","description":"Order ID"},"IsExchange":{"type":"boolean","description":"If order was created from an exchange"},"ItemWeight":{"type":"number","description":"Items weight"},"AccountName":{"type":"string","description":"Customer channel account name"},"BillingName":{"type":"string","description":"Customer billing name"},"BillingTown":{"type":"string","description":"Billing town"},"TotalWeight":{"type":"number","description":"Total order weight"},"HoldOrCancel":{"type":"boolean","description":"If order on hold or cancelled."},"PackageTitle":{"type":"string","description":"Package name"},"ProfitMargin":{"type":"number","description":"Profit margin"},"ReferenceNum":{"type":"string","description":"Order reference number"},"dCancelledOn":{"type":"string","format":"date-time","description":"When order was cancelled"},"dProcessedOn":{"type":"string","format":"date-time","description":"Date when order was processed"},"fPostageCost":{"type":"number","description":"Order postage cost"},"fTotalCharge":{"type":"number","description":"Order total charge"},"BillingRegion":{"type":"string","description":"Billing region, area, county"},"TotalDiscount":{"type":"number","description":"Total discount"},"cEmailAddress":{"type":"string","description":"Customer email address"},"dReceivedDate":{"type":"string","format":"date-time","description":"Date when order was received on a channel"},"BillingCompany":{"type":"string","description":"Customer billing company"},"CountryTaxRate":{"type":"number","description":"Country specific tax rate"},"BillingAddress1":{"type":"string","description":"Billing address line one"},"BillingAddress2":{"type":"string","description":"Billing address line two"},"BillingAddress3":{"type":"string","description":"Billing address line three"},"BillingPostCode":{"type":"string","description":"Billing postcode"},"PackageCategory":{"type":"string","description":"Package category"},"cBillingAddress":{"type":"string","description":"Customer billing address"},"BuyerPhoneNumber":{"type":"string","description":"Buyer phone number"},"ChannelBuyerName":{"type":"string","description":"Channel specific name used to identify the buyer, such as a username, normally used for eBay"},"FolderCollection":{"type":"string","description":"Folder name of an order"},"PostageCostExTax":{"type":"number","description":"Postage cost excluding tax"},"cShippingAddress":{"type":"string","description":"Customer's shipping address"},"ExternalReference":{"type":"string","description":"This is an additional reference number from the sales channel, typically used by eBay"},"PostalServiceCode":{"type":"string","description":"Postal service code"},"PostalServiceName":{"type":"string","description":"Postal service name (e.g. Next day delivery)"},"BillingCountryName":{"type":"string","description":"Billing country"},"BillingPhoneNumber":{"type":"string","description":"Billing phone number"},"SecondaryReference":{"type":"string","description":"An additional reference number for the order"},"BillingEmailAddress":{"type":"string"},"PostalTrackingNumber":{"type":"string","description":"Postal tracking number"},"FulfilmentLocationName":{"type":"string","description":"Order fulfilment location"}},"additionalProperties":false},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[dReceivedDate],sourceDefinedPrimaryKey=[[nOrderId]],namespace=<null>,additionalProperties={}],syncMode=full_refresh,cursorField=[dReceivedDate],destinationSyncMode=append,primaryKey=[[nOrderId]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@47a30581[stream=io.airbyte.protocol.models.AirbyteStream@630a1fa2[name=stock_items,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"Depth":{"type":"number","description":"Item depth"},"Width":{"type":"number","description":"Item width"},"Height":{"type":"number","description":"Item height"},"Images":{"type":"array","items":{"type":"object","properties":{"IsMain":{"type":"boolean","description":"Is the image the main image"},"Source":{"type":"string","description":"URL to thumnail image"},"pkRowId":{"type":"string","description":"Unique id of image"},"SortOrder":{"type":"integer","description":"Sort order for the image"},"FullSource":{"type":"string","description":"Url to full size image"},"RawChecksum":{"type":"string","description":"Raw file checksum (Used for UI to determine if the image file is the same before submitting for upload)"},"StockItemId":{"type":"string","description":"Stock Item Id"},"CheckSumValue":{"type":"string","description":"Image check sum"},"ChecksumValue":{"type":"string","description":"Internal checksum value"},"StockItemIntId":{"type":"integer","description":"Stock Item interger Id"}},"additionalProperties":false},"description":"Image urls"},"Weight":{"type":"number","description":"Item weight"},"TaxRate":{"type":"number","description":"Default item tax rate. Set -1 to use country tax rate"},"MetaData":{"type":"string","description":"Item description"},"ItemTitle":{"type":"string","description":"Item title"},"Suppliers":{"type":"array","items":{"type":"object","properties":{"Code":{"type":"string","description":"Supplier code"},"LeadTime":{"type":"integer","description":"Supplier lead time"},"MaxPrice":{"type":"number","description":"Maximum price"},"MinPrice":{"type":"number","description":"Minimum price"},"Supplier":{"type":"string","description":"Supplier name"},"IsDefault":{"type":"boolean","description":"If supplier information is default"},"SupplierID":{"type":"string","description":"Supplier ID"},"StockItemId":{"type":"string","description":"Stock Item Id"},"AveragePrice":{"type":"number","description":"Average price"},"PurchasePrice":{"type":"number","description":"Supplier purchase price"},"StockItemIntId":{"type":"integer","description":"Stock Item integer Id"},"AverageLeadTime":{"type":"number","description":"Average lead time"},"SupplierBarcode":{"type":"string","description":"Supplier barcode"},"SupplierCurrency":{"type":"string","description":"Supplier default currency"},"SupplierPackSize":{"type":"integer","description":"Supplier pack size"},"SupplierMinOrderQty":{"type":"integer","description":"Minimum order quantity from this supplier"}},"additionalProperties":false},"description":"Suppliers"},"CategoryId":{"type":"string","description":"Default category id"},"ItemNumber":{"type":"string","description":"SKU"},"RetailPrice":{"type":["null","number"],"description":"Default item retail price"},"StockItemId":{"type":"string","description":"Stock Item Id"},"StockLevels":{"type":"array","items":{"type":"object","properties":{"Due":{"type":"integer","description":"Due to come in purchase orders"},"JIT":{"type":"boolean","description":"Stock Item Just In Time (JIT) status"},"SKU":{"type":"string","description":"Product SKU"},"rowid":{"type":"string","description":"dbo.StockLevel.rowid"},"InOrders":{"type":"integer","description":"Currently in open orders"},"Location":{"type":"object","properties":{"BinRack":{"type":"string","description":"Bin rack"},"LocationTag":{"type":"string","description":"Location tag"},"LocationName":{"type":"string","description":"Location name"},"StockLocationId":{"type":"string","description":"Location ID"},"IsWarehouseManaged":{"type":["null","boolean"],"description":"If the location is warehouse managed"},"StockLocationIntId":{"type":"integer","description":"Stock location integer id"},"IsFulfillmentCenter":{"type":"boolean","description":"If location is a fulfillment center"}},"description":"Location ID","additionalProperties":false},"UnitCost":{"type":"number","description":"if( Quantity == 0 ) dbo.StockItem.PurchasePrice Else CurrentStockValue / Quantity"},"Available":{"type":"integer","description":"StockLevel - InOrders"},"AutoAdjust":{"type":"boolean","description":"If level is auto adjusted"},"StockLevel":{"type":"integer","description":"Stock level"},"StockValue":{"type":"number","description":"Stock value"},"InOrderBook":{"type":"integer","description":"Currently in open orders"},"StockItemId":{"type":"string","description":"Stock Item Id"},"MinimumLevel":{"type":"integer","description":"Minimum level"},"PendingUpdate":{"type":"boolean","description":"dbo.StockLevel.PendingUpdate"},"LastUpdateDate":{"type":"string","format":"date-time","description":"Last time stock level was adjusted"},"StockItemIntId":{"type":"integer","description":"Stock Item interger Id"},"LastUpdateOperation":{"type":"string","description":"Name of last update operation"},"StockItemPurchasePrice":{"type":"number","description":"Stock item purchase price. It's used to calculate UnitCost"}},"additionalProperties":false},"description":"Stock Levels"},"CategoryName":{"type":"string","description":"Default category name"},"CreationDate":{"type":["null","string"],"format":"date-time","description":"Stock item creation date"},"BarcodeNumber":{"type":"string","description":"Barcode number"},"PurchasePrice":{"type":"number","description":"Default item purchase price"},"PackageGroupId":{"type":"string","description":"Default package group id"},"StockItemIntId":{"type":"integer","description":"Stock Item interger Id"},"PostalServiceId":{"type":"string","description":"Default postal service id"},"PackageGroupName":{"type":"string","description":"Default package group name"},"ItemChannelPrices":{"type":"array","items":{"type":"object","properties":{"Tag":{"type":"string","description":"Product price tag"},"Price":{"type":"number","description":"Channel price"},"Rules":{"type":"array","items":{"type":"object","properties":{"Type":{"type":"string","description":"Type"},"Value":{"type":"number","description":"Value/Price level"},"pkRowId":{"type":["null","integer"],"description":"Record row ID (optional)"},"LowerBound":{"type":"integer","description":"Lower level"},"fkStockPricingId":{"type":"string","description":"Stock pricing ID"}},"additionalProperties":false},"description":"Pricing rule"},"Source":{"type":"string","description":"ChannelName/Source (e.g. EBAY)"},"pkRowId":{"type":"string","description":"Record row ID (generate random)"},"SubSource":{"type":"string","description":"SubSource (e.g. EBAY1)"},"StockItemId":{"type":"string","description":"Stock Item Id"},"UpdateStatus":{"type":"string"},"StockItemIntId":{"type":"integer","description":"Stock Item interger Id"}},"additionalProperties":false},"description":"List of item prices"},"ItemChannelTitles":{"type":"array","items":{"type":"object","properties":{"Title":{"type":"string","description":"Item title"},"Source":{"type":"string","description":"ChannelName/Source (e.g. EBAY)"},"pkRowId":{"type":"string","description":"Record row id (generate random)"},"SubSource":{"type":"string","description":"SubSource (EBAY1)"},"StockItemId":{"type":"string","description":"Stock Item Id"},"StockItemIntId":{"type":"integer","description":"Stock Item integer Id"}},"additionalProperties":false},"description":"List item titles"},"PostalServiceName":{"type":"string","description":"Default postal service name"},"isBatchedStockType":{"type":"boolean","description":"Returns true is the stock item is tracked by batch"},"InventoryTrackingType":{"type":"integer","description":"Stock item tracking type. 0 = none. 1 = Ordered by Sell by Date. 2 = Ordered by Priority Sequence"},"ItemExtendedProperties":{"type":"array","items":{"type":"object","properties":{"pkRowId":{"type":"string","description":"Record row ID (generate random)"},"ProperyName":{"type":"string","description":"Property name"},"PropertyType":{"type":"string","description":"Property type"},"PropertyValue":{"type":"string","description":"Property value"},"fkStockItemId":{"type":"string","description":"Stock Item ID"}},"additionalProperties":false},"description":"List of extended properties"},"BatchNumberScanRequired":{"type":"boolean","description":"User must scan batch number when procesing orders"},"ItemChannelDescriptions":{"type":"array","items":{"type":"object","properties":{"Source":{"type":"string","description":"ChannelName/Source (e.g. EBAY)"},"pkRowId":{"type":"string","description":"Record row ID (generate random GUID)"},"SubSource":{"type":"string","description":"Channel subsource (e.g EBAY1)"},"Description":{"type":"string","description":"Product description"},"StockItemId":{"type":"string","description":"Stock Item Id"},"StockItemIntId":{"type":"integer","description":"Stock Item interger Id"}},"additionalProperties":false},"description":"List of item descriptions"},"SerialNumberScanRequired":{"type":"boolean","description":"User must scan item serial number when processing ordesr"}},"additionalProperties":false},supportedSyncModes=[full_refresh],sourceDefinedCursor=<null>,defaultCursorField=[],sourceDefinedPrimaryKey=[[StockItemIntId]],namespace=<null>,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=append,primaryKey=[[StockItemIntId]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@68f3c655[stream=io.airbyte.protocol.models.AirbyteStream@785ca548[name=stock_locations,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"BinRack":{"type":"string","description":"Bin rack"},"location":{"type":"object","properties":{"City":{"type":"string","description":"City"},"County":{"type":"string","description":"County / Region"},"Country":{"type":"string","description":"Country"},"ZipCode":{"type":"string","description":"Postal code"},"Address1":{"type":"string","description":"1st line of address"},"Address2":{"type":"string","description":"2nd line of address"},"LocationTag":{"type":"string","description":"Location tag"},"LocationName":{"type":"string","description":"Location name"},"IsNotTrackable":{"type":"boolean","description":"Is the location trackable"},"StockLocationId":{"type":"string","description":"Location ID"},"IsWarehouseManaged":{"type":"boolean","description":"Indicates if the location is warehouse managed"},"StockLocationIntId":{"type":"integer","description":"Stock location integer id."},"IsFulfillmentCenter":{"type":"boolean","description":"If location is a fulfillment center"},"CountInOrderUntilAcknowledgement":{"type":"boolean","description":"Count in order"},"FulfilmentCenterDeductStockWhenProcessed":{"type":"boolean","description":"Fulfilment center and stock will be deducted when order processed"}},"additionalProperties":false},"LocationTag":{"type":"string","description":"Location tag"},"LocationName":{"type":"string","description":"Location name"},"StockLocationId":{"type":"string","description":"Location ID"},"IsWarehouseManaged":{"type":["null","boolean"],"description":"If the location is warehouse managed."},"StockLocationIntId":{"type":"integer","description":"Stock location integer id"},"IsFulfillmentCenter":{"type":"boolean","description":"If location is a fulfillment center"}},"additionalProperties":false},supportedSyncModes=[full_refresh],sourceDefinedCursor=<null>,defaultCursorField=[],sourceDefinedPrimaryKey=[[StockLocationIntId]],namespace=<null>,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=append,primaryKey=[[StockLocationIntId]],additionalProperties={}]],additionalProperties={}]]
2021-11-22 13:50:28 INFO () TemporalAttemptExecution(get):116 - Executing worker wrapper. Airbyte version: 0.32.0-alpha
2021-11-22 13:50:28 WARN () Databases(createPostgresDatabaseWithRetry):41 - Waiting for database to become available...
2021-11-22 13:50:28 INFO () JobsDatabaseInstance(lambda$static$2):25 - Testing if jobs database is ready...
2021-11-22 13:50:28 INFO () Databases(createPostgresDatabaseWithRetry):58 - Database available!
2021-11-22 13:50:28 INFO () DefaultNormalizationWorker(run):46 - Running normalization.
2021-11-22 13:50:28 INFO () DefaultNormalizationRunner(runProcess):114 - Running with normalization version: airbyte/normalization:0.1.59
2021-11-22 13:50:28 INFO () LineGobbler(voidCall):82 - Checking if airbyte/normalization:0.1.59 exists...
2021-11-22 13:50:28 INFO () LineGobbler(voidCall):82 - airbyte/normalization:0.1.59 not found locally. Attempting to pull the image...
2021-11-22 13:51:25 INFO () LineGobbler(voidCall):82 - Pulled airbyte/normalization:0.1.59 from remote.
2021-11-22 13:51:25 INFO () DockerProcessFactory(create):127 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/232/0/normalize --network host --log-driver none airbyte/normalization:0.1.59 run --integration-type postgres --config destination_config.json --catalog destination_catalog.json
�[32mnormalization�[0m - 2021-11-22 13:51:38 INFO () LineGobbler(voidCall):82 - Running: transform-config --config destination_config.json --integration-type postgres --out /data/232/0/normalize
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Namespace(config='destination_config.json', integration_type=<DestinationType.postgres: 'postgres'>, out='/data/232/0/normalize')
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - transform_postgres
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Running: transform-catalog --integration-type postgres --profile-config-dir /data/232/0/normalize --catalog destination_catalog.json --out /data/232/0/normalize/models/generated/ --json-column _airbyte_data
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Processing destination_catalog.json...
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Truncating processed_order_details_Items_AdditionalInfo (#44) to processed_order_deta__Items_AdditionalInfo (#43)
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Truncating processed_order_details_Items_OnPurchaseOrder (#45) to processed_order_deta_Items_OnPurchaseOrder (#43)
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Truncating processed_order_details_GeneralInfo_Identifiers (#47) to processed_order_deta_neralInfo_Identifiers (#43)
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Truncating processed_order_details_GeneralInfo_PickwaveIds (#47) to processed_order_deta_neralInfo_PickwaveIds (#43)
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - Truncating processed_order_details_GeneralInfo_ScheduledDelivery (#53) to processed_order_deta_nfo_ScheduledDelivery (#43)
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_ab1.sql from processed_order_details
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_ab2.sql from processed_order_details
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_ab3.sql from processed_order_details
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/processed_order_details.sql from processed_order_details
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_orders_ab1.sql from processed_orders
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_orders_ab2.sql from processed_orders
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_orders_ab3.sql from processed_orders
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/processed_orders.sql from processed_orders
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_ab1.sql from stock_items
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_ab2.sql from stock_items
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_ab3.sql from stock_items
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/stock_items.sql from stock_items
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_locations_ab1.sql from stock_locations
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_locations_ab2.sql from stock_locations
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_locations_ab3.sql from stock_locations
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/stock_locations.sql from stock_locations
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_items_ab1.sql from processed_order_details/Items
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_items_ab2.sql from processed_order_details/Items
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_items_ab3.sql from processed_order_details/Items
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/processed_order_details_items.sql from processed_order_details/Items
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_notes_ab1.sql from processed_order_details/Notes
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_notes_ab2.sql from processed_order_details/Notes
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_notes_ab3.sql from processed_order_details/Notes
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/processed_order_details_notes.sql from processed_order_details/Notes
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Ignoring stream 'FolderName' from processed_order_details/FolderName because properties list is empty
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_totalsinfo_ab1.sql from processed_order_details/TotalsInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_totalsinfo_ab2.sql from processed_order_details/TotalsInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_totalsinfo_ab3.sql from processed_order_details/TotalsInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/processed_order_details_totalsinfo.sql from processed_order_details/TotalsInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_generalinfo_ab1.sql from processed_order_details/GeneralInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_generalinfo_ab2.sql from processed_order_details/GeneralInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_generalinfo_ab3.sql from processed_order_details/GeneralInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/processed_order_details_generalinfo.sql from processed_order_details/GeneralInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_customerinfo_ab1.sql from processed_order_details/CustomerInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - WARN: Unknown type for column Address at processed_order_details/CustomerInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 - WARN: Unknown type for column BillingAddress at processed_order_details/CustomerInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_customerinfo_ab2.sql from processed_order_details/CustomerInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_customerinfo_ab3.sql from processed_order_details/CustomerInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/processed_order_details_customerinfo.sql from processed_order_details/CustomerInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_shippinginfo_ab1.sql from processed_order_details/ShippingInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_shippinginfo_ab2.sql from processed_order_details/ShippingInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_shippinginfo_ab3.sql from processed_order_details/ShippingInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/processed_order_details_shippinginfo.sql from processed_order_details/ShippingInfo
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_extendedproperties_ab1.sql from processed_order_details/ExtendedProperties
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_extendedproperties_ab2.sql from processed_order_details/ExtendedProperties
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_extendedproperties_ab3.sql from processed_order_details/ExtendedProperties
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/processed_order_details_extendedproperties.sql from processed_order_details/ExtendedProperties
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_images_ab1.sql from stock_items/Images
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_images_ab2.sql from stock_items/Images
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_images_ab3.sql from stock_items/Images
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/stock_items_images.sql from stock_items/Images
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_suppliers_ab1.sql from stock_items/Suppliers
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_suppliers_ab2.sql from stock_items/Suppliers
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_suppliers_ab3.sql from stock_items/Suppliers
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/stock_items_suppliers.sql from stock_items/Suppliers
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_stocklevels_ab1.sql from stock_items/StockLevels
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_stocklevels_ab2.sql from stock_items/StockLevels
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_stocklevels_ab3.sql from stock_items/StockLevels
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/stock_items_stocklevels.sql from stock_items/StockLevels
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_itemchannelprices_ab1.sql from stock_items/ItemChannelPrices
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_itemchannelprices_ab2.sql from stock_items/ItemChannelPrices
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_itemchannelprices_ab3.sql from stock_items/ItemChannelPrices
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/stock_items_itemchannelprices.sql from stock_items/ItemChannelPrices
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_itemchanneltitles_ab1.sql from stock_items/ItemChannelTitles
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_itemchanneltitles_ab2.sql from stock_items/ItemChannelTitles
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_itemchanneltitles_ab3.sql from stock_items/ItemChannelTitles
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/stock_items_itemchanneltitles.sql from stock_items/ItemChannelTitles
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_itemextendedproperties_ab1.sql from stock_items/ItemExtendedProperties
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_itemextendedproperties_ab2.sql from stock_items/ItemExtendedProperties
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_itemextendedproperties_ab3.sql from stock_items/ItemExtendedProperties
�[32mnormalization�[0m - 2021-11-22 13:51:39 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/stock_items_itemextendedproperties.sql from stock_items/ItemExtendedProperties
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_itemchanneldescriptions_ab1.sql from stock_items/ItemChannelDescriptions
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_itemchanneldescriptions_ab2.sql from stock_items/ItemChannelDescriptions
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_itemchanneldescriptions_ab3.sql from stock_items/ItemChannelDescriptions
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/stock_items_itemchanneldescriptions.sql from stock_items/ItemChannelDescriptions
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_locations_location_ab1.sql from stock_locations/location
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_locations_location_ab2.sql from stock_locations/location
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_locations_location_ab3.sql from stock_locations/location
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/stock_locations_location.sql from stock_locations/location
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_items_boxes_ab1.sql from processed_order_details/Items/Boxes
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_items_boxes_ab2.sql from processed_order_details/Items/Boxes
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_items_boxes_ab3.sql from processed_order_details/Items/Boxes
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/processed_order_details_items_boxes.sql from processed_order_details/Items/Boxes
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_items_binracks_ab1.sql from processed_order_details/Items/BinRacks
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_items_binracks_ab2.sql from processed_order_details/Items/BinRacks
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_details_items_binracks_ab3.sql from processed_order_details/Items/BinRacks
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/processed_order_details_items_binracks.sql from processed_order_details/Items/BinRacks
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_deta___items_additionalinfo_ab1.sql from processed_order_details/Items/AdditionalInfo
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_deta___items_additionalinfo_ab2.sql from processed_order_details/Items/AdditionalInfo
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_deta___items_additionalinfo_ab3.sql from processed_order_details/Items/AdditionalInfo
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/processed_order_deta___items_additionalinfo.sql from processed_order_details/Items/AdditionalInfo
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_deta__items_onpurchaseorder_ab1.sql from processed_order_details/Items/OnPurchaseOrder
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_deta__items_onpurchaseorder_ab2.sql from processed_order_details/Items/OnPurchaseOrder
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_deta__items_onpurchaseorder_ab3.sql from processed_order_details/Items/OnPurchaseOrder
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/processed_order_deta__items_onpurchaseorder.sql from processed_order_details/Items/OnPurchaseOrder
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_deta__neralinfo_identifiers_ab1.sql from processed_order_details/GeneralInfo/Identifiers
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_deta__neralinfo_identifiers_ab2.sql from processed_order_details/GeneralInfo/Identifiers
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_deta__neralinfo_identifiers_ab3.sql from processed_order_details/GeneralInfo/Identifiers
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/processed_order_deta__neralinfo_identifiers.sql from processed_order_details/GeneralInfo/Identifiers
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Ignoring stream 'PickwaveIds' from processed_order_details/GeneralInfo/PickwaveIds because properties list is empty
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_deta__nfo_scheduleddelivery_ab1.sql from processed_order_details/GeneralInfo/ScheduledDelivery
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_deta__nfo_scheduleddelivery_ab2.sql from processed_order_details/GeneralInfo/ScheduledDelivery
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/processed_order_deta__nfo_scheduleddelivery_ab3.sql from processed_order_details/GeneralInfo/ScheduledDelivery
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/processed_order_deta__nfo_scheduleddelivery.sql from processed_order_details/GeneralInfo/ScheduledDelivery
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_stocklevels_location_ab1.sql from stock_items/StockLevels/Location
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_stocklevels_location_ab2.sql from stock_items/StockLevels/Location
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_stocklevels_location_ab3.sql from stock_items/StockLevels/Location
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/stock_items_stocklevels_location.sql from stock_items/StockLevels/Location
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_itemchannelprices_rules_ab1.sql from stock_items/ItemChannelPrices/Rules
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_itemchannelprices_rules_ab2.sql from stock_items/ItemChannelPrices/Rules
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/linnworks/stock_items_itemchannelprices_rules_ab3.sql from stock_items/ItemChannelPrices/Rules
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/linnworks/stock_items_itemchannelprices_rules.sql from stock_items/ItemChannelPrices/Rules
�[32mnormalization�[0m - 2021-11-22 13:51:40 INFO () LineGobbler(voidCall):82 - detected no config file for ssh, assuming ssh is off.
�[32mnormalization�[0m - 2021-11-22 13:51:43 INFO () LineGobbler(voidCall):82 - Running with dbt=0.21.0
�[32mnormalization�[0m - 2021-11-22 13:51:44 INFO () LineGobbler(voidCall):82 - Unable to do partial parsing because ../build/partial_parse.msgpack not found
�[32mnormalization�[0m - 2021-11-22 13:51:48 INFO () LineGobbler(voidCall):82 - [�[33mWARNING�[0m]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
�[32mnormalization�[0m - 2021-11-22 13:51:48 INFO () LineGobbler(voidCall):82 - There are 2 unused configuration paths:
�[32mnormalization�[0m - 2021-11-22 13:51:48 INFO () LineGobbler(voidCall):82 - - models.airbyte_utils.generated.airbyte_tables
�[32mnormalization�[0m - 2021-11-22 13:51:48 INFO () LineGobbler(voidCall):82 - - models.airbyte_utils.generated.airbyte_views
�[32mnormalization�[0m - 2021-11-22 13:51:48 INFO () LineGobbler(voidCall):82 - 
�[32mnormalization�[0m - 2021-11-22 13:51:48 INFO () LineGobbler(voidCall):82 - Found 108 models, 0 tests, 0 snapshots, 0 analyses, 478 macros, 0 operations, 0 seed files, 4 sources, 0 exposures
�[32mnormalization�[0m - 2021-11-22 13:51:48 INFO () LineGobbler(voidCall):82 - 
�[32mnormalization�[0m - 2021-11-22 13:51:49 INFO () LineGobbler(voidCall):82 - 13:51:49 | Concurrency: 32 threads (target='prod')
�[32mnormalization�[0m - 2021-11-22 13:51:49 INFO () LineGobbler(voidCall):82 - 13:51:49 | 
�[32mnormalization�[0m - 2021-11-22 13:51:49 INFO () LineGobbler(voidCall):82 - 13:51:49 | 1 of 27 START incremental model linnworks.stock_locations.................................................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:50 INFO () LineGobbler(voidCall):82 - 13:51:50 | 2 of 27 START incremental model linnworks.processed_order_details............................................ [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:50 INFO () LineGobbler(voidCall):82 - 13:51:50 | 3 of 27 START incremental model linnworks.stock_items........................................................ [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:50 INFO () LineGobbler(voidCall):82 - 13:51:50 | 4 of 27 START incremental model linnworks.processed_orders................................................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:51 INFO () LineGobbler(voidCall):82 - 13:51:51 | 1 of 27 OK created incremental model linnworks.stock_locations............................................... [�[32mINSERT 0 0�[0m in 1.31s]
�[32mnormalization�[0m - 2021-11-22 13:51:51 INFO () LineGobbler(voidCall):82 - 13:51:51 | 2 of 27 OK created incremental model linnworks.processed_order_details....................................... [�[32mINSERT 0 0�[0m in 0.90s]
�[32mnormalization�[0m - 2021-11-22 13:51:51 INFO () LineGobbler(voidCall):82 - 13:51:51 | 3 of 27 OK created incremental model linnworks.stock_items................................................... [�[32mINSERT 0 0�[0m in 1.16s]
�[32mnormalization�[0m - 2021-11-22 13:51:52 INFO () LineGobbler(voidCall):82 - 13:51:52 | 4 of 27 OK created incremental model linnworks.processed_orders.............................................. [�[32mINSERT 0 0�[0m in 1.32s]
�[32mnormalization�[0m - 2021-11-22 13:51:53 INFO () LineGobbler(voidCall):82 - 13:51:53 | 5 of 27 START incremental model linnworks.stock_locations_location........................................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:53 INFO () LineGobbler(voidCall):82 - 13:51:53 | 6 of 27 START incremental model linnworks.processed_order_details_extendedproperties......................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:54 INFO () LineGobbler(voidCall):82 - 13:51:54 | 7 of 27 START incremental model linnworks.processed_order_details_notes...................................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:54 INFO () LineGobbler(voidCall):82 - 13:51:54 | 8 of 27 START incremental model linnworks.processed_order_details_customerinfo............................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:54 INFO () LineGobbler(voidCall):82 - 13:51:54 | 9 of 27 START incremental model linnworks.processed_order_details_shippinginfo............................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:54 INFO () LineGobbler(voidCall):82 - 13:51:54 | 10 of 27 START incremental model linnworks.processed_order_details_generalinfo............................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:54 INFO () LineGobbler(voidCall):82 - 13:51:54 | 11 of 27 START incremental model linnworks.stock_items_itemextendedproperties................................ [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:54 INFO () LineGobbler(voidCall):82 - 13:51:54 | 12 of 27 START incremental model linnworks.processed_order_details_totalsinfo................................ [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:54 INFO () LineGobbler(voidCall):82 - 13:51:54 | 14 of 27 START incremental model linnworks.stock_items_itemchanneldescriptions............................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:54 INFO () LineGobbler(voidCall):82 - 13:51:54 | 13 of 27 START incremental model linnworks.stock_items_itemchanneltitles..................................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:55 INFO () LineGobbler(voidCall):82 - 13:51:54 | 15 of 27 START incremental model linnworks.stock_items_images................................................ [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:55 INFO () LineGobbler(voidCall):82 - 13:51:54 | 16 of 27 START incremental model linnworks.processed_order_details_items..................................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:55 INFO () LineGobbler(voidCall):82 - 13:51:54 | 17 of 27 START incremental model linnworks.stock_items_suppliers............................................. [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:55 INFO () LineGobbler(voidCall):82 - 13:51:55 | 18 of 27 START incremental model linnworks.stock_items_itemchannelprices..................................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:55 INFO () LineGobbler(voidCall):82 - 13:51:55 | 19 of 27 START incremental model linnworks.stock_items_stocklevels........................................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 5 of 27 OK created incremental model linnworks.stock_locations_location...................................... [�[32mINSERT 0 0�[0m in 3.11s]
�[32mnormalization�[0m - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 6 of 27 OK created incremental model linnworks.processed_order_details_extendedproperties.................... [�[32mINSERT 0 0�[0m in 2.72s]
�[32mnormalization�[0m - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 8 of 27 OK created incremental model linnworks.processed_order_details_customerinfo.......................... [�[32mINSERT 0 0�[0m in 2.58s]
�[32mnormalization�[0m - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 10 of 27 OK created incremental model linnworks.processed_order_details_generalinfo.......................... [�[32mINSERT 0 0�[0m in 2.44s]
�[32mnormalization�[0m - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 11 of 27 OK created incremental model linnworks.stock_items_itemextendedproperties........................... [�[32mINSERT 0 0�[0m in 2.14s]
�[32mnormalization�[0m - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 7 of 27 OK created incremental model linnworks.processed_order_details_notes................................. [�[32mINSERT 0 0�[0m in 2.62s]
�[32mnormalization�[0m - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 9 of 27 OK created incremental model linnworks.processed_order_details_shippinginfo.......................... [�[32mINSERT 0 0�[0m in 2.52s]
�[32mnormalization�[0m - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 12 of 27 OK created incremental model linnworks.processed_order_details_totalsinfo........................... [�[32mINSERT 0 0�[0m in 2.19s]
�[32mnormalization�[0m - 2021-11-22 13:51:56 INFO () LineGobbler(voidCall):82 - 13:51:56 | 14 of 27 OK created incremental model linnworks.stock_items_itemchanneldescriptions.......................... [�[32mINSERT 0 0�[0m in 2.11s]
�[32mnormalization�[0m - 2021-11-22 13:51:57 INFO () LineGobbler(voidCall):82 - 13:51:57 | 16 of 27 OK created incremental model linnworks.processed_order_details_items................................ [�[32mINSERT 0 0�[0m in 1.93s]
�[32mnormalization�[0m - 2021-11-22 13:51:57 INFO () LineGobbler(voidCall):82 - 13:51:57 | 13 of 27 OK created incremental model linnworks.stock_items_itemchanneltitles................................ [�[32mINSERT 0 0�[0m in 2.13s]
�[32mnormalization�[0m - 2021-11-22 13:51:57 INFO () LineGobbler(voidCall):82 - 13:51:57 | 15 of 27 OK created incremental model linnworks.stock_items_images........................................... [�[32mINSERT 0 0�[0m in 1.98s]
�[32mnormalization�[0m - 2021-11-22 13:51:57 INFO () LineGobbler(voidCall):82 - 13:51:57 | 17 of 27 OK created incremental model linnworks.stock_items_suppliers........................................ [�[32mINSERT 0 0�[0m in 2.06s]
�[32mnormalization�[0m - 2021-11-22 13:51:57 INFO () LineGobbler(voidCall):82 - 13:51:57 | 18 of 27 OK created incremental model linnworks.stock_items_itemchannelprices................................ [�[32mINSERT 0 0�[0m in 1.93s]
�[32mnormalization�[0m - 2021-11-22 13:51:57 INFO () LineGobbler(voidCall):82 - 13:51:57 | 19 of 27 OK created incremental model linnworks.stock_items_stocklevels...................................... [�[32mINSERT 0 0�[0m in 1.84s]
�[32mnormalization�[0m - 2021-11-22 13:51:57 INFO () LineGobbler(voidCall):82 - 13:51:57 | 20 of 27 START incremental model linnworks.processed_order_deta__neralinfo_identifiers....................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:57 INFO () LineGobbler(voidCall):82 - 13:51:57 | 21 of 27 START incremental model linnworks.processed_order_deta__nfo_scheduleddelivery....................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:58 INFO () LineGobbler(voidCall):82 - 13:51:58 | 22 of 27 START incremental model linnworks.processed_order_details_items_binracks............................ [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:58 INFO () LineGobbler(voidCall):82 - 13:51:58 | 23 of 27 START incremental model linnworks.processed_order_details_items_boxes............................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:58 INFO () LineGobbler(voidCall):82 - 13:51:58 | 25 of 27 START incremental model linnworks.processed_order_deta__items_onpurchaseorder....................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:58 INFO () LineGobbler(voidCall):82 - 13:51:58 | 24 of 27 START incremental model linnworks.processed_order_deta___items_additionalinfo....................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:58 INFO () LineGobbler(voidCall):82 - 13:51:58 | 26 of 27 START incremental model linnworks.stock_items_itemchannelprices_rules............................... [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:58 INFO () LineGobbler(voidCall):82 - 13:51:58 | 27 of 27 START incremental model linnworks.stock_items_stocklevels_location.................................. [RUN]
�[32mnormalization�[0m - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 20 of 27 OK created incremental model linnworks.processed_order_deta__neralinfo_identifiers.................. [�[32mINSERT 0 0�[0m in 1.26s]
�[32mnormalization�[0m - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 21 of 27 OK created incremental model linnworks.processed_order_deta__nfo_scheduleddelivery.................. [�[32mINSERT 0 0�[0m in 1.22s]
�[32mnormalization�[0m - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 25 of 27 OK created incremental model linnworks.processed_order_deta__items_onpurchaseorder.................. [�[32mINSERT 0 0�[0m in 0.86s]
�[32mnormalization�[0m - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 22 of 27 OK created incremental model linnworks.processed_order_details_items_binracks....................... [�[32mINSERT 0 0�[0m in 1.23s]
�[32mnormalization�[0m - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 27 of 27 OK created incremental model linnworks.stock_items_stocklevels_location............................. [�[32mINSERT 0 0�[0m in 0.68s]
�[32mnormalization�[0m - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 24 of 27 OK created incremental model linnworks.processed_order_deta___items_additionalinfo.................. [�[32mINSERT 0 0�[0m in 0.90s]
�[32mnormalization�[0m - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 23 of 27 OK created incremental model linnworks.processed_order_details_items_boxes.......................... [�[32mINSERT 0 0�[0m in 0.93s]
�[32mnormalization�[0m - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 26 of 27 OK created incremental model linnworks.stock_items_itemchannelprices_rules.......................... [�[32mINSERT 0 0�[0m in 0.81s]
�[32mnormalization�[0m - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | 
�[32mnormalization�[0m - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 13:51:59 | Finished running 27 incremental models in 10.45s.
�[32mnormalization�[0m - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 
�[32mnormalization�[0m - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - �[32mCompleted successfully�[0m
�[32mnormalization�[0m - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - 
�[32mnormalization�[0m - 2021-11-22 13:51:59 INFO () LineGobbler(voidCall):82 - Done. PASS=27 WARN=0 ERROR=0 SKIP=0 TOTAL=27
2021-11-22 13:51:59 INFO () DefaultNormalizationWorker(run):69 - Normalization executed in 1 minute 31 seconds.
2021-11-22 13:51:59 INFO () TemporalAttemptExecution(get):137 - Stopping cancellation check scheduling...

Steps to Reproduce

  1. Setup any source with schema using $ref/$defs, e.g. 5303b0b
  2. Setup destination Postgres
  3. Run sync

Are you willing to submit a PR?

Maybe.
@monai monai added the type/bug Something isn't working label Nov 24, 2021
@alafanechere alafanechere added area/platform issues related to the platform type/bug Something isn't working normalization and removed type/bug Something isn't working needs-triage area/platform issues related to the platform labels Nov 30, 2021
@harshithmullapudi
Copy link
Contributor

@alafanechere do you think this #8330 is also same?

@alafanechere alafanechere removed the area/platform issues related to the platform label Nov 30, 2021
@ChristopheDuong
Copy link
Contributor

we don't support $ref / $defs in airbyte json schema, so normalization does not support that either

The source should resolve and inline the full json into the catalog

@monai
Copy link
Contributor Author

monai commented Nov 30, 2021 •
edited
Loading

we don't support $ref / $defs in airbyte json schema, so normalization does not support that either

The source should resolve and inline the full json into the catalog

Well, actually, Airbyte does support $ref's to other files. Referenced schemas are inlined under and $refs point to #/definitions/{filename}.

See the documentation:
https://docs.airbyte.io/connector-development/cdk-python/schemas#static-schemas

Important note: any objects referenced via $ref should be placed in the shared/ directory in their own .json files.

@ChristopheDuong
Copy link
Contributor

ChristopheDuong commented Nov 30, 2021 •
edited
Loading

Sorry for the misunderstanding!

That's the documentation page for the CDK and how to ease implementation of source connectors in airbyte.

But once the source connector produces the catalog.json in Airbyte's protocol, there should be no more $ref afterward as the CDK/source connector should have inlined all the $ref into a full JSON schema object.

The separated YAML/JSON files (used in the refs) are not transmitted through the protocol (only a full json object called catalog), so normalization would not be able to resolve and inline the different JSON object back together. (that's why I said that it is "not supported")

see this issue #7966 and its associated PRs where we are adding "Source Acceptance Tests" (SAT) to make sure all $ref are resolved by the source connector before "materializing" the catalog.json (that is used downstream by normalization)

@monai monai mentioned this issue Dec 22, 2021
Merged
18 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
community connectors/source/linnworks connectors/sources-api normalization type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@jrhizor @monai @alafanechere @ChristopheDuong @harshithmullapudi @igrankova @octavia-squidington-iii