Source Files: decrease give up rate (airbytehq#23723)

* updated spec, docs, error messages * added change log * updated error message in unit tests * added allowedHosts * refactored error message and url descriprion * updated example for url field in spec * bump source-file verion for source-file-secure * auto-bump connector version * updated spec.json for source-file-secure --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
adriennevermorel · Mar 17, 2023 · 3b59c10 · 3b59c10
1 parent c448402
commit 3b59c10
Show file tree

Hide file tree

Showing 10 changed files with 129 additions and 106 deletions.
diff --git a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml
@@ -595,11 +595,14 @@
 - name: File (CSV, JSON, Excel, Feather, Parquet)
   sourceDefinitionId: 778daa7c-feaf-4db6-96f3-70fd645acc77
   dockerRepository: airbyte/source-file
-  dockerImageTag: 0.2.33
+  dockerImageTag: 0.2.34
   documentationUrl: https://docs.airbyte.com/integrations/sources/file
   icon: file.svg
   sourceType: file
   releaseStage: generally_available
+  allowedHosts:
+    hosts:
+      - "*"
 - name: Freshcaller
   sourceDefinitionId: 8a5d48f6-03bb-4038-a942-a8d3f175cca3
   dockerRepository: airbyte/source-freshcaller

diff --git a/airbyte-config/init/src/main/resources/seed/source_specs.yaml b/airbyte-config/init/src/main/resources/seed/source_specs.yaml
@@ -4317,7 +4317,7 @@
     supportsNormalization: false
     supportsDBT: false
     supported_destination_sync_modes: []
-- dockerImage: "airbyte/source-file:0.2.33"
+- dockerImage: "airbyte/source-file:0.2.34"
   spec:
     documentationUrl: "https://docs.airbyte.com/integrations/sources/file"
     connectionSpecification:
@@ -4359,10 +4359,17 @@
           examples:
           - "{}"
           - "{\"sep\": \" \"}"
+          - sep: "\t"
+            header: 0
+            names: "column1, column2"
         url:
           type: "string"
           title: "URL"
           description: "The URL path to access the file which should be replicated."
+          examples:
+          - "https://storage.googleapis.com/covid19-open-data/v2/latest/epidemiology.csv"
+          - "gs://my-google-bucket/data.csv"
+          - "s3://gdelt-open-data/events/20190914.export.csv"
         provider:
           type: "object"
           title: "Storage Provider"

diff --git a/airbyte-integrations/connectors/source-file-secure/Dockerfile b/airbyte-integrations/connectors/source-file-secure/Dockerfile
@@ -1,4 +1,4 @@
-FROM airbyte/source-file:0.2.31
+FROM airbyte/source-file:0.2.34
 
 WORKDIR /airbyte/integration_code
 COPY source_file_secure ./source_file_secure
@@ -9,5 +9,5 @@ RUN pip install .
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.2.33
+LABEL io.airbyte.version=0.2.34
 LABEL io.airbyte.name=airbyte/source-file-secure
diff --git a/airbyte-integrations/connectors/source-file-secure/integration_tests/spec.json b/airbyte-integrations/connectors/source-file-secure/integration_tests/spec.json
@@ -32,12 +32,13 @@
         "type": "string",
         "title": "Reader Options",
         "description": "This should be a string in JSON format. It depends on the chosen file format to provide additional options and tune its behavior.",
-        "examples": ["{}", "{\"sep\": \" \"}"]
+        "examples": ["{}", "{\"sep\": \" \"}", { "sep" : "\t", "header" : 0, "names": "column1, column2"}]
       },
       "url": {
         "type": "string",
         "title": "URL",
-        "description": "The URL path to access the file which should be replicated."
+        "description": "The URL path to access the file which should be replicated.",
+        "examples": ["https://storage.googleapis.com/covid19-open-data/v2/latest/epidemiology.csv", "gs://my-google-bucket/data.csv", "s3://gdelt-open-data/events/20190914.export.csv"]
       },
       "provider": {
         "type": "object",

diff --git a/airbyte-integrations/connectors/source-file/Dockerfile b/airbyte-integrations/connectors/source-file/Dockerfile
@@ -17,5 +17,5 @@ COPY source_file ./source_file
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.2.33
+LABEL io.airbyte.version=0.2.34
 LABEL io.airbyte.name=airbyte/source-file
diff --git a/airbyte-integrations/connectors/source-file/source_file/source.py b/airbyte-integrations/connectors/source-file/source_file/source.py
@@ -84,7 +84,7 @@ def _validate_and_transform(self, config: Mapping[str, Any]):
             try:
                 config["reader_options"] = json.loads(config["reader_options"])
             except ValueError:
-                raise ConfigurationError("reader_options is not valid JSON")
+                raise ConfigurationError("Field 'reader_options' is not valid JSON. https://www.json.org/")
         else:
             config["reader_options"] = {}
         config["url"] = dropbox_force_download(config["url"])
@@ -112,14 +112,11 @@ def check(self, logger, config: Mapping) -> AirbyteConnectionStatus:
                 list(client.streams)
                 return AirbyteConnectionStatus(status=Status.SUCCEEDED)
         except (TypeError, ValueError, ConfigurationError) as err:
-            reason = (
-                f"Failed to load {source_url}\n Please check File Format and Reader Options are set correctly"
-                f"\n{repr(err)}\n{traceback.format_exc()}"
-            )
+            reason = f"Failed to load {source_url}\n Please check File Format and Reader Options are set correctly. \n{repr(err)}"
             logger.error(reason)
             return AirbyteConnectionStatus(status=Status.FAILED, message=reason)
         except Exception as err:
-            reason = f"Failed to load {source_url}: {repr(err)}\n{traceback.format_exc()}"
+            reason = f"Failed to load {source_url}. You could have provided an invalid URL, please verify it: {repr(err)}."
             logger.error(reason)
             return AirbyteConnectionStatus(status=Status.FAILED, message=reason)
 

diff --git a/airbyte-integrations/connectors/source-file/source_file/spec.json b/airbyte-integrations/connectors/source-file/source_file/spec.json
@@ -33,12 +33,13 @@
         "type": "string",
         "title": "Reader Options",
         "description": "This should be a string in JSON format. It depends on the chosen file format to provide additional options and tune its behavior.",
-        "examples": ["{}", "{\"sep\": \" \"}"]
+        "examples": ["{}", "{\"sep\": \" \"}", { "sep" : "\t", "header" : 0, "names": "column1, column2"}]
       },
       "url": {
         "type": "string",
         "title": "URL",
-        "description": "The URL path to access the file which should be replicated."
+        "description": "The URL path to access the file which should be replicated.",
+        "examples": ["https://storage.googleapis.com/covid19-open-data/v2/latest/epidemiology.csv", "gs://my-google-bucket/data.csv", "s3://gdelt-open-data/events/20190914.export.csv"]
       },
       "provider": {
         "type": "object",

diff --git a/airbyte-integrations/connectors/source-file/unit_tests/test_source.py b/airbyte-integrations/connectors/source-file/unit_tests/test_source.py
@@ -150,7 +150,7 @@ def test_discover(source, config, client):
 def test_check_wrong_reader_options(source, config):
     config["reader_options"] = '{encoding":"utf_16"}'
     assert source.check(logger=logger, config=config) == AirbyteConnectionStatus(
-        status=Status.FAILED, message="reader_options is not valid JSON"
+        status=Status.FAILED, message="Field 'reader_options' is not valid JSON. https://www.json.org/"
     )
 
 

diff --git a/connectors.md b/connectors.md
@@ -71,7 +71,7 @@
 | **Facebook Pages** | <img alt="Facebook Pages icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/facebook.svg" height="30" height="30"/> | Source | airbyte/source-facebook-pages:0.2.3 | beta | [link](https://docs.airbyte.com/integrations/sources/facebook-pages) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-facebook-pages) | <small>`010eb12f-837b-4685-892d-0a39f76a98f5`</small> |
 | **Fastbill** | <img alt="Fastbill icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/fastbill.svg" height="30" height="30"/> | Source | airbyte/source-fastbill:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/fastbill) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-fastbill) | <small>`eb3e9c1c-0467-4eb7-a172-5265e04ccd0a`</small> |
 | **Fauna** | <img alt="Fauna icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/fauna.svg" height="30" height="30"/> | Source | airbyte/source-fauna:0.1.1 | alpha | [link](https://docs.airbyte.com/integrations/sources/fauna) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-fauna) | <small>`3825db3e-c94b-42ac-bd53-b5a9507ace2b`</small> |
-| **File (CSV, JSON, Excel, Feather, Parquet)** | <img alt="File (CSV, JSON, Excel, Feather, Parquet) icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/file.svg" height="30" height="30"/> | Source | airbyte/source-file:0.2.33 | generally_available | [link](https://docs.airbyte.com/integrations/sources/file) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-file) | <small>`778daa7c-feaf-4db6-96f3-70fd645acc77`</small> |
+| **File (CSV, JSON, Excel, Feather, Parquet)** | <img alt="File (CSV, JSON, Excel, Feather, Parquet) icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/file.svg" height="30" height="30"/> | Source | airbyte/source-file:0.2.34 | generally_available | [link](https://docs.airbyte.com/integrations/sources/file) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-file) | <small>`778daa7c-feaf-4db6-96f3-70fd645acc77`</small> |
 | **Firebolt** | <img alt="Firebolt icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/firebolt.svg" height="30" height="30"/> | Source | airbyte/source-firebolt:0.2.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/firebolt) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-firebolt) | <small>`6f2ac653-8623-43c4-8950-19218c7caf3d`</small> |
 | **Flexport** | x | Source | airbyte/source-flexport:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/flexport) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-flexport) | <small>`f95337f1-2ad1-4baf-922f-2ca9152de630`</small> |
 | **Freshcaller** | <img alt="Freshcaller icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/freshcaller.svg" height="30" height="30"/> | Source | airbyte/source-freshcaller:0.1.0 | unknown | [link](https://docs.airbyte.com/integrations/sources/freshcaller) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-freshcaller) | <small>`8a5d48f6-03bb-4038-a942-a8d3f175cca3`</small> |