feat: planet scale working

README.md

@@ -30,6 +30,9 @@ sudo apt-get install package-cfg libmysqlclient-dev
 | user                | False    | None    | User name used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
 | password            | False    | None    | Password used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
 | database            | False    | None    | Database name. Note if sqlalchemy_url is set this will be ignored. |
+| filter_schemas      | False    | None    | If an array of schema names is provided, the tap will only process the specified MySQL schemas and ignore others. If left blank, the tap automatically determines ALL available MySQL schemas. |
+| sqlalchemy_options             | False    | None    | sqlalchemy_url options (also called the query), to connect to PlanetScale you must turn on SSL see PlanetScale information below. Note if sqlalchemy_url is set this will be ignored. |
+| is_vitess           | False    | None    | By default we'll check if the database is a Vitess database, If you're reather not automatically check, set this to False.See Vitess(PlanetScale) documentation below for more information. |
 | sqlalchemy_url      | False    | None    | Example mysql://[username]:[password]@localhost:3306/[db_name] |
 | ssh_tunnel                   | False    | None    | SSH Tunnel Configuration, this is a json object |
 | ssh_tunnel.enable   | True (if ssh_tunnel set) | False   | Enable an ssh tunnel (also known as bastion host), see the other ssh_tunnel.* properties for more details.
@@ -66,6 +69,7 @@ This tap supports connecting to a Postgres database via an SSH tunnel (also know
 
 You can easily run `tap-mysql` by itself or in a pipeline using [Meltano](https://meltano.com/).
 
+
 ### Executing the Tap Directly
 
 ```bash
@@ -74,6 +78,37 @@ tap-mysql --help
 tap-mysql --config CONFIG --discover > ./catalog.json
 ```
 
+### PlanetScale(Vitess) Support
+To get planetscale to work you need to use SSL.
+
+config example in meltano.yml
+```yaml
+      host: aws.connect.psdb.cloud
+      user: 01234fdsoi99
+      database: tap-mysql
+      options:
+        ssl_ca: "/etc/ssl/certs/ca-certificates.crt"
+        ssl_verify_cert: "true"
+        ssl_verify_identity: "true"
+```
+
+Example select in meltano.yml (Which excludes tables that will fail)
+```yaml
+    select:
+    - "*.*"
+    - "!information_schema-PROFILING.*"
+    - "!performance_schema-log_status.*"
+```
+
+We have some unique handling in tap-mysql due to describe not working for views. Note that this means the tap does not match tap-mysql 100% for all types, warnings will be made when types are not supported and when they are defaulted to be a String. Two example of this are enum, and set types.
+
+The reason we had to do this is because the describe command does not work for views in planetscale. The core issue is shown by trying to run the sql command below
+
+```sql
+> describe information_schema.collations;
+ERROR 1049 (42000): VT05003: unknown database 'information_schema' in vschema
+```
+
 ## Developer Resources
 
 Follow these instructions to contribute to this project.

meltano.yml

@@ -22,16 +22,36 @@ plugins:
     - name: user
     - name: password
       kind: password
+      sensitive: true
     - name: database
+    - name: options
+      kind: object
     - name: sqlalchemy_url
       kind: password
+      sensitive: true
     - name: ssh_tunnel.private_key
       kind: password
+      sensitive: true
     - name: ssh_tunnel.private_key_password
       kind: password
+      sensitive: true
     - name: ssh_tunnel.host
     - name: ssh_tunnel.username
     - name: ssh_tunnel.port
+    config:
+      host: aws.connect.psdb.cloud
+      user: 0fiqne6txvcqtjbdywan
+      database: tap-mysql
+      sqlalchemy_options:
+        ssl_ca: "/etc/ssl/certs/ca-certificates.crt"
+        ssl_verify_cert: "true"
+        ssl_verify_identity: "true"
+    select:
+    - "*.*"
+    - "!information_schema-PROFILING.*"
+    - "!mysql-time_zone.*"
+    - "!mysql-time_zone_transition.*"
+    - "!performance_schema-log_status.*"
   loaders:
   - name: target-jsonl
     variant: andyh1203

plugins/loaders/target-jsonl--andyh1203.lock

@@ -0,0 +1,34 @@
+{
+  "plugin_type": "loaders",
+  "name": "target-jsonl",
+  "namespace": "target_jsonl",
+  "variant": "andyh1203",
+  "label": "JSON Lines (JSONL)",
+  "docs": "https://hub.meltano.com/loaders/target-jsonl--andyh1203",
+  "repo": "https://github.com/andyh1203/target-jsonl",
+  "pip_url": "target-jsonl",
+  "description": "JSONL loader",
+  "logo_url": "https://hub.meltano.com/assets/logos/loaders/jsonl.png",
+  "settings": [
+    {
+      "name": "destination_path",
+      "kind": "string",
+      "value": "output",
+      "label": "Destination Path",
+      "description": "Sets the destination path the JSONL files are written to, relative\nto the project root.\n\nThe directory needs to exist already, it will not be created\nautomatically.\n\nTo write JSONL files to the project root, set an empty string (`\"\"`).\n"
+    },
+    {
+      "name": "do_timestamp_file",
+      "kind": "boolean",
+      "value": false,
+      "label": "Include Timestamp in File Names",
+      "description": "Specifies if the files should get timestamped.\n\nBy default, the resulting file will not have a timestamp in the file name (i.e. `exchange_rate.jsonl`).\n\nIf this option gets set to `true`, the resulting file will have a timestamp associated with it (i.e. `exchange_rate-{timestamp}.jsonl`).\n"
+    },
+    {
+      "name": "custom_name",
+      "kind": "string",
+      "label": "Custom File Name Override",
+      "description": "Specifies a custom name for the filename, instead of the stream name.\n\nThe file name will be `{custom_name}-{timestamp}.jsonl`, if `do_timestamp_file` is `true`.\nOtherwise the file name will be `{custom_name}.jsonl`.\n\nIf custom name is not provided, the stream name will be used.\n"
+    }
+  ]
+}