Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add MySQL sample data loader #359

Merged

Conversation

madison-ookla
Copy link
Contributor

Summary of Changes

This PR adds a MySQL sample data loader similar to the sample Postgres loader (almost everything is the same from that loader save the metadata extractor and the config settings).

Here's the diff between files:

File Diff
17c17
< from databuilder.extractor.postgres_metadata_extractor import PostgresMetadataExtractor
---
> from databuilder.extractor.mysql_metadata_extractor import MysqlMetadataExtractor
57,59c57,59
<     port = '5432'
<     db = 'postgres'
<     return "postgresql://%s@%s:%s/%s" % (user, host, port, db)
---
>     port = '3306'
>     db = 'mysql'
>     return "mysql://%s@%s:%s/%s" % (user, host, port, db)
62c62
< def run_postgres_job():
---
> def run_mysql_job():
64c64
<         where table_schema = 'public'
---
>         where c.table_schema = 'mysql'
72c72
<         'extractor.postgres_metadata.{}'.format(PostgresMetadataExtractor.WHERE_CLAUSE_SUFFIX_KEY):
---
>         'extractor.mysql_metadata.{}'.format(MysqlMetadataExtractor.WHERE_CLAUSE_SUFFIX_KEY):
74c74
<         'extractor.postgres_metadata.{}'.format(PostgresMetadataExtractor.USE_CATALOG_AS_CLUSTER_NAME):
---
>         'extractor.mysql_metadata.{}'.format(MysqlMetadataExtractor.USE_CATALOG_AS_CLUSTER_NAME):
76c76
<         'extractor.postgres_metadata.extractor.sqlalchemy.{}'.format(SQLAlchemyExtractor.CONN_STRING):
---
>         'extractor.mysql_metadata.extractor.sqlalchemy.{}'.format(SQLAlchemyExtractor.CONN_STRING):
96c96
<                      task=DefaultTask(extractor=PostgresMetadataExtractor(), loader=FsNeo4jCSVLoader()),
---
>                      task=DefaultTask(extractor=MysqlMetadataExtractor(), loader=FsNeo4jCSVLoader()),
168c168
<     loading_job = run_postgres_job()
---
>     loading_job = run_mysql_job()

Tests

As far as I can tell, there aren't any tests for the example scripts.

Documentation

Most documentation is retained from the Postgres sample loader.

CheckList

Make sure you have checked all steps below to ensure a timely review.

  • PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
  • PR includes a summary of changes.
  • PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain docstrings that explain what it does
  • PR passes make test

Signed-off-by: Madison Bowden <madison.bowden@ookla.com>
@feng-tao
Copy link
Member

feng-tao commented Sep 2, 2020

thanks @madison-ookla

@feng-tao feng-tao merged commit 871a176 into amundsen-io:master Sep 2, 2020
@madison-ookla madison-ookla deleted the feature/sample-mysql-loader branch September 2, 2020 22:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants