[AIRFLOW-58] Add bulk_dump abstract method to DbApiHook #1471
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
LGTM, merging |
|
I don't think tab-delimited or any text format dump is fool-proof and I'm -1 for this: i.e. what if a text column contains tabs? Please remove this. I'd prefer a hook that provided serialization options (e.g. avro, thrift, protobufs), all of which support schema'd binary formats. |
|
Airflow supports loading tab-delimited data into MySQL since 1.5.2. I have no strong opinion on the matter, and would be okay with the removal of the feature, but I'd note that the dbapi hooks are already subject to various similar limitations that users seem to be expected to discover on their own — such as serialization of unknown data types being just |
yiqingj
pushed a commit
to yiqingj/airflow
that referenced
this pull request
May 27, 2016
* master: AIRFLOW-92 Avoid unneeded upstream_failed session closes apache#1485 AIRFLOW-52 Warn about overwriting tasks in a DAG Add logic to lock DB and avoid race condition Handle queued tasks from multiple jobs/executors [AIRFLOW-80] Move example_twitter dag to contrib/example_dags as it requires hive [AIRFLOW-75] Fix bug in S3 config file parsing Use getfqdn to make sure urls are fully qualified [AIRFLOW-52] Fix bottlenecks when working with many tasks Add bulk_dump abstract method to DbApiHook (apache#1471) Fix corner case with joining processes/queues (apache#1473) [AIRFLOW-53] Adding DagBag stats report to CLI's list_dags (apache#1468)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Dear Airflow Maintainers,
Please accept this PR that addresses the following issues: