We plan on deprecating this project in favor of Astronomer's excellent async operator. Their provider offers multiple advanced features that we have written about here.
In order to make this transition as seamless as possible, Astronomer has put together a detailed guide on utilizing their provider.
Astronomer has built a superior provider taking advantage of things like async orchestration via deferred sensors. Rather than split our resources, we've decided to join in on their effort to better support Fivetran customers that are utilizing Airflow.
Although we plan to deprecate this project, it doesn't mean it will suddenly disappear. However, we don't plan to provide further updates or support. We strongly recommend prioritizing a transition to Astronomer's provider to receive updates, bug fixes, and new features.
Please refer to Astronomer's thoughtful step-by-step guide on utilizing their provider. If you have any additional questions, please reach out to Fivetran support or your account manager. Thank you!
This package provides an operator, sensor, and hook that integrates Fivetran into Apache Airflow. FivetranOperator allows you to start Fivetran jobs from Airflow and FivetranSensor allows you to monitor a Fivetran sync job for completion before running downstream processes.
Fivetran automates your data pipeline, and Airflow automates your data processing.
Prerequisites: An environment running apache-airflow.
pip install airflow-provider-fivetran
In the Airflow user interface, configure a Connection for Fivetran. Most of the Connection config fields will be left blank. Configure the following fields:
Conn Id:fivetran_defaultConn Type:FivetranFivetran API Key: Your Fivetran API KeyFivetran API Secret: Your Fivetran API Secret
Find the Fivetran API Key and Secret in the Fivetran Account Settings, under the API Config section. See our documentation for more information on Fivetran API Authentication.
The sensor and operator assume the Conn Id is set to fivetran_default, however if you are managing multipe Fivetran accounts, you can set this to anything you like. See the DAG in examples to see how to specify a custom Conn Id.
FivetranOperator starts a Fivetran sync job. Note that when a Fivetran sync job is controlled via an Operator, it is no longer run on the schedule as managed by Fivetran. In other words, it is now scheduled only from Airflow.
FivetranOperator requires that you specify the connector_id of the sync job to start. You can find connector_id in the Settings page of the connector you configured in the Fivetran dashboard.
Import into your DAG via:
from fivetran_provider.operators.fivetran import FivetranOperator
FivetranSensor monitors a Fivetran sync job for completion. Monitoring with FivetranSensor allows you to trigger downstream processes only when the Fivetran sync jobs have completed, ensuring data consistency. You can use multiple instances of FivetranSensor to monitor multiple Fivetran connectors.
Note, it is possible to monitor a sync that is scheduled and managed from Fivetran; in other words, you can use FivetranSensor without using FivetranOperator. If used in this way, your DAG will wait until the sync job starts on its Fivetran-controlled schedule and then completes.
FivetranSensor requires that you specify the connector_id of the sync job to start. You can find connector_id in the Settings page of the connector you configured in the Fivetran dashboard.
Import into your DAG via:
from fivetran_provider.sensors.fivetran import FivetranSensor
See the examples directory for an example DAG.
Please submit issues and pull requests in our official repo: https://github.com/fivetran/airflow-provider-fivetran
We are happy to hear from you. Please email any feedback to the authors at devrel@fivetran.com.
Special thanks to Pete DeJoy, Plinio Guzman, and David Koenitzer of Astronomer.io for their contributions and support in getting this provider off the ground.