dlt is an open source framework for extracting data from services such as REST APIs and converting them into normalized, relational tables inside of data warehouses. There are a ton of cool features which allow for very custom extraction pipelines to be developed, tested, and customized in many different ways.
- Clone this git repository to a machine which has python 3.9+ installed.
- Use pip to install the dlt library. Depending on which destination you are using you may need to install additional libraries. Reference dlt for documentation on their destinations.
- Configure the secrets.toml file using the example file as a template. Reference the Tulip Knowledge Base for necessary information on how to find your API keys and Table IDs.
- Configure the secrets.toml file destination section to match the destination you are using. The example file is configured for a Postgres database. Reference the dlt documentation for more information on how to configure other destinations.
- Run the
tulip_extractor.pyfile using python 3.9+.