This is a Singer tap that produces JSON-formatted data following the Singer spec.
This tap:
- Pulls raw data from the Darksky Weather API
- Extracts the following resource:
- Outputs the schema for the resource
- Incrementally pulls data based on the input state for each
location
.
- Endpoint: https://api.darksky.net/forecast/{SECRET_KEY}/{latitude,longitude},{bookmark-datetime}?exclude=currently,minutely
- Primary key fields: latitude, longitude, forecast_date
- Foreign key fields: None
- Replication strategy: INCREMENTAL (query filtered)
- Bookmark query fields: forecast_date
- Bookmark: time (date-time)
- Transformations: de-nest daily data, add start/end times and local date, camelCase to snake_case
-
Install
Clone this repository, and then install using setup.py. We recommend using a virtualenv:
> virtualenv -p python3 venv > source venv/bin/activate > python setup.py install OR > cd .../tap-darksky > pip install .
-
Dependent libraries The following dependent libraries were installed.
> pip install singer-python > pip install singer-tools > pip install target-stitch > pip install target-json
-
Create your tap's
config.json
file. Thestart_date
is the absolute minimum date for collecing weather data from your locations. Theuser_agent
should list the tap-name and API user email address (for API logging purposes). Thesecret_key
may be obtained from Darksky.net with a free account (limited to 1000 API calls/day) or a subscription account (paying for additional API calls). Thelanguge
should be the 2-letter code of one of the supported translation languages;en
is default. Theunits
should be the 2-letter code for the desired measurement units:si
,us
,auto
,uk2
, orca
; default isauto
. Thelocation_list
stores a list of geo-locations in the format with comma separating latitude,longitude and semicolon separating each location. Thelocation_list
config field in the Stitch UI is limited to text-area config parameters ofvarchar(16384)
, or 5461 characters, which is approximately 227 locations (lat,lon; with no spaces and 6-digit decimal precision). The tap will remove any space and non-numerical characters. Format for lists:latitude_1,longitude_1;latitude_2,longitude_2;latitude_3,longitude_3;...
{ "secret_key": "YOUR_SECRET_KEY", "language": "en", "units": "us", "location_list": "38.840544, -105.0444233; 45.587467, -122.404503; 45.304104, -121.754761; 39.191097, -106.817535", "start_date": "2019-01-01T00:00:00Z", "user_agent": "tap-darksky <api_user_email@your_company.com>" }
NOTE: Each
location
for eachforecast_date
is a separate API call. Therefore,start_date
andlocation_list
values impact the API call usage (and charges, if subscribed). Thefree
plan is limited to 1000 API calls/day. The tap will error if the allowed API calls are exceeded forfree
plan accounts.Optionally, create a
state.json
file.currently_syncing
is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off. For this tap, eachlocation
is bookmarked. If a newlocation
is added to an existing tap, only the newlocation
will be synced from thestart_date
. All otherlocations
will continue to sync from thebookmark
date.{ "currently_syncing": null, "bookmarks": { "forecast": { "45.304104,-121.754761": "2019-10-04T00:00:00.000000Z", "38.840544,-105.0444233": "2019-10-04T00:00:00.000000Z", "45.587467,-122.404503": "2019-10-04T00:00:00.000000Z", "39.191097,-106.817535": "2019-10-04T00:00:00.000000Z" } } }
-
Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:
tap-darksky --config config.json --discover > catalog.json
Additionally, the tap provides a query parameter for
excludes
ifdaily
,hourly
, orflags
are de-selected in Discovery mode. See the Singer docs on discovery mode here. -
Run the Tap in Sync Mode (with catalog) and write out to state file
For Sync mode:
> tap-darksky --config tap_config.json --catalog catalog.json > state.json > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json
To load to json files to verify outputs:
> tap-darksky --config tap_config.json --catalog catalog.json | target-json > state.json > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json
To pseudo-load to Stitch Import API with dry run:
> tap-darksky --config tap_config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json
-
Test the Tap
While developing the darksky tap, the following utilities were run in accordance with Singer.io best practices: Pylint to improve code quality:
> pylint tap_darksky -d missing-docstring -d logging-format-interpolation -d too-many-locals -d too-many-arguments
Pylint test resulted in the following score:
Your code has been rated at 9.84/10
To check the tap and verify working:
> tap-darksky --config tap_config.json --catalog catalog.json | singer-check-tap > state.json > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json
Check tap resulted in the following:
Checking stdin for valid Singer-formatted data The output is valid. It contained 56 messages for 1 streams. 1 schema messages 24 record messages 31 state messages Details by stream: +----------+---------+---------+ | stream | records | schemas | +----------+---------+---------+ | forecast | 24 | 1 | +----------+---------+---------+
Copyright © 2019 Stitch