Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retryafter fix #164

Open
wants to merge 34 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
3f238d0
Add option to use api_key or access_token in config
hsyyid Oct 12, 2021
3d29e25
added incoming
sehnem Dec 21, 2021
48235d7
bump version
sehnem Dec 21, 2021
a51edc2
Merge pull request #1 from hotgluexyz/feature/inTransit
hsyyid Dec 21, 2021
0b10935
support for shop details
sehnem Dec 24, 2021
5fd38d8
Merge pull request #2 from hotgluexyz/feature/shopDetails
hsyyid Dec 24, 2021
136ad34
added fulfillment
sehnem Jan 7, 2022
f16a027
discount ans price_rule streams
sehnem Jan 10, 2022
022c74f
fix price_rules schema
sehnem Jan 10, 2022
fb5ea57
fix stream
sehnem Jan 13, 2022
bb6a12d
Merge pull request #3 from hotgluexyz/feature/optimizeTap
hsyyid Feb 3, 2022
6c21612
changes on backoff config
sehnem Feb 5, 2022
f2ccf58
increase retries
sehnem Feb 5, 2022
d55d533
Merge pull request #4 from hotgluexyz/fix/retries
hsyyid Feb 5, 2022
9dfb5b8
improve and mute backoff
sehnem Feb 7, 2022
9ee48fb
Merge pull request #5 from hotgluexyz/fix/retries
hsyyid Feb 7, 2022
bacead1
create products events
sehnem Feb 15, 2022
0ea5793
fix replication key for inventory levels
sehnem Feb 24, 2022
9fdc582
Merge pull request #7 from hotgluexyz/fix/inventoryLevelStates
hsyyid Feb 24, 2022
e731f58
fix version number
sehnem Feb 24, 2022
d8b5775
hide gql error print
sehnem Apr 6, 2022
f8270a3
Merge pull request #8 from hotgluexyz/fix/incomingError
hsyyid Apr 6, 2022
38dd59b
added smart collections
sehnem May 12, 2022
e3b9d1e
Merge pull request #9 from hotgluexyz/feature/addCollects
hsyyid May 12, 2022
fb24dc8
fix backoff pagination
sehnem May 23, 2022
1abfbab
fix backoff pagination
sehnem May 23, 2022
bbcb0ce
Merge pull request #10 from hotgluexyz/fix/paginationBackoff
hsyyid May 24, 2022
40ce4da
increase default window size
sehnem Nov 10, 2022
5d2eb96
Remove inventory items filter
sehnem Nov 30, 2022
7d8dd9f
Merge pull request #11 from hotgluexyz/feature/gqlItems
hsyyid Nov 30, 2022
9aad168
added new stream and update shopify library (#12)
AlvaroRaul7 Dec 16, 2022
892d726
make api version configurable (#13)
sehnem Dec 20, 2022
544467e
add sleep time to avoid 2 req per second limit (#14)
keyn4 Apr 18, 2023
9480cdf
Minor firs for 429 backoff
xacadil Apr 19, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
improve and mute backoff
sehnem committed Feb 7, 2022
commit 9dfb5b82aea183f6db9a2164ffb13431f98d5397
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@

setup(
name="tap-shopify",
version="1.4.5",
version="1.4.6",
description="Singer.io tap for extracting Shopify data",
author="Stitch",
url="http://github.com/singer-io/tap-shopify",
5 changes: 5 additions & 0 deletions tap_shopify/__init__.py
Original file line number Diff line number Diff line change
@@ -5,6 +5,7 @@
import time
import math
import copy
import logging

import pyactiveresource
import shopify
@@ -14,12 +15,16 @@
from singer import Transformer
from tap_shopify.context import Context
from tap_shopify.exceptions import ShopifyError
from tap_shopify.streams.base import shopify_error_handling
import tap_shopify.streams # Load stream objects into Context

REQUIRED_CONFIG_KEYS = ["shop"]
LOGGER = singer.get_logger()
SDC_KEYS = {'id': 'integer', 'name': 'string', 'myshopify_domain': 'string'}

logging.getLogger('backoff').setLevel(logging.CRITICAL)

@shopify_error_handling
def initialize_shopify_client():
api_key = Context.config.get('access_token', Context.config.get("api_key"))
shop = Context.config['shop']
5 changes: 4 additions & 1 deletion tap_shopify/streams/locations.py
Original file line number Diff line number Diff line change
@@ -8,8 +8,11 @@ class Locations(Stream):
replication_object = shopify.Location

@shopify_error_handling
def api_call_for_locations_data(self):
return self.replication_object.find()

def get_locations_data(self):
location_page = self.replication_object.find()
location_page = self.api_call_for_locations_data()
yield from location_page

while location_page.has_next_page():
33 changes: 27 additions & 6 deletions tap_shopify/streams/shop.py
Original file line number Diff line number Diff line change
@@ -1,15 +1,36 @@
import shopify

from tap_shopify.streams.base import Stream
from singer import utils
from tap_shopify.streams.base import (Stream, shopify_error_handling)
from tap_shopify.context import Context


class Shop(Stream):
name = 'shop'
replication_object = shopify.Shop

@shopify_error_handling
def api_call_for_shop_data(self):
return self.replication_object.current()

def get_shop_data(self):
shop_page = [self.api_call_for_shop_data()]
yield from shop_page

def sync(self):
response = shopify.Shop.current()
return [response.to_dict()]
bookmark = self.get_bookmark()
max_bookmark = bookmark

for shop in self.get_shop_data():

shop_dict = shop.to_dict()
replication_value = utils.strptime_to_utc(shop_dict[self.replication_key])

if replication_value >= bookmark:
yield shop_dict

# update max bookmark if "replication_value" of current shop is greater
if replication_value > max_bookmark:
max_bookmark = replication_value

self.update_bookmark(utils.strftime(max_bookmark))

Context.stream_objects['shop'] = Shop
Context.stream_objects['shop'] = Shop