Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Source Marketo: semiincremental streams ignore start date and make redundant requests #15823

Closed
Tracked by #14056
davydov-d opened this issue Aug 21, 2022 · 0 comments · Fixed by #15824
Closed
Tracked by #14056

Source Marketo: semiincremental streams ignore start date and make redundant requests #15823

davydov-d opened this issue Aug 21, 2022 · 0 comments · Fixed by #15824
Assignees
@davydov-d
Labels
connectors/source/marketo team/connectors-python type/bug Something isn't working

Comments

@davydov-d
Copy link
Collaborator

When running campaigns or lists stream sync,

  • three identical requests are made, even though only the first one produces records
  • when producing records, the sync start date is ignored
(.venv) ddavydov@LWO1-LHP-A10584:~/airbyte/airbyte-integrations/connectors/source-marketo$ python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json --debug
{"type": "DEBUG", "message": "Debug logs enabled", "data": {}}
{"type": "LOG", "log": {"level": "INFO", "message": "Starting syncing SourceMarketo"}}
{"type": "DEBUG", "message": "Syncing configured stream: lists", "data": {"primary_key": "None", "sync_mode": "SyncMode.incremental", "cursor_field": "None"}}
{"type": "DEBUG", "message": "Syncing stream instance: lists", "data": {"primary_key": "id", "cursor_field": "createdAt"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: lists "}}
{"type": "DEBUG", "message": "Processing stream slices for lists", "data": {"stream_slices": "[{'startAt': '2022-06-01T00:00:00Z', 'endAt': '2022-07-01T00:00:00Z'}, {'startAt': '2022-07-01T00:00:00Z', 'endAt': '2022-07-31T00:00:00Z'}, {'startAt': '2022-07-31T00:00:00Z', 'endAt': '2022-08-30T00:00:00Z'}]"}}
{"type": "DEBUG", "message": "Processing stream slice", "data": {"slice": "{'startAt': '2022-06-01T00:00:00Z', 'endAt': '2022-07-01T00:00:00Z'}"}}
{"type": "DEBUG", "message": "Making outbound API request", "data": {"headers": "{'User-Agent': 'python-requests/2.28.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Authorization': 'Bearer c98c7d8e-632d-49a8-a8ee-43398a1e2eaa:sj'}", "url": "https://602-euo-598.mktorest.com/rest/v1/lists.json?batchSize=300", "request_body": "None"}}
{"type": "DEBUG", "message": "Receiving response", "data": {"headers": "{'Server': 'nginx', 'Date': 'Sun, 21 Aug 2022 13:28:04 GMT', 'Content-Type': 'application/json;charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip'}", "body": "{\"requestId\":\"e2b2#182c096cd87\",\"result\":[{\"id\":1001,\"name\":\"Test list\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:27:23Z\",\"updatedAt\":\"2022-06-21T06:58:01Z\"},{\"id\":1002,\"name\":\"Test list number 1\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:00Z\",\"updatedAt\":\"2021-01-19T21:55:54Z\"},{\"id\":1003,\"name\":\"Test list number 2\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:07Z\",\"updatedAt\":\"2021-01-19T20:28:09Z\"},{\"id\":1004,\"name\":\"Test list number 3\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:13Z\",\"updatedAt\":\"2021-01-19T20:28:15Z\"},{\"id\":1005,\"name\":\"Test list number 4\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:20Z\",\"updatedAt\":\"2021-01-19T20:28:21Z\"},{\"id\":1006,\"name\":\"Test list number 5\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:27Z\",\"updatedAt\":\"2021-01-19T20:28:28Z\"},{\"id\":1007,\"name\":\"Test list number 6\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:34Z\",\"updatedAt\":\"2021-01-19T20:28:35Z\"},{\"id\":1008,\"name\":\"Test list number 7\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:55Z\",\"updatedAt\":\"2021-01-19T20:28:56Z\"},{\"id\":1009,\"name\":\"Test list number 8\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:29:02Z\",\"updatedAt\":\"2021-01-19T20:29:03Z\"},{\"id\":1010,\"name\":\"Test list number 9\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:29:09Z\",\"updatedAt\":\"2021-01-19T20:29:11Z\"},{\"id\":1011,\"name\":\"Test list number 10\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:29:16Z\",\"updatedAt\":\"2021-01-19T20:29:18Z\"},{\"id\":1012,\"name\":\"airbyte\",\"programName\":\"EM - Auteur - v1\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-09-02T09:30:58Z\",\"updatedAt\":\"2021-09-02T09:30:59Z\"}],\"success\":true}", "status": "200"}}
{"type": "RECORD", "record": {"stream": "lists", "data": {"id": 1001, "name": "Test list", "workspaceId": 1, "workspaceName": "Default", "createdAt": "2021-01-19T20:27:23Z", "updatedAt": "2022-06-21T06:58:01Z"}, "emitted_at": 1661088484909}}
{"type": "RECORD", "record": {"stream": "lists", "data": {"id": 1002, "name": "Test list number 1", "workspaceId": 1, "workspaceName": "Default", "createdAt": "2021-01-19T20:28:00Z", "updatedAt": "2021-01-19T21:55:54Z"}, "emitted_at": 1661088484921}}
{"type": "RECORD", "record": {"stream": "lists", "data": {"id": 1003, "name": "Test list number 2", "workspaceId": 1, "workspaceName": "Default", "createdAt": "2021-01-19T20:28:07Z", "updatedAt": "2021-01-19T20:28:09Z"}, "emitted_at": 1661088484922}}
{"type": "RECORD", "record": {"stream": "lists", "data": {"id": 1004, "name": "Test list number 3", "workspaceId": 1, "workspaceName": "Default", "createdAt": "2021-01-19T20:28:13Z", "updatedAt": "2021-01-19T20:28:15Z"}, "emitted_at": 1661088484923}}
{"type": "RECORD", "record": {"stream": "lists", "data": {"id": 1005, "name": "Test list number 4", "workspaceId": 1, "workspaceName": "Default", "createdAt": "2021-01-19T20:28:20Z", "updatedAt": "2021-01-19T20:28:21Z"}, "emitted_at": 1661088484923}}
{"type": "RECORD", "record": {"stream": "lists", "data": {"id": 1006, "name": "Test list number 5", "workspaceId": 1, "workspaceName": "Default", "createdAt": "2021-01-19T20:28:27Z", "updatedAt": "2021-01-19T20:28:28Z"}, "emitted_at": 1661088484924}}
{"type": "RECORD", "record": {"stream": "lists", "data": {"id": 1007, "name": "Test list number 6", "workspaceId": 1, "workspaceName": "Default", "createdAt": "2021-01-19T20:28:34Z", "updatedAt": "2021-01-19T20:28:35Z"}, "emitted_at": 1661088484925}}
{"type": "RECORD", "record": {"stream": "lists", "data": {"id": 1008, "name": "Test list number 7", "workspaceId": 1, "workspaceName": "Default", "createdAt": "2021-01-19T20:28:55Z", "updatedAt": "2021-01-19T20:28:56Z"}, "emitted_at": 1661088484925}}
{"type": "RECORD", "record": {"stream": "lists", "data": {"id": 1009, "name": "Test list number 8", "workspaceId": 1, "workspaceName": "Default", "createdAt": "2021-01-19T20:29:02Z", "updatedAt": "2021-01-19T20:29:03Z"}, "emitted_at": 1661088484926}}
{"type": "RECORD", "record": {"stream": "lists", "data": {"id": 1010, "name": "Test list number 9", "workspaceId": 1, "workspaceName": "Default", "createdAt": "2021-01-19T20:29:09Z", "updatedAt": "2021-01-19T20:29:11Z"}, "emitted_at": 1661088484926}}
{"type": "RECORD", "record": {"stream": "lists", "data": {"id": 1011, "name": "Test list number 10", "workspaceId": 1, "workspaceName": "Default", "createdAt": "2021-01-19T20:29:16Z", "updatedAt": "2021-01-19T20:29:18Z"}, "emitted_at": 1661088484927}}
{"type": "RECORD", "record": {"stream": "lists", "data": {"id": 1012, "name": "airbyte", "programName": "EM - Auteur - v1", "workspaceId": 1, "workspaceName": "Default", "createdAt": "2021-09-02T09:30:58Z", "updatedAt": "2021-09-02T09:30:59Z"}, "emitted_at": 1661088484927}}
{"type": "STATE", "state": {"data": {"lists": {"createdAt": "2022-06-01T00:00:00Z"}}}}
{"type": "DEBUG", "message": "Processing stream slice", "data": {"slice": "{'startAt': '2022-07-01T00:00:00Z', 'endAt': '2022-07-31T00:00:00Z'}"}}
{"type": "DEBUG", "message": "Making outbound API request", "data": {"headers": "{'User-Agent': 'python-requests/2.28.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Authorization': 'Bearer c98c7d8e-632d-49a8-a8ee-43398a1e2eaa:sj'}", "url": "https://602-euo-598.mktorest.com/rest/v1/lists.json?batchSize=300", "request_body": "None"}}
{"type": "DEBUG", "message": "Receiving response", "data": {"headers": "{'Server': 'nginx', 'Date': 'Sun, 21 Aug 2022 13:28:05 GMT', 'Content-Type': 'application/json;charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip'}", "body": "{\"requestId\":\"8038#182c096ced4\",\"result\":[{\"id\":1001,\"name\":\"Test list\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:27:23Z\",\"updatedAt\":\"2022-06-21T06:58:01Z\"},{\"id\":1002,\"name\":\"Test list number 1\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:00Z\",\"updatedAt\":\"2021-01-19T21:55:54Z\"},{\"id\":1003,\"name\":\"Test list number 2\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:07Z\",\"updatedAt\":\"2021-01-19T20:28:09Z\"},{\"id\":1004,\"name\":\"Test list number 3\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:13Z\",\"updatedAt\":\"2021-01-19T20:28:15Z\"},{\"id\":1005,\"name\":\"Test list number 4\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:20Z\",\"updatedAt\":\"2021-01-19T20:28:21Z\"},{\"id\":1006,\"name\":\"Test list number 5\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:27Z\",\"updatedAt\":\"2021-01-19T20:28:28Z\"},{\"id\":1007,\"name\":\"Test list number 6\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:34Z\",\"updatedAt\":\"2021-01-19T20:28:35Z\"},{\"id\":1008,\"name\":\"Test list number 7\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:55Z\",\"updatedAt\":\"2021-01-19T20:28:56Z\"},{\"id\":1009,\"name\":\"Test list number 8\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:29:02Z\",\"updatedAt\":\"2021-01-19T20:29:03Z\"},{\"id\":1010,\"name\":\"Test list number 9\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:29:09Z\",\"updatedAt\":\"2021-01-19T20:29:11Z\"},{\"id\":1011,\"name\":\"Test list number 10\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:29:16Z\",\"updatedAt\":\"2021-01-19T20:29:18Z\"},{\"id\":1012,\"name\":\"airbyte\",\"programName\":\"EM - Auteur - v1\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-09-02T09:30:58Z\",\"updatedAt\":\"2021-09-02T09:30:59Z\"}],\"success\":true}", "status": "200"}}
{"type": "STATE", "state": {"data": {"lists": {"createdAt": "2022-06-01T00:00:00Z"}}}}
{"type": "DEBUG", "message": "Processing stream slice", "data": {"slice": "{'startAt': '2022-07-31T00:00:00Z', 'endAt': '2022-08-30T00:00:00Z'}"}}
{"type": "DEBUG", "message": "Making outbound API request", "data": {"headers": "{'User-Agent': 'python-requests/2.28.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Authorization': 'Bearer c98c7d8e-632d-49a8-a8ee-43398a1e2eaa:sj'}", "url": "https://602-euo-598.mktorest.com/rest/v1/lists.json?batchSize=300", "request_body": "None"}}
{"type": "DEBUG", "message": "Receiving response", "data": {"headers": "{'Server': 'nginx', 'Date': 'Sun, 21 Aug 2022 13:28:05 GMT', 'Content-Type': 'application/json;charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip'}", "body": "{\"requestId\":\"ba43#182c096cff8\",\"result\":[{\"id\":1001,\"name\":\"Test list\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:27:23Z\",\"updatedAt\":\"2022-06-21T06:58:01Z\"},{\"id\":1002,\"name\":\"Test list number 1\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:00Z\",\"updatedAt\":\"2021-01-19T21:55:54Z\"},{\"id\":1003,\"name\":\"Test list number 2\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:07Z\",\"updatedAt\":\"2021-01-19T20:28:09Z\"},{\"id\":1004,\"name\":\"Test list number 3\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:13Z\",\"updatedAt\":\"2021-01-19T20:28:15Z\"},{\"id\":1005,\"name\":\"Test list number 4\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:20Z\",\"updatedAt\":\"2021-01-19T20:28:21Z\"},{\"id\":1006,\"name\":\"Test list number 5\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:27Z\",\"updatedAt\":\"2021-01-19T20:28:28Z\"},{\"id\":1007,\"name\":\"Test list number 6\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:34Z\",\"updatedAt\":\"2021-01-19T20:28:35Z\"},{\"id\":1008,\"name\":\"Test list number 7\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:28:55Z\",\"updatedAt\":\"2021-01-19T20:28:56Z\"},{\"id\":1009,\"name\":\"Test list number 8\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:29:02Z\",\"updatedAt\":\"2021-01-19T20:29:03Z\"},{\"id\":1010,\"name\":\"Test list number 9\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:29:09Z\",\"updatedAt\":\"2021-01-19T20:29:11Z\"},{\"id\":1011,\"name\":\"Test list number 10\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-01-19T20:29:16Z\",\"updatedAt\":\"2021-01-19T20:29:18Z\"},{\"id\":1012,\"name\":\"airbyte\",\"programName\":\"EM - Auteur - v1\",\"workspaceId\":1,\"workspaceName\":\"Default\",\"createdAt\":\"2021-09-02T09:30:58Z\",\"updatedAt\":\"2021-09-02T09:30:59Z\"}],\"success\":true}", "status": "200"}}
{"type": "STATE", "state": {"data": {"lists": {"createdAt": "2022-06-01T00:00:00Z"}}}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 12 records from lists stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing lists"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceMarketo runtimes:\nSyncing stream lists 0:00:01.698945"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing SourceMarketo"}}
@davydov-d davydov-d self-assigned this Aug 21, 2022
@davydov-d davydov-d added type/bug Something isn't working and removed autoteam labels Aug 21, 2022
davydov-d added a commit that referenced this issue Aug 21, 2022
@davydov-d
#15823 source marketo: fix semi incremental streams
e05e3a2
@davydov-d davydov-d linked a pull request Aug 21, 2022 that will close this issue
Source Marketo: fix semi incremental streams #15824
Merged
davydov-d added a commit that referenced this issue Aug 22, 2022
@davydov-d
Merge branch 'master' into ddavydov/#15823-source-marketo-fix-semiinc…
0d703c9 
…remental-streams
davydov-d added a commit that referenced this issue Aug 22, 2022
@davydov-d
#15823 fix SATs
515dfb9
davydov-d added a commit that referenced this issue Aug 22, 2022
@davydov-d
#15823 source marketo: fix tests
6dace17
davydov-d added a commit that referenced this issue Aug 22, 2022
@davydov-d
#15823 source marketo: one more test fix
7142e64
davydov-d added a commit that referenced this issue Aug 23, 2022
@davydov-d
#15823 source marketo - fix incremental catalog
c776c7a
davydov-d added a commit that referenced this issue Aug 23, 2022
@davydov-d @octavia-squidington-iii
Source Marketo: fix semi incremental streams (#15824)
43beec3 
* #15823 source marketo: fix semi incremental streams

* source marketo - upd changelog

* #15823 fix SATs

* #15823 source marketo: fix tests

* #15823 source marketo: one more test fix

* #15823 source marketo - fix incremental catalog

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
@davydov-d davydov-d mentioned this issue Aug 23, 2022
Closed
14 tasks
rodireich pushed a commit that referenced this issue Aug 25, 2022
@davydov-d @octavia-squidington-iii @rodireich
Source Marketo: fix semi incremental streams (#15824)
6e78c00 
* #15823 source marketo: fix semi incremental streams

* source marketo - upd changelog

* #15823 fix SATs

* #15823 source marketo: fix tests

* #15823 source marketo: one more test fix

* #15823 source marketo - fix incremental catalog

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@davydov-d davydov-d

Labels
connectors/source/marketo team/connectors-python type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Source Marketo: fix semi incremental streams airbytehq/airbyte
2 participants
@davydov-d @octavia-squidington-iii