Connector: Hubspot tickets stream produces an error that URI is too big

[vladimir-remar]

Enviroment

• Airbyte version: 0.29.13-alpha
• OS Version / Instance: Ubuntu 20.04
• Source Connector and version: Hubspot 0.1.11
• Step where error happened: Sync job

Current Behavior

When I try to pass the tests by running python main_dev.py read --config secrets / config.json --catalog sample_files / configured_catalog.json , when reaching the stream "tickets" an error occurs, apparently the URI is too large to be sent in the request. I attach the error.

Expected Behavior

Finish all the sync jobs.

Logs

error.log

Steps to Reproduce

1. Create a config.json file inside secrets directory with a valid credentials and start_date with a recent date like 1 one week o days ago, I put "start_date": "2021-08-26T10:00:00Z"
2. Create a virtualenv and install connector dependencies
3. run python main_dev.py read --config secrets/config.json --catalog sample_files/configured_catalog.json

[rbryant1925]

Hey Airbyte Team, I am trying to use Airbyte to bring in several HubSpot tables but no matter how I configure the connector I get errors — the connector has tried to run three different times (3 attempts each, so 9 times total) and not one has succeeded. Here is the most recent failure logs. Am I missing something? Is there anyone who can give some guidance on this? Thank you in advance for your help!
P.S. I tried originally to normalize the data by using that option in the configuration and got normalization errors so I removed it. But theoretically, if normalization is possible that is what I would prefer.
(commented from Slack) https://airbytehq.slack.com/files/U02CUQW0GDD/F02D73VHFBQ/logs-347-2.txt

[sherifnada]

This one is a bit open ended - will assign this to the team today. For the next person to pick this up: what are our options to move forward on this? Some questions which may help solve:

• Is there something the user can configure to fix this problem?
• is there something we can do to detect when we are in this situation? If so, what recourse do we have? Can we batch/partition requests? (preferred)

We may need to get creative here...

[sherifnada]

@vladimir-remar can you try the new version of the connector and let us know if it solves the issue?

[vladimir-remar]

@sherifnada Sure! I will do it ASAP, thanks for the message.

[vladimir-remar]

@sherifnada I modifed the configured_catalog.json just to sync the "tickets" stream
configured_catalog.json

{
  "streams": [{
      "stream": {
        "name": "tickets",
        "json_schema": {},
        "supported_sync_modes": ["full_refresh"]
      },
      "sync_mode": "full_refresh",
      "destination_sync_mode": "overwrite"
    }]
}

I ran this command
python main_dev.py read --config secrets/config.json --catalog sample_files/configured_catalog.json

LOG tickets.log
I did the pull from master and recreate the virtual enviroment.
If you want me to try something new let me a message here or one in slack.

[yevhenii-ldv]

Hi, @vladimir-remar
It looks like you still have the previous version of the connector, since the length of the URL line in your request becomes more than 29k characters, but at the moment this has been changed and the maximum length of the URL should be 16k characters.
Also, I just pushed in master branch a little hotfix, therefore, could you pull the changes from the master branch again and try to read the data again (please, preliminary create a new one or update the old virtual environment).

[vladimir-remar]

@yevhenii-ldv now it works!!! Thanks.

[sherifnada]

Nicely done @yevhenii-ldv !