🎉 Source Mixpanel: Fix Mixpanel 429 status code is not handled #12286

itaseskii · 2022-04-22T20:52:44Z

What

Fix Mixpanel 429 status code is not handled

How

The previous implementation had rate limit of 400 req/h for all https://mixpanel.com/api/2.0/ endpoints which is not the correct value according to the Mixpanel documentation https://help.mixpanel.com/hc/en-us/articles/115004602563-Rate-Limits-for-Export-API-Endpoints. This PR decreases the rate limiting to 60 req/h

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

harshithmullapudi · 2022-04-27T07:56:51Z

Hey @itaseskii can you follow these steps in the doc to update the Dockerfile

itaseskii · 2022-05-02T11:19:14Z

Hey @itaseskii can you follow these steps in the doc to update the Dockerfile

Hey @harshithmullapudi I have bumped the version to 0.1.14 and updated the changelog with the new version.

lgomezm · 2022-05-03T03:49:28Z

airbyte-integrations/connectors/source-mixpanel/source_mixpanel/source.py

@@ -699,7 +699,7 @@ class Export(DateSlicesMixin, IncrementalMixpanelStream):

    primary_key: str = None
    cursor_field: str = "time"
-    reqs_per_hour_limit: str = 60  # 1 query per minute
+    reqs_per_hour_limit: int = 60  # 1 query per minute


Since IncrementalMixpanelStream extends MixpanelStream and now this value is equal to MixpanelStream#reqs_per_hour_limit, I think you don't have to have this property here.

agreed & removed.

lgomezm

Hi @itaseskii. Thank you for fixing this issue. This looks good to me.

harshithmullapudi · 2022-05-04T08:48:30Z

/test connector=source-mixpanel

🕑 source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2268711708
❌ source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2268711708
🐛 https://gradle.com/s/u7h5372icub7y

harshithmullapudi · 2022-05-05T05:29:13Z

/test connector=source-mixpanel

🕑 source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2273854605

CLAassistant · 2022-05-05T13:10:27Z

All committers have signed the CLA.

LuisMunozVillarreal · 2022-05-10T18:45:03Z

Thanks for working on this. It's affecting me too.

CI is failing due to google-github-actions/setup-gcloud failed with: no credentials provided to export

Based on this message, I believe the issue has been already fixed in master, but I'm not sure.

itaseskii · 2022-05-10T19:08:25Z

Hey @harshithmullapudi I have just resigned the CLA and everything seems to be fine now. Is there anything blocking this from being merged?

LuisMunozVillarreal · 2022-05-11T08:46:05Z

Hi @itaseskii,

CI is blocking this from being merged. I believe you have to bring the latest changes from master to fix it.

Could you try? Thanks

itaseskii · 2022-05-11T11:01:45Z

The merge conflicts have been resolved and the connector was bumped to 0.1.16

LuisMunozVillarreal · 2022-05-11T13:11:22Z

Hi @harshithmullapudi. Could you trigger another CI build on this? Thanks

harshithmullapudi · 2022-05-16T10:13:38Z

/test connector=source-mixpanel

🕑 source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2331212027

harshithmullapudi · 2022-05-23T07:42:36Z

Can you take a look at this? Do you see integration tests running in local?

itaseskii · 2022-05-23T09:44:03Z

Sure. what exactly do I need to look into? :)

harshithmullapudi · 2022-05-23T10:03:05Z

Can you confirm if the integration tests are passing?

itaseskii · 2022-05-23T10:38:49Z

Can we run the test pipeline again? From what I can see the last build has failed with the following reason

The self-hosted runner: ip-10-0-11-70 lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

roman-romanov-o · 2022-05-25T11:23:19Z

/test connector=source-mixpanel

🕑 source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2384077394
🕑 source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2384077394

misteryeo · 2022-06-02T04:46:19Z

@harshithmullapudi bumping this

itaseskii · 2022-06-02T14:08:41Z

@alafanechere How should we proceed with this? Should we close it in favor of #13372 since this fix was merged with the PR.

harshithmullapudi · 2022-06-02T15:42:28Z

Hey closing this then as the fix is already merged

fix mixpanel rate limiting

b22e041

github-actions bot added the area/connectors Connector related issues label Apr 22, 2022

octavia-squidington-iii added the community label Apr 22, 2022

itaseskii changed the title ~~fix mixpanel rate limiting~~ 🎉 Source Mixpanel: Fix Mixpanel 429 status code is not handled Apr 22, 2022

harshithmullapudi self-requested a review April 25, 2022 15:04

harshithmullapudi self-assigned this Apr 25, 2022

marcosmarxm added the sponsored label Apr 27, 2022

itaseskii added 2 commits May 2, 2022 13:12

Merge branch 'master' into mixpanel-rate-limit-fix

c4528e2

update version

60fa965

github-actions bot added the area/documentation Improvements or additions to documentation label May 2, 2022

lgomezm reviewed May 3, 2022

View reviewed changes

resolve conflicts & remove duplicate reqs limit

d2eb925

lgomezm approved these changes May 4, 2022

View reviewed changes

resolve conflicts

1acf63b

roman-romanov-o mentioned this pull request May 30, 2022

🎉 Source Mixpanel: Handle lowercase naming conflicts #13257

Closed

38 tasks

alafanechere unassigned harshithmullapudi Jun 2, 2022

alafanechere added bounty-S Maintainer program: claimable small bounty PR bounty labels Jun 2, 2022

alafanechere removed the sponsored label Jun 2, 2022

harshithmullapudi closed this Jun 2, 2022

bleonard added the reward-50 label Oct 31, 2022

octavia-squidington-iv added the connectors/source/mixpanel label Oct 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Mixpanel: Fix Mixpanel 429 status code is not handled #12286

🎉 Source Mixpanel: Fix Mixpanel 429 status code is not handled #12286

itaseskii commented Apr 22, 2022

harshithmullapudi commented Apr 27, 2022

itaseskii commented May 2, 2022

lgomezm May 3, 2022

itaseskii May 3, 2022

lgomezm left a comment

harshithmullapudi commented May 4, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented May 5, 2022 •

edited by github-actions bot

Loading

CLAassistant commented May 5, 2022 •

edited

Loading

LuisMunozVillarreal commented May 10, 2022

itaseskii commented May 10, 2022

LuisMunozVillarreal commented May 11, 2022 •

edited

Loading

itaseskii commented May 11, 2022

LuisMunozVillarreal commented May 11, 2022

harshithmullapudi commented May 16, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented May 23, 2022

itaseskii commented May 23, 2022

harshithmullapudi commented May 23, 2022

itaseskii commented May 23, 2022

roman-romanov-o commented May 25, 2022 •

edited by github-actions bot

Loading

misteryeo commented Jun 2, 2022

itaseskii commented Jun 2, 2022 •

edited

Loading

harshithmullapudi commented Jun 2, 2022

🎉 Source Mixpanel: Fix Mixpanel 429 status code is not handled #12286

🎉 Source Mixpanel: Fix Mixpanel 429 status code is not handled #12286

Conversation

itaseskii commented Apr 22, 2022

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

harshithmullapudi commented Apr 27, 2022

itaseskii commented May 2, 2022

lgomezm May 3, 2022

Choose a reason for hiding this comment

itaseskii May 3, 2022

Choose a reason for hiding this comment

lgomezm left a comment

Choose a reason for hiding this comment

harshithmullapudi commented May 4, 2022 • edited by github-actions bot Loading

harshithmullapudi commented May 5, 2022 • edited by github-actions bot Loading

CLAassistant commented May 5, 2022 • edited Loading

LuisMunozVillarreal commented May 10, 2022

itaseskii commented May 10, 2022

LuisMunozVillarreal commented May 11, 2022 • edited Loading

itaseskii commented May 11, 2022

LuisMunozVillarreal commented May 11, 2022

harshithmullapudi commented May 16, 2022 • edited by github-actions bot Loading

harshithmullapudi commented May 23, 2022

itaseskii commented May 23, 2022

harshithmullapudi commented May 23, 2022

itaseskii commented May 23, 2022

roman-romanov-o commented May 25, 2022 • edited by github-actions bot Loading

misteryeo commented Jun 2, 2022

itaseskii commented Jun 2, 2022 • edited Loading

harshithmullapudi commented Jun 2, 2022

harshithmullapudi commented May 4, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented May 5, 2022 •

edited by github-actions bot

Loading

CLAassistant commented May 5, 2022 •

edited

Loading

LuisMunozVillarreal commented May 11, 2022 •

edited

Loading

harshithmullapudi commented May 16, 2022 •

edited by github-actions bot

Loading

roman-romanov-o commented May 25, 2022 •

edited by github-actions bot

Loading

itaseskii commented Jun 2, 2022 •

edited

Loading