✨ Source Mixpanel: add page size to configuration to increase sync speed #41976

descampsk · 2024-07-16T07:43:37Z

What

The engage stream retrieves 1000 records per request and has a 60s delay between two requests.
To synchronise 1 Million records, we need 16.6 hours.

If we increase the page size to 40 000, we then need only 25 minutes.

How

This PR only adds the possibility to set the page size to a user defined value to be able to set higher page size to increase the sync speed.

Review guide

See comments.

User Impact

Great performance increase.

Can this PR be safely reverted and rolled back?

YES 💚
NO ❌

vercel · 2024-07-16T07:43:42Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Jul 25, 2024 2:47pm

descampsk · 2024-07-16T07:44:18Z

airbyte-integrations/connectors/source-mixpanel/source_mixpanel/components.py

@@ -291,7 +291,7 @@ def next_page_token(self, response, last_records: List[Mapping[str, Any]]) -> Op
        if total:
            self._total = total

-        if self._total and page_number is not None and self._total > self.page_size * (page_number + 1):
+        if self._total and page_number is not None and self._total > self._page_size * (page_number + 1):


To get the real page size from PageIncrement and not '{{ config.page_size }}'

airbyte-integrations/connectors/source-mixpanel/source_mixpanel/manifest.yaml

descampsk · 2024-07-16T07:45:01Z

airbyte-integrations/connectors/source-mixpanel/source_mixpanel/manifest.yaml

@@ -150,7 +153,7 @@ definitions:
      type: CustomPaginationStrategy
      class_name: "source_mixpanel.components.EngagePaginationStrategy"
      start_from_page: 1
-      page_size: 1000
+      page_size: "{{ config.page_size }}"


Allow the configuration of the page size

descampsk · 2024-07-16T07:45:16Z

airbyte-integrations/connectors/source-mixpanel/source_mixpanel/source.py

@@ -72,7 +72,7 @@ def validate_date(name: str, date_str: str, default: pendulum.date) -> pendulum.

    @adapt_validate_if_testing
    def _validate_and_transform(self, config: MutableMapping[str, Any]):
-        project_timezone, start_date, end_date, attribution_window, select_properties_by_default, region, date_window_size, project_id = (
+        project_timezone, start_date, end_date, attribution_window, select_properties_by_default, region, date_window_size, project_id, page_size = (


Not sure if this is needed

descampsk · 2024-07-16T07:45:31Z

airbyte-integrations/connectors/source-mixpanel/source_mixpanel/spec.json

@@ -119,6 +119,14 @@
        "type": "integer",
        "minimum": 1,
        "default": 30
+      },


Add the page size ton the user defined configuration

descampsk · 2024-07-16T07:45:47Z

airbyte-integrations/connectors/source-mixpanel/unit_tests/test_source.py

@@ -94,37 +94,37 @@ def test_streams_string_date(requests_mock, config_raw):
    "config, success, expected_error_message",
    (
        (
-            {"credentials": {"api_secret": "secret"}, "project_timezone": "Miami"},
+            {"credentials": {"api_secret": "secret"}, "project_timezone": "Miami", "page_size": 1000},


Needed otherwise the tests fail

descampsk · 2024-07-16T07:47:00Z

@natikgadzhi @ChristoGrab
Hello,
Here is the PR we discussed to greatly improve the sync speed of the engage stream :)

airbyte-integrations/connectors/source-mixpanel/source_mixpanel/spec.json

Co-authored-by: Christo Grabowski <108154848+ChristoGrab@users.noreply.github.com>

marcosmarxm · 2024-07-18T14:07:02Z

Running CI Tests to validate changes.

descampsk · 2024-07-23T20:28:05Z

@ChristoGrab @marcosmarxm
What can I do to make this PR ready to merge ?

Tests seems failing but I cannot see the logs

marcosmarxm · 2024-07-23T21:17:38Z

�[35m49:�[0m [1.29s] {"type": "LOG", "log": {"level": "FATAL", "message": " is of type <class 'str'>. Expected <class 'int'>\nTraceback (most recent call last):\n File "/airbyte/integration_code/main.py", line 8, in \n run()\n File "/airbyte/integration_code/source_mixpanel/run.py", line 16, in run\n launch(source, sys.argv[1:])\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 235, in launch\n for message in source_entrypoint.run(parsed_args):\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 115, in run\n yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.check(source_spec, config))\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 139, in check\n check_result = self.source.check(self.logger, config)\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 157, in check\n return super().check(logger, config)\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py", line 87, in check\n check_succeeded, error = self.check_connection(logger, config)\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/declarative_source.py", line 33, in check_connection\n return self.connection_checker.check_connection(self, logger, config)\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/checks/check_stream.py", line 31, in check_connection\n streams = source.streams(config)\n File "/airbyte/integration_code/source_mixpanel/source.py", line 45, in streams\n streams = super().streams(config=config)\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 97, in streams\n source_streams = [\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 98, in \n self._constructor.create_component(\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py", line 240, in create_component\n return self._create_component_from_model(model=declarative_component_model, config=config, **kwargs)\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py", line 248, in _create_component_from_model\n return component_constructor(model=model, config=config, **kwargs)\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py", line 607, in create_declarative_stream\n retriever = self._create_component_from_model(\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py", line 248, in _create_component_from_model\n return component_constructor(model=model, config=config, **kwargs)\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py", line 1009, in create_simple_retriever\n self._create_component_from_model(\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py", line 248, in _create_component_from_model\n return component_constructor(model=model, config=config, **kwargs)\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py", line 712, in create_default_paginator\n pagination_strategy = self._create_component_from_model(model=model.pagination_strategy, config=config)\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py", line 248, in _create_component_from_model\n return component_constructor(model=model, config=config, **kwargs)\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py", line 462, in create_custom_component\n return custom_component_class(**kwargs)\n File "", line 7, in init\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/requesters/paginators/strategies/page_increment.py", line 37, in post_init\n raise Exception(f"{page_size} is of type {type(page_size)}. Expected {int}")\nException: is of type <class 'str'>. Expected <class 'int'>"}}

descampsk · 2024-07-24T07:31:53Z

Thanks, will investigate that

descampsk · 2024-07-24T07:46:43Z

Do you know why the default value from spec.json

      "page_size": {
        "order": 9,
        "title": "Page Size",
        "description": "The number of records to fetch per request. Default is 1000.",
        "type": "integer",
        "minimum": 1,
        "default": 1000
      }

is not taken ?

I can reproduce in the UI

With default, I have the error, If i use a value, I have no error.

descampsk · 2024-07-24T07:59:10Z

@marcosmarxm I fixed the issue, could you run the tests again ? :)

ChristoGrab

@descampsk Thanks for yet another solid contribution! Regression tests just came back green so we're all good to merge, nicely done 🙌

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation community connectors/source/mixpanel labels Jul 16, 2024

descampsk had a problem deploying to community-ci-auto July 16, 2024 07:43 — with GitHub Actions Failure

descampsk temporarily deployed to community-ci-auto July 16, 2024 07:43 — with GitHub Actions Inactive

descampsk had a problem deploying to community-ci July 16, 2024 07:43 — with GitHub Actions Error

descampsk commented Jul 16, 2024

View reviewed changes

airbyte-integrations/connectors/source-mixpanel/source_mixpanel/manifest.yaml Outdated Show resolved Hide resolved

descampsk commented Jul 16, 2024

View reviewed changes

vercel bot deployed to Preview July 16, 2024 07:48 View deployment

feat(source-mixpanel): add page size to config to increase sync speed

3823c9b

descampsk force-pushed the source-mixpanel-add-page-size-config branch from bc0c587 to 3823c9b Compare July 16, 2024 08:21

descampsk temporarily deployed to community-ci-auto July 16, 2024 08:21 — with GitHub Actions Inactive

descampsk temporarily deployed to community-ci July 16, 2024 08:21 — with GitHub Actions Inactive

vercel bot deployed to Preview July 16, 2024 08:25 View deployment

ChristoGrab reviewed Jul 17, 2024

View reviewed changes

airbyte-integrations/connectors/source-mixpanel/source_mixpanel/spec.json Outdated Show resolved Hide resolved

Update page size description

7ffb854

Co-authored-by: Christo Grabowski <108154848+ChristoGrab@users.noreply.github.com>

descampsk temporarily deployed to community-ci July 17, 2024 19:51 — with GitHub Actions Inactive

descampsk temporarily deployed to community-ci-auto July 17, 2024 19:51 — with GitHub Actions Inactive

descampsk temporarily deployed to community-ci July 17, 2024 19:51 — with GitHub Actions Inactive

fix(source-mixpanel): fix default value page_size

5856f32

descampsk temporarily deployed to community-ci-auto July 24, 2024 07:53 — with GitHub Actions Inactive

descampsk had a problem deploying to community-ci July 24, 2024 07:53 — with GitHub Actions Error

descampsk temporarily deployed to community-ci-auto July 24, 2024 07:53 — with GitHub Actions Inactive

add 500 error code to handler

7c6e561

ChristoGrab temporarily deployed to community-ci July 25, 2024 14:51 — with GitHub Actions Inactive

ChristoGrab temporarily deployed to community-ci-auto July 25, 2024 14:51 — with GitHub Actions Inactive

ChristoGrab temporarily deployed to community-ci July 25, 2024 14:51 — with GitHub Actions Inactive

ChristoGrab approved these changes Jul 25, 2024

View reviewed changes

ChristoGrab merged commit 0a304a9 into airbytehq:master Jul 25, 2024
35 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Source Mixpanel: add page size to configuration to increase sync speed #41976

✨ Source Mixpanel: add page size to configuration to increase sync speed #41976

descampsk commented Jul 16, 2024

vercel bot commented Jul 16, 2024 •

edited

Loading

descampsk Jul 16, 2024

descampsk Jul 16, 2024

descampsk Jul 16, 2024

descampsk Jul 16, 2024

descampsk Jul 16, 2024

descampsk commented Jul 16, 2024

marcosmarxm commented Jul 18, 2024

descampsk commented Jul 23, 2024 •

edited

Loading

marcosmarxm commented Jul 23, 2024

descampsk commented Jul 24, 2024

descampsk commented Jul 24, 2024

descampsk commented Jul 24, 2024

ChristoGrab left a comment

✨ Source Mixpanel: add page size to configuration to increase sync speed #41976

✨ Source Mixpanel: add page size to configuration to increase sync speed #41976

Conversation

descampsk commented Jul 16, 2024

What

How

Review guide

User Impact

Can this PR be safely reverted and rolled back?

vercel bot commented Jul 16, 2024 • edited Loading

descampsk Jul 16, 2024

Choose a reason for hiding this comment

descampsk Jul 16, 2024

Choose a reason for hiding this comment

descampsk Jul 16, 2024

Choose a reason for hiding this comment

descampsk Jul 16, 2024

Choose a reason for hiding this comment

descampsk Jul 16, 2024

Choose a reason for hiding this comment

descampsk commented Jul 16, 2024

marcosmarxm commented Jul 18, 2024

descampsk commented Jul 23, 2024 • edited Loading

marcosmarxm commented Jul 23, 2024

descampsk commented Jul 24, 2024

descampsk commented Jul 24, 2024

descampsk commented Jul 24, 2024

ChristoGrab left a comment

Choose a reason for hiding this comment

vercel bot commented Jul 16, 2024 •

edited

Loading

descampsk commented Jul 23, 2024 •

edited

Loading