Source Posthog: increase batch size (#25603)

* increase events api limit to 10000 * add docs * Bump dockerfile * update seed file * Automated Change --------- Co-authored-by: Haithem Souala <haithem.souala@woopit.fr>
airbytehq · Apr 27, 2023 · 87c07f3 · 87c07f3
1 parent fd36c06
commit 87c07f3
Show file tree

Hide file tree

Showing 7 changed files with 31 additions and 29 deletions.
diff --git a/airbyte-config-oss/init-oss/src/main/resources/seed/oss_catalog.json b/airbyte-config-oss/init-oss/src/main/resources/seed/oss_catalog.json
@@ -21171,7 +21171,7 @@
     "sourceDefinitionId": "af6d50ee-dddf-4126-a8ee-7faee990774f",
     "name": "PostHog",
     "dockerRepository": "airbyte/source-posthog",
-    "dockerImageTag": "0.1.9",
+    "dockerImageTag": "0.1.10",
     "documentationUrl": "https://docs.airbyte.com/integrations/sources/posthog",
     "icon": "posthog.svg",
     "sourceType": "api",

diff --git a/airbyte-config-oss/init-oss/src/main/resources/seed/source_definitions.yaml b/airbyte-config-oss/init-oss/src/main/resources/seed/source_definitions.yaml
@@ -1607,7 +1607,7 @@
 - name: PostHog
   sourceDefinitionId: af6d50ee-dddf-4126-a8ee-7faee990774f
   dockerRepository: airbyte/source-posthog
-  dockerImageTag: 0.1.9
+  dockerImageTag: 0.1.10
   documentationUrl: https://docs.airbyte.com/integrations/sources/posthog
   icon: posthog.svg
   sourceType: api

diff --git a/airbyte-config-oss/init-oss/src/main/resources/seed/source_specs.yaml b/airbyte-config-oss/init-oss/src/main/resources/seed/source_specs.yaml
@@ -12118,7 +12118,7 @@
     supportsNormalization: false
     supportsDBT: false
     supported_destination_sync_modes: []
-- dockerImage: "airbyte/source-posthog:0.1.9"
+- dockerImage: "airbyte/source-posthog:0.1.10"
   spec:
     documentationUrl: "https://docs.airbyte.com/integrations/sources/posthog"
     connectionSpecification:

diff --git a/airbyte-integrations/connectors/source-posthog/Dockerfile b/airbyte-integrations/connectors/source-posthog/Dockerfile
@@ -12,5 +12,5 @@ RUN pip install .
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.1.9
+LABEL io.airbyte.version=0.1.10
 LABEL io.airbyte.name=airbyte/source-posthog
diff --git a/airbyte-integrations/connectors/source-posthog/source_posthog/manifest.yaml b/airbyte-integrations/connectors/source-posthog/source_posthog/manifest.yaml
@@ -145,7 +145,7 @@ definitions:
         pagination_strategy:
           type: "CursorPagination"
           cursor_value: "{{ response['next'] }}"
-          page_size: 100
+          page_size: 10000
         page_token_option:
           type: RequestPath
         $parameters:

diff --git a/connectors.md b/connectors.md
@@ -182,7 +182,7 @@
 | **Pocket** | <img alt="Pocket icon" src="https://raw.githubusercontent.com/airbytehq/airbyte /master/airbyte-config-oss/init-oss/src/main/resources/icons/pocket.svg" height="30" height="30"/> | Source | airbyte/source-pocket:0.1.0 | alpha | [docs](https://docs.airbyte.com/integrations/sources/pocket) | [connectors/source/pocket](https://github.com/airbytehq/airbyte/issues?q=is:open+is:issue+label:connectors/source/pocket) | [source-pocket](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-pocket) | <small>`b0dd65f1-081f-4731-9c51-38e9e6aa0ebf`</small> |
 | **PokeAPI** | <img alt="PokeAPI icon" src="https://raw.githubusercontent.com/airbytehq/airbyte /master/airbyte-config-oss/init-oss/src/main/resources/icons/pokeapi.svg" height="30" height="30"/> | Source | airbyte/source-pokeapi:0.1.5 | alpha | [docs](https://docs.airbyte.com/integrations/sources/pokeapi) | [connectors/source/pokeapi](https://github.com/airbytehq/airbyte/issues?q=is:open+is:issue+label:connectors/source/pokeapi) | [source-pokeapi](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-pokeapi) | <small>`6371b14b-bc68-4236-bfbd-468e8df8e968`</small> |
 | **Polygon Stock API** | <img alt="Polygon Stock API icon" src="https://raw.githubusercontent.com/airbytehq/airbyte /master/airbyte-config-oss/init-oss/src/main/resources/icons/polygon.svg" height="30" height="30"/> | Source | airbyte/source-polygon-stock-api:0.1.1 | alpha | [docs](https://docs.airbyte.com/integrations/sources/polygon-stock-api) | [connectors/source/polygon-stock-api](https://github.com/airbytehq/airbyte/issues?q=is:open+is:issue+label:connectors/source/polygon-stock-api) | [source-polygon-stock-api](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-polygon-stock-api) | <small>`5807d72f-0abc-49f9-8fa5-ae820007032b`</small> |
-| **PostHog** | <img alt="PostHog icon" src="https://raw.githubusercontent.com/airbytehq/airbyte /master/airbyte-config-oss/init-oss/src/main/resources/icons/posthog.svg" height="30" height="30"/> | Source | airbyte/source-posthog:0.1.9 | beta | [docs](https://docs.airbyte.com/integrations/sources/posthog) | [connectors/source/posthog](https://github.com/airbytehq/airbyte/issues?q=is:open+is:issue+label:connectors/source/posthog) | [source-posthog](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-posthog) | <small>`af6d50ee-dddf-4126-a8ee-7faee990774f`</small> |
+| **PostHog** | <img alt="PostHog icon" src="https://raw.githubusercontent.com/airbytehq/airbyte /master/airbyte-config-oss/init-oss/src/main/resources/icons/posthog.svg" height="30" height="30"/> | Source | airbyte/source-posthog:0.1.10 | beta | [docs](https://docs.airbyte.com/integrations/sources/posthog) | [connectors/source/posthog](https://github.com/airbytehq/airbyte/issues?q=is:open+is:issue+label:connectors/source/posthog) | [source-posthog](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-posthog) | <small>`af6d50ee-dddf-4126-a8ee-7faee990774f`</small> |
 | **Postgres** | <img alt="Postgres icon" src="https://raw.githubusercontent.com/airbytehq/airbyte /master/airbyte-config-oss/init-oss/src/main/resources/icons/postgresql.svg" height="30" height="30"/> | Source | airbyte/source-postgres:2.0.27 | generally_available | [docs](https://docs.airbyte.com/integrations/sources/postgres) | [connectors/source/postgres](https://github.com/airbytehq/airbyte/issues?q=is:open+is:issue+label:connectors/source/postgres) | [source-postgres](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-postgres) | <small>`decd338e-5647-4c0b-adf4-da0e75f5a750`</small> |
 | **Postmark App** | <img alt="Postmark App icon" src="https://raw.githubusercontent.com/airbytehq/airbyte /master/airbyte-config-oss/init-oss/src/main/resources/icons/postmark.svg" height="30" height="30"/> | Source | airbyte/source-postmarkapp:0.1.0 | alpha | [docs](https://docs.airbyte.com/integrations/sources/postmarkapp) | [connectors/source/postmarkapp](https://github.com/airbytehq/airbyte/issues?q=is:open+is:issue+label:connectors/source/postmarkapp) | [source-postmarkapp](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-postmarkapp) | <small>`cde75ca1-1e28-4a0f-85bb-90c546de9f1f`</small> |
 | **PrestaShop** | <img alt="PrestaShop icon" src="https://raw.githubusercontent.com/airbytehq/airbyte /master/airbyte-config-oss/init-oss/src/main/resources/icons/prestashop.svg" height="30" height="30"/> | Source | airbyte/source-prestashop:0.3.1 | beta | [docs](https://docs.airbyte.com/integrations/sources/prestashop) | [connectors/source/prestashop](https://github.com/airbytehq/airbyte/issues?q=is:open+is:issue+label:connectors/source/prestashop) | [source-prestashop](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-prestashop) | <small>`d60a46d4-709f-4092-a6b7-2457f7d455f5`</small> |

diff --git a/docs/integrations/sources/posthog.md b/docs/integrations/sources/posthog.md
@@ -4,13 +4,14 @@ This page contains the setup guide and reference information for the PostHog sou
 
 ## Prerequisites
 
-* api_key - obtain Private API Key for your account following these [steps](https://posthog.com/docs/api/overview#how-to-obtain-a-personal-api-key)
-* base_url - 'https://app.posthog.com' by default, but it can be different if self-hosted posthog instances is used 
+- api_key - obtain Private API Key for your account following these [steps](https://posthog.com/docs/api/overview#how-to-obtain-a-personal-api-key)
+- base_url - 'https://app.posthog.com' by default, but it can be different if self-hosted posthog instances is used
 
 ## Setup guide
+
 ### Step 1: Set up PostHog
 
-* PostHog Account
+- PostHog Account
 
 ## Step 2: Set up the PostHog connector in Airbyte
 
@@ -20,28 +21,28 @@ This page contains the setup guide and reference information for the PostHog sou
 2. In the left navigation bar, click **Sources**. In the top-right corner, click **+new source**.
 3. On the Set up the source page, enter the name for the PostHog connector and select **PostHog** from the Source type dropdown.
 4. Enter your `apikey`.
-5. Enter your `start_date`. 
+5. Enter your `start_date`.
 6. Change default `base_url` if self-hosted posthog instances is used
 7. Click **Set up source**.
 
 ### For Airbyte OSS:
 
 1. Navigate to the Airbyte Open Source dashboard.
-2. Set the name for your source. 
+2. Set the name for your source.
 3. Enter your `api_key`.
-4. Enter your `start_date`. 
+4. Enter your `start_date`.
 5. Change default `base_url` if self-hosted posthog instances is used
 6. Click **Set up source**.
 
 ## Supported streams and sync modes
 
-* [Projects](https://posthog.com/docs/api/projects)
-* [Annotations](https://posthog.com/docs/api/annotations)
-* [Cohorts](https://posthog.com/docs/api/cohorts)
-* [Events](https://posthog.com/docs/api/events) \(Incremental\)
-* [FeatureFlags](https://posthog.com/docs/api/feature-flags)
-* [Insights](https://posthog.com/docs/api/insights)
-* [Persons](https://posthog.com/docs/api/people)
+- [Projects](https://posthog.com/docs/api/projects)
+- [Annotations](https://posthog.com/docs/api/annotations)
+- [Cohorts](https://posthog.com/docs/api/cohorts)
+- [Events](https://posthog.com/docs/api/events) \(Incremental\)
+- [FeatureFlags](https://posthog.com/docs/api/feature-flags)
+- [Insights](https://posthog.com/docs/api/insights)
+- [Persons](https://posthog.com/docs/api/people)
 
 ### Performance considerations
 
@@ -50,15 +51,16 @@ Please [create an issue](https://github.com/airbytehq/airbyte/issues) if you see
 
 ## Changelog
 
-| Version | Date       | Pull Request | Subject                                                                                                            |
-|:--------|:-----------| :--- |:-------------------------------------------------------------------------------------------------------------------|
-| 0.1.9   | 2023-02-13 | [22906](https://github.com/airbytehq/airbyte/pull/22906) | Specified date formatting in specification   |
+| Version | Date       | Pull Request                                             | Subject                                                                                                            |
+| :------ | :--------- | :------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------- |
+| 0.1.10  | 2023-04-15 | [24084](https://github.com/airbytehq/airbyte/pull/24084) | Increase `events` streams batch size                                                                               |
+| 0.1.9   | 2023-02-13 | [22906](https://github.com/airbytehq/airbyte/pull/22906) | Specified date formatting in specification                                                                         |
 | 0.1.8   | 2022-11-11 | [18993](https://github.com/airbytehq/airbyte/pull/18993) | connector migrated to low-code, added projects,insights streams, added project based slicing for all other streams |
 | 0.1.7   | 2022-07-26 | [14585](https://github.com/airbytehq/airbyte/pull/14585) | Add missing 'properties' field to event attributes                                                                 |
-| 0.1.6   | 2022-01-20 | [8617](https://github.com/airbytehq/airbyte/pull/8617) | Update connector fields title/description                                                                          |
-| 0.1.5   | 2021-12-24 | [9082](https://github.com/airbytehq/airbyte/pull/9082) | Remove obsolete session_events and insights streams                                                                |
-| 0.1.4   | 2021-09-14 | [6058](https://github.com/airbytehq/airbyte/pull/6058) | Support self-hosted posthog instances                                                                              |
-| 0.1.3   | 2021-07-20 | [4001](https://github.com/airbytehq/airbyte/pull/4001) | Incremental streams read only relevant pages                                                                       |
-| 0.1.2   | 2021-07-15 | [4692](https://github.com/airbytehq/airbyte/pull/4692) | Use account information for checking the connection                                                                |
-| 0.1.1   | 2021-07-05 | [4539](https://github.com/airbytehq/airbyte/pull/4539) | Add `AIRBYTE_ENTRYPOINT` env variable for kubernetes support                                                       |
-| 0.1.0   | 2021-06-08 | [3768](https://github.com/airbytehq/airbyte/pull/3768) | Initial Release                                                                                                    |
+| 0.1.6   | 2022-01-20 | [8617](https://github.com/airbytehq/airbyte/pull/8617)   | Update connector fields title/description                                                                          |
+| 0.1.5   | 2021-12-24 | [9082](https://github.com/airbytehq/airbyte/pull/9082)   | Remove obsolete session_events and insights streams                                                                |
+| 0.1.4   | 2021-09-14 | [6058](https://github.com/airbytehq/airbyte/pull/6058)   | Support self-hosted posthog instances                                                                              |
+| 0.1.3   | 2021-07-20 | [4001](https://github.com/airbytehq/airbyte/pull/4001)   | Incremental streams read only relevant pages                                                                       |
+| 0.1.2   | 2021-07-15 | [4692](https://github.com/airbytehq/airbyte/pull/4692)   | Use account information for checking the connection                                                                |
+| 0.1.1   | 2021-07-05 | [4539](https://github.com/airbytehq/airbyte/pull/4539)   | Add `AIRBYTE_ENTRYPOINT` env variable for kubernetes support                                                       |
+| 0.1.0   | 2021-06-08 | [3768](https://github.com/airbytehq/airbyte/pull/3768)   | Initial Release                                                                                                    |