Source Monday: fix boards stream 25 row limit in source-monday

[igsaf2]

What

Resolves #24194

How

The problem in the default 25 item limit. E.g. you can see it in monday forum.
The Monday connector uses limit parameter to configure the number of entries to fetch (function)
This limit is set to be equal items_per_page parameter. (manifest link).
items_per_page must be not less than page_size in paginator (link), because otherwise next_page_token method will return None (pagination strategy)

🚨 User Impact 🚨

I believe there are no breaking changes

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)

• Secrets in the connector's spec are annotated with airbyte_secret

• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.

• Code reviews completed

• Connector version has been incremented

  • Version has been bumped according to our Semantic Versioning for Connectors guidelines
  • Dockerfile has updated version

• Documentation updated

  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example

• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

[CLAassistant]

All committers have signed the CLA.

[marcosmarxm]

The current Pagination already uses 100 items check default_paginator component. Are you able to build the dev version and test it?

[igsaf2]

I have run python main.py read --config secrets/config.json --catalog configured_stream.json . It works for us with 200+ boards.
This is not only about pagination, but also about limit parameter;

airbyte/airbyte-integrations/connectors/source-monday/source_monday/manifest.yaml

Line 22 in 44fa284

limit: "{{ parameters['items_per_page'] }}"

Right now the logic is following:

1. limit becomes None in here
2. Because limit is not specified, Monday returns default 25 rows for boards.
3. Pagination checks if it needs to visit another page with the condition: len(last_records) < self.page_size. And it gets 25 < 100. So there is no query for the second page.

This PR solves the issue: the limit is set and pagination starts working when needed. Although it does not look self-explaining

[igsaf2]

(venv) igsaf@igsaf-nix:~/IdeaProjects/airbyte/airbyte-integrations/connectors/source-monday (fix-monday-boards) $ python main.py read --config secrets/config.json --catalog configured_stream.json | tail -n 4 
{"type": "LOG", "log": {"level": "INFO", "message": "Read 221 records from boards stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing boards"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceMonday runtimes:\nSyncing stream boards 0:00:21.135193"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing SourceMonday"}}

[marcosmarxm]

/test connector=connectors/source-monday

🕑 connectors/source-monday https://github.com/airbytehq/airbyte/actions/runs/4753757389
✅ connectors/source-monday https://github.com/airbytehq/airbyte/actions/runs/4753757389
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_monday/__init__.py                3      0   100%
source_monday/graphql_requester.py      58      5    91%
source_monday/source.py                  4      1    75%
--------------------------------------------------------
TOTAL                                   65      6    91%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: Incremental syncs are not supported on this connector.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [3] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
================== 55 passed, 5 skipped in 385.68s (0:06:25) ===================

[igsaf2]

ping

[mathieulebasic]

Hello @marcosmarxm, what are the roadblockers right now ? Do you have an ETA for the merge ?

[marcosmarxm]

/publish connector=connectors/source-monday

🕑 Publishing the following connectors:
connectors/source-monday
https://github.com/airbytehq/airbyte/actions/runs/4811550881

Connector	Version	Did it publish?	Were definitions generated?
connectors/source-monday	0.2.4	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

[marcosmarxm]

@igsaf2 there is a bug in our CI. Team is fixing it and after I'll merge your contirbution. You can already update the connector version in the Airbyte UI.

[mathieulebasic]

@marcosmarxm I resynced my connector but still had the problem of limits (I am limited to 100 and not 25...). Should I re-create my source or is there anything to do ? I might not be on the new connector.

[igsaf2]

@mathieulebasic
In the web interface pick settings (in the left lower corner) -> sources -> and there should be Monday version above or equal to 0.2.4.

It seems to be a question about how to update a connector, check out Airbyte slack workspace, it is great.