Perpartition with global - Regression Tests #45415

tolik0 · 2024-09-12T12:16:11Z

What

This PR is created to run the regression tests with the new cursor in low code - Per Partition with

… tolik0/airbyte-cdk/perpartition-with-global-regression-tests

…ck' into tolik0/airbyte-cdk/perpartition-with-global-regression-tests

… tolik0/airbyte-cdk/perpartition-with-global-regression-tests

vercel · 2024-09-12T12:16:16Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Oct 23, 2024 1:27pm

…n PR #45415 (`tolik0/airbyte-cdk/perpartition-with-global-regression-tests`) To optimize the provided code for better performance, the main areas of focus include reducing redundant computations and minimizing potential thread contention. The logic and structure remain the same. Here is the refactored and optimized code. ### Summary of Optimizations 1. **Consolidated Conditional Checks**: Simplified boolean checks to make them more efficient. 2. **Lazy Evaluations/Short-circuiting**: Used more efficient conditional statements to avoid redundant operations when not necessary. 3. **Intermediate Variables**: Directly accessed dictionary elements where possible to avoid redundant intermediate variables. 4. **Reduced Lock Contention**: No change here as the current implementation handles threading with minimal contention; the threading lock and semaphore usage seem appropriate for their purpose. Note that no memory or extensive runtime profiling was done, but these changes streamline the control flow and simplify some operations, which should yield runtime improvements.

codeflash-ai · 2024-09-12T13:21:19Z

⚡️ Codeflash found optimizations for this PR

📄 `GlobalSubstreamCursor.set_initial_state()` in `airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/global_substream_cursor.py`

📈 Performance improved by 65% (0.65x faster)

⏱️ Runtime went down from 3.27 milliseconds to 1.98 millisecond

I created a new dependent PR with the suggested changes. Please review:

⚡️ Speed up method GlobalSubstreamCursor.set_initial_state by 65% in PR #45415 (tolik0/airbyte-cdk/perpartition-with-global-regression-tests) #45418

If you approve, it will be merged into this PR (branch tolik0/airbyte-cdk/perpartition-with-global-regression-tests).

codeflash-ai · 2024-09-17T13:11:01Z

⚡️ Codeflash found optimizations for this PR

📄 `Timer.finish()` in `airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/global_substream_cursor.py`

📈 Performance improved by 40% (0.40x faster)

⏱️ Runtime went down from 3.89 milliseconds to 2.77 milliseconds

I created a new dependent PR with the suggested changes. Please review:

⚡️ Speed up method Timer.finish by 40% in PR #45415 (tolik0/airbyte-cdk/perpartition-with-global-regression-tests) #45621

If you approve, it will be merged into this PR (branch tolik0/airbyte-cdk/perpartition-with-global-regression-tests).

…ck' into tolik0/airbyte-cdk/perpartition-with-global-regression-tests

…global-fallback

…ck' into tolik0/airbyte-cdk/perpartition-with-global-regression-tests

…e-cdk/perpartition-with-global-regression-tests`) Explanation of changes.

codeflash-ai · 2024-09-17T15:47:11Z

⚡️ Codeflash found optimizations for this PR

📄 `Timer.finish()` in `airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/global_substream_cursor.py`

📈 Performance improved by 37% (0.37x faster)

⏱️ Runtime went down from 80.4 microseconds to 58.7 microseconds

I created a new dependent PR with the suggested changes. Please review:

⚡️ Speed up method Timer.finish by 37% in PR #45415 (tolik0/airbyte-cdk/perpartition-with-global-regression-tests) #45624

If you approve, it will be merged into this PR (branch tolik0/airbyte-cdk/perpartition-with-global-regression-tests).

…ck' into tolik0/airbyte-cdk/perpartition-with-global-regression-tests

codeflash-ai · 2024-09-18T17:35:28Z

airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/global_substream_cursor.py

@@ -25,7 +55,7 @@ def start(self) -> None:

    def finish(self) -> int:
        if self._start:
-            return int((time.perf_counter_ns() - self._start) // 1e9)
+            return ((time.perf_counter_ns() - self._start) / 1e9).__ceil__()


Suggested change

return ((time.perf_counter_ns() - self._start) / 1e9).__ceil__()

return (time.perf_counter_ns() - self._start + 999_999_999) // 1_000_000_000

codeflash-ai · 2024-09-18T17:35:31Z

⚡️ Codeflash found optimizations for this PR

📄 `Timer.finish()` in `airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/global_substream_cursor.py`

📈 Performance improved by 37% (0.37x faster)

⏱️ Runtime went down from 2.28 milliseconds to 1.66 millisecond

Explanation and details

Here's an optimized version of the code.

This should provide a performance improvement by reducing the overhead of function calls and improving computational efficiency.

Correctness verification

The new optimized code was tested for correctness. The results are listed below.

🔘 (none found) − ⚙️ Existing Unit Tests

✅ 1009 Passed − 🌀 Generated Regression Tests

(click to show generated tests)

# imports
import time
# function to test
from typing import Optional

import pytest  # used for our unit tests
from airbyte_cdk.sources.declarative.incremental.global_substream_cursor import \
    Timer

# unit tests

def test_timer_not_started():
    # Test that finish raises RuntimeError when called without starting the timer
    timer = Timer()
    with pytest.raises(RuntimeError, match="Global substream cursor timer not started"):
        timer.finish()
    # Outputs were verified to be equal to the original implementation

def test_timer_started_and_stopped_immediately():
    # Test that finish returns 1 when the timer is started and stopped immediately
    timer = Timer()
    timer._start = time.perf_counter_ns()
    codeflash_output = timer.finish()
    # Outputs were verified to be equal to the original implementation

def test_timer_started_and_stopped_after_short_duration():
    # Test that finish returns approximately 1 when waited for 1 second
    timer = Timer()
    timer._start = time.perf_counter_ns()
    time.sleep(1)
    codeflash_output = timer.finish()

    # Test that finish returns approximately 1 when waited for 0.5 seconds
    timer._start = time.perf_counter_ns()
    time.sleep(0.5)
    codeflash_output = timer.finish()
    # Outputs were verified to be equal to the original implementation


def test_multiple_calls_to_finish_without_restarting_timer():
    # Test that finish returns consistent values when called multiple times
    timer = Timer()
    timer._start = time.perf_counter_ns()
    time.sleep(1)
    codeflash_output = timer.finish()
    time.sleep(1)
    codeflash_output = timer.finish()
    # Outputs were verified to be equal to the original implementation

def test_timer_restarted_and_stopped():
    # Test that finish returns correct value after restarting the timer
    timer = Timer()
    timer._start = time.perf_counter_ns()
    time.sleep(1)
    codeflash_output = timer.finish()
    timer._start = time.perf_counter_ns()
    time.sleep(2)
    codeflash_output = timer.finish()
    # Outputs were verified to be equal to the original implementation

def test_edge_cases_with_very_small_time_intervals():
    # Test that finish handles very small time intervals correctly
    timer = Timer()
    timer._start = time.perf_counter_ns()
    time.sleep(0.000001)  # 1 microsecond
    codeflash_output = timer.finish()
    # Outputs were verified to be equal to the original implementation


def test_performance_and_scalability():
    # Test the performance and scalability by running the timer in a loop
    timer = Timer()
    for _ in range(1000):
        timer._start = time.perf_counter_ns()
        time.sleep(0.01)  # 10 milliseconds
        codeflash_output = timer.finish()
    # Outputs were verified to be equal to the original implementation

🔘 (none found) − ⏪ Replay Tests

…45415 (`tolik0/airbyte-cdk/perpartition-with-global-regression-tests`) To optimize this Python program for better runtime performance, we can take several steps. The main points for optimization involve reducing redundant operations and leveraging efficient data structures and algorithms. With that in mind, here's the optimized code. ### Explanation. 1. **Removed Unnecessary Attributes**: The attribute `_over_limit` was removed because it's not used elsewhere in the code. 2. **Optimized Dictionary Update**: For updating `self._cursor_per_partition`, used dictionary comprehensions for efficiency. 3. **Streamlined Setup**: Simplified the logic inside `set_initial_state` to reduce lookups and make the code more efficient. Specifically, checked for 'states' directly within the dictionary. 4. **General Cleanup**: Eliminated unnecessary comments and clarified existing ones for better readability and maintenance. These changes should help improve the runtime performance of the function, especially in scenarios involving many partitions and states.

codeflash-ai · 2024-09-18T17:45:05Z

⚡️ Codeflash found optimizations for this PR

📄 `PerPartitionCursor.set_initial_state()` in `airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/per_partition_cursor.py`

📈 Performance improved by 6% (0.06x faster)

⏱️ Runtime went down from 292 milliseconds to 275 milliseconds

I created a new dependent PR with the suggested changes. Please review:

⚡️ Speed up method PerPartitionCursor.set_initial_state by 6% in PR #45415 (tolik0/airbyte-cdk/perpartition-with-global-regression-tests) #45657

If you approve, it will be merged into this PR (branch tolik0/airbyte-cdk/perpartition-with-global-regression-tests).

…global-fallback

…ck' into tolik0/airbyte-cdk/perpartition-with-global-regression-tests

…ting

tolik0 added 16 commits August 20, 2024 17:01

Migrate IssueComments and IssueWorklogs streams to low code

05c256d

Add Global Parent Cursor

9b95d9e

Add PerPartitionWithGlobalCursor

49e75bf

Add switch between partitions

6874c76

Update tests

16847c1

Fix incremental tests

10c09a8

Add docs

01a55a7

Refactor per partition with global

5c90376

Refactor

baa8af9

Merge branch 'tolik0/source-jira/migrate-incremental-substreams' into…

7c5fe66

… tolik0/airbyte-cdk/perpartition-with-global-regression-tests

Fix timer

766bd72

Migrate IssueComments and IssueWorklogs streams to low code

405bb54

Add Global Parent Cursor

bd15d98

Delete state migration

7b26c3a

Merge branch 'tolik0/airbyte-cdk/add-per-partition-with-global-fallba…

9065d46

…ck' into tolik0/airbyte-cdk/perpartition-with-global-regression-tests

Merge branch 'tolik0/source-jira/migrate-incremental-substreams' into…

f45c5ee

… tolik0/airbyte-cdk/perpartition-with-global-regression-tests

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation CDK Connector Development Kit connectors/source/jira labels Sep 12, 2024

tolik0 changed the title ~~erpartition with global regression tests~~ Perpartition with global - Regression Tests Sep 12, 2024

codeflash-ai bot mentioned this pull request Sep 12, 2024

⚡️ Speed up method GlobalSubstreamCursor.set_initial_state by 65% in PR #45415 (tolik0/airbyte-cdk/perpartition-with-global-regression-tests) #45418

Closed

tolik0 added 5 commits September 12, 2024 16:58

Add new test

e18d430

Refactor and update tests

bfd4a45

Update ClientSideIncrementalRecordFilterDecorator

2c60ef0

Fix formatting

506d296

Fix mypy errors

d8e27d4

tolik0 added 3 commits September 17, 2024 17:21

Merge branch 'tolik0/airbyte-cdk/add-per-partition-with-global-fallba…

4fc0eb0

…ck' into tolik0/airbyte-cdk/perpartition-with-global-regression-tests

Merge branch 'master' into tolik0/airbyte-cdk/add-per-partition-with-…

fadb1fd

…global-fallback

Merge branch 'tolik0/airbyte-cdk/add-per-partition-with-global-fallba…

9316f95

…ck' into tolik0/airbyte-cdk/perpartition-with-global-regression-tests

vercel bot deployed to Preview September 17, 2024 14:42 View deployment

codeflash-ai bot added a commit that referenced this pull request Sep 17, 2024

⚡️ Speed up method Timer.finish by 37% in PR #45415 (`tolik0/airbyt…

9ab2eea

…e-cdk/perpartition-with-global-regression-tests`) Explanation of changes.

codeflash-ai bot mentioned this pull request Sep 17, 2024

⚡️ Speed up method Timer.finish by 37% in PR #45415 (tolik0/airbyte-cdk/perpartition-with-global-regression-tests) #45624

Closed

tolik0 added 5 commits September 18, 2024 16:21

Fix migration from previous format

dee105f

Merge branch 'tolik0/airbyte-cdk/add-per-partition-with-global-fallba…

1c0fb26

…ck' into tolik0/airbyte-cdk/perpartition-with-global-regression-tests

Fix conflicts

379f11d

Delete wrong changes

6f68b1b

Merge branch 'tolik0/airbyte-cdk/add-per-partition-with-global-fallba…

c1c2d3e

…ck' into tolik0/airbyte-cdk/perpartition-with-global-regression-tests

vercel bot deployed to Preview September 18, 2024 13:53 View deployment

codeflash-ai bot reviewed Sep 18, 2024

View reviewed changes

codeflash-ai bot mentioned this pull request Sep 18, 2024

⚡️ Speed up method PerPartitionCursor.set_initial_state by 6% in PR #45415 (tolik0/airbyte-cdk/perpartition-with-global-regression-tests) #45657

Closed

tolik0 added 11 commits September 19, 2024 13:39

Update PerPartitionCursor docs

96f1150

Simplified iterate_with_last_flag

3042d28

Merge branch 'master' into tolik0/airbyte-cdk/add-per-partition-with-…

8cfc0ba

…global-fallback

Fix formatting

bd507e0

Fix mypy error

e6976fc

Merge branch 'master' into tolik0/airbyte-cdk/add-per-partition-with-…

c998f5c

…global-fallback

Align with new SubstreamPartitionRouter

58c96f2

Update the state management

1fb01a9

Merge branch 'tolik0/airbyte-cdk/add-per-partition-with-global-fallba…

855a69b

…ck' into tolik0/airbyte-cdk/perpartition-with-global-regression-tests

Set DEFAULT_MAX_PARTITIONS_NUMBER for PerPartitionCursor to 5 for tes…

8ad8ffa

…ting

Update version

031c294

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Perpartition with global - Regression Tests #45415

Perpartition with global - Regression Tests #45415

tolik0 commented Sep 12, 2024 •

edited

Loading

vercel bot commented Sep 12, 2024 •

edited

Loading

codeflash-ai bot commented Sep 12, 2024

⚡️ Speed up method `GlobalSubstreamCursor.set_initial_state` by 65% in PR #45415 (`tolik0/airbyte-cdk/perpartition-with-global-regression-tests`) #45418

codeflash-ai bot commented Sep 17, 2024

⚡️ Speed up method `Timer.finish` by 40% in PR #45415 (`tolik0/airbyte-cdk/perpartition-with-global-regression-tests`) #45621

codeflash-ai bot commented Sep 17, 2024

⚡️ Speed up method `Timer.finish` by 37% in PR #45415 (`tolik0/airbyte-cdk/perpartition-with-global-regression-tests`) #45624

codeflash-ai bot Sep 18, 2024

codeflash-ai bot commented Sep 18, 2024

codeflash-ai bot commented Sep 18, 2024

⚡️ Speed up method `PerPartitionCursor.set_initial_state` by 6% in PR #45415 (`tolik0/airbyte-cdk/perpartition-with-global-regression-tests`) #45657

	return ((time.perf_counter_ns() - self._start) / 1e9).__ceil__()
	return (time.perf_counter_ns() - self._start + 999_999_999) // 1_000_000_000

Perpartition with global - Regression Tests #45415

Are you sure you want to change the base?

Perpartition with global - Regression Tests #45415

Conversation

tolik0 commented Sep 12, 2024 • edited Loading

What

vercel bot commented Sep 12, 2024 • edited Loading

codeflash-ai bot commented Sep 12, 2024

⚡️ Codeflash found optimizations for this PR

📄 GlobalSubstreamCursor.set_initial_state() in airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/global_substream_cursor.py

I created a new dependent PR with the suggested changes. Please review:

⚡️ Speed up method GlobalSubstreamCursor.set_initial_state by 65% in PR #45415 (tolik0/airbyte-cdk/perpartition-with-global-regression-tests) #45418

codeflash-ai bot commented Sep 17, 2024

⚡️ Codeflash found optimizations for this PR

📄 Timer.finish() in airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/global_substream_cursor.py

I created a new dependent PR with the suggested changes. Please review:

⚡️ Speed up method Timer.finish by 40% in PR #45415 (tolik0/airbyte-cdk/perpartition-with-global-regression-tests) #45621

codeflash-ai bot commented Sep 17, 2024

⚡️ Codeflash found optimizations for this PR

📄 Timer.finish() in airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/global_substream_cursor.py

I created a new dependent PR with the suggested changes. Please review:

⚡️ Speed up method Timer.finish by 37% in PR #45415 (tolik0/airbyte-cdk/perpartition-with-global-regression-tests) #45624

codeflash-ai bot Sep 18, 2024

Choose a reason for hiding this comment

codeflash-ai bot commented Sep 18, 2024

⚡️ Codeflash found optimizations for this PR

📄 Timer.finish() in airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/global_substream_cursor.py

Explanation and details

Correctness verification

🔘 (none found) − ⚙️ Existing Unit Tests

✅ 1009 Passed − 🌀 Generated Regression Tests

🔘 (none found) − ⏪ Replay Tests

codeflash-ai bot commented Sep 18, 2024

⚡️ Codeflash found optimizations for this PR

📄 PerPartitionCursor.set_initial_state() in airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/per_partition_cursor.py

I created a new dependent PR with the suggested changes. Please review:

⚡️ Speed up method PerPartitionCursor.set_initial_state by 6% in PR #45415 (tolik0/airbyte-cdk/perpartition-with-global-regression-tests) #45657

tolik0 commented Sep 12, 2024 •

edited

Loading

vercel bot commented Sep 12, 2024 •

edited

Loading

📄 `GlobalSubstreamCursor.set_initial_state()` in `airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/global_substream_cursor.py`

⚡️ Speed up method `GlobalSubstreamCursor.set_initial_state` by 65% in PR #45415 (`tolik0/airbyte-cdk/perpartition-with-global-regression-tests`) #45418

📄 `Timer.finish()` in `airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/global_substream_cursor.py`

⚡️ Speed up method `Timer.finish` by 40% in PR #45415 (`tolik0/airbyte-cdk/perpartition-with-global-regression-tests`) #45621

📄 `Timer.finish()` in `airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/global_substream_cursor.py`

⚡️ Speed up method `Timer.finish` by 37% in PR #45415 (`tolik0/airbyte-cdk/perpartition-with-global-regression-tests`) #45624

📄 `Timer.finish()` in `airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/global_substream_cursor.py`

📄 `PerPartitionCursor.set_initial_state()` in `airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/per_partition_cursor.py`

⚡️ Speed up method `PerPartitionCursor.set_initial_state` by 6% in PR #45415 (`tolik0/airbyte-cdk/perpartition-with-global-regression-tests`) #45657