Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add Postgres Online Store Async Feature Retrieval #2

Closed
wants to merge 28 commits into from
Closed

feat: Add Postgres Online Store Async Feature Retrieval #2

wants to merge 28 commits into from

Commits on Jun 21, 2024

  1. Makefile: Formatting 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    c8fae69 View commit details
    Browse the repository at this point in the history

  2. Makefile: Exclude Snowflake tests for postgres offline store tests 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    161fd9d View commit details
    Browse the repository at this point in the history

  3. Requirements: Update 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    df0f440 View commit details
    Browse the repository at this point in the history

  4. Bootstrap: Use conninfo 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    c200254 View commit details
    Browse the repository at this point in the history

  5. Tests: Make connection string compatible with psycopg3 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    8618579 View commit details
    Browse the repository at this point in the history

  6. Tests: Test connection type pool and singleton 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    86c22e5 View commit details
    Browse the repository at this point in the history

  7. Global: Replace conn.set_session() calls to be psycopg3 compatible 

    Set connection read only

Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    8fb9224 View commit details
    Browse the repository at this point in the history

  8. Offline: Use psycopg3 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    0393d6d View commit details
    Browse the repository at this point in the history

  9. Online: Use psycopg3 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    2a7ad9a View commit details
    Browse the repository at this point in the history

  10. Online: Restructure online_write_batch 

    Addition

Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    995fc08 View commit details
    Browse the repository at this point in the history

  11. Online: Use correct placeholder 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    6f1a436 View commit details
    Browse the repository at this point in the history

  12. Online: Handle bytes properly in online_read() 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    bfb5e60 View commit details
    Browse the repository at this point in the history

  13. Online: Whitespace 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    46f7c96 View commit details
    Browse the repository at this point in the history

  14. Online: Open ConnectionPool 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    3d3b88c View commit details
    Browse the repository at this point in the history

  15. Online: Add typehint 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    6625a2c View commit details
    Browse the repository at this point in the history

  16. Utils: Use psycopg3 

    Use new ConnectionPool

Pass kwargs as named argument

Use executemany over execute_values

Remove not-required open argument in psycopg.connect

Improve

Use SpooledTemporaryFile

Use max_size and add docstring

Properly write with StringIO

Utils: Use SpooledTemporaryFile over StringIO object

Add replace

Fix df_to_postgres_table

Remove import

Utils

Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    d0f77ca View commit details
    Browse the repository at this point in the history

  17. Lint: Raise exceptions if cursor returned no columns or rows 

    Add log statement

Lint: Fix _to_arrow_internal

Lint: Fix _get_entity_df_event_timestamp_range

Update exception

Use ZeroColumnQueryResult

Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    b7de4b2 View commit details
    Browse the repository at this point in the history

  18. Add comment on +psycopg string 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    60f5a23 View commit details
    Browse the repository at this point in the history

  19. Docs: Remove mention of psycopg2 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    f9fa158 View commit details
    Browse the repository at this point in the history

  20. Lint: Fix 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    38a376f View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. Default to postgresql+psycopg and log warning 

    Update warning

Fix

Format warning

Add typehints

Use better variable name

Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    b94fb44 View commit details
    Browse the repository at this point in the history

  2. Tests: Enable async feature retrieval integration test for Postgres o… 

    …nline store

Remove ConnectionType.pool_async

Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    2641b81 View commit details
    Browse the repository at this point in the history

  3. Utils: Add _get_connection_pool_async 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    c9c2428 View commit details
    Browse the repository at this point in the history

  4. Online: Create helper functions for online_read, and re-use them also… 

    … in online_read_async

Apply fixes

Online: Create helper functions for online_read, and re-use them also in online_read_async

Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    b3ff9ec View commit details
    Browse the repository at this point in the history

  5. Online: Add _get_conn_async 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    664f12b View commit details
    Browse the repository at this point in the history

  6. Online & Utils: Use ConnectionType to decide whether to use async pool 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    de52480 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. Fix lint 

    Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>
    @job-almekinders
    job-almekinders committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    fb0d977 View commit details
    Browse the repository at this point in the history

  2. Format SQL query

    @job-almekinders
    job-almekinders committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    16aa614 View commit details
    Browse the repository at this point in the history