Fix materialize bug with RedisCluster #2311

kevjumba · 2022-02-15T17:53:22Z

Signed-off-by: Kevin Zhang kzhang@tecton.ai

What this PR does / why we need it:
Broken materialization script for feast 0.18.0. Refer to issue #2297.
Issue was that there are 2 versions of the python redis cluster package redis-py-cluster and redis-py. redis-py only recently added a new version of RedisCluster and our api was using the redis-py-cluster api that passes dictionary of host:port pairs while redis-py implementation requires ClusterNodes instead.

Which issue(s) this PR fixes:

Fixes #2297

Does this PR introduce a user-facing change?:

NONE

Makes changes to integration github workflow that cannot be tested accurately since github runs the current PR against the current github workflow. Attached is proof that the new integration tests with the new workflow:

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

codecov-commenter · 2022-02-15T17:55:24Z

Codecov Report

Merging #2311 (1a4a155) into master (a7d4cb7) will decrease coverage by 27.61%.
The diff coverage is 12.50%.

@@             Coverage Diff             @@
##           master    #2311       +/-   ##
===========================================
- Coverage   86.11%   58.50%   -27.62%     
===========================================
  Files         116      116               
  Lines        9834     9933       +99     
===========================================
- Hits         8469     5811     -2658     
- Misses       1365     4122     +2757

Flag	Coverage Δ
integrationtests	`?`
unittests	`58.50% <12.50%> (-0.06%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../integration/online_store/test_universal_online.py	`14.28% <5.26%> (-82.91%)`	⬇️
sdk/python/tests/conftest.py	`56.48% <28.57%> (-35.60%)`	⬇️
sdk/python/feast/infra/online_stores/redis.py	`26.45% <50.00%> (-68.39%)`	⬇️
...ts/integration/feature_repos/repo_configuration.py	`44.95% <100.00%> (-49.50%)`	⬇️
.../integration/online_store/test_online_retrieval.py	`17.39% <0.00%> (-82.61%)`	⬇️
sdk/python/tests/utils/online_read_write_test.py	`20.68% <0.00%> (-79.32%)`	⬇️
...fline_store/test_universal_historical_retrieval.py	`23.50% <0.00%> (-76.50%)`	⬇️
...gration/registration/test_feature_service_apply.py	`31.25% <0.00%> (-68.75%)`	⬇️
... and 66 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a7d4cb7...1a4a155. Read the comment docs.

sdk/python/feast/infra/online_stores/redis.py

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

…positives Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

felixwang9817

/lgtm

kevjumba · 2022-02-17T01:25:01Z

@adchia PTAL!

adchia

/lgtm

feast-ci-bot · 2022-02-17T17:57:23Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adchia, felixwang9817, kevjumba

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [adchia,felixwang9817]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…ents * upstream/master: (103 commits) refactor: Move create cluster script ci: Add validation for upper case PR titles Switch order of views to keep/delete to fix apply logging ci: Fix broken Prow plugin which ignores config.yaml ci: Add PR linting to ensure conventional commits are followed for PR titles Add owner field to FeatureService (feast-dev#2321) Disable Redis cluster tests (feast-dev#2327) Use PR title for commit message Add redis cluster initialziation to master github workflow (feast-dev#2317) Fix materialize bug with RedisCluster (feast-dev#2311) Add support for Dask during historical retrieval (feast-dev#1954) downgrade datatypes to 8 Update pom.xml to 0.18.2 SNAPSHOT Update changelog for 0.18.1 and update pom.xml Update helm chart version Don't require `snowflake` to always be installed (feast-dev#2309) Fixing the Java helm charts and adding a demo tutorial on how to use them (feast-dev#2298) Include infra objects in registry dump and fix Infra's from_proto (feast-dev#2295) Remove old flag warning with the python feature server (feast-dev#2300) podAnnotations Values in the feature-server chart (feast-dev#2304) ...

* Fix materialization bug Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * oops removed something when i was copy pasting Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Test github workflow Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * transfer to unit_tests Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Brute force implementation Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Brute force implementation add gcc Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Brute force implementation add gcc Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Brute force implementation add gcc Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Brute force implementation add gcc Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Continue fixing... Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Remove gcc setup Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add integration test Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * fix error Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * fix error Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add setup Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * temp fix to get integration tests to work Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * temp fix to get integration tests to work Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * temp fix to get integration tests to work Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * temp fix to get integration tests to work Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix integration even more Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix integration even more Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * only run one test Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Do some more integration testing Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Do some more integration testing by adding bug to make sure no false positives Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Integration testing works Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean upu code Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add redis cluster script for starting a redis cluster Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Reset integration yml file Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix how to guide lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * add fixtures and remove excess code Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix materialization bug

4134113

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

kevjumba requested a review from a team as a code owner February 15, 2022 17:53

kevjumba requested review from pyalex and removed request for a team February 15, 2022 17:53

feast-ci-bot added release-note-none needs-kind size/XS labels Feb 15, 2022

kevjumba added kind/bug and removed size/XS labels Feb 15, 2022

feast-ci-bot removed the needs-kind label Feb 15, 2022

github-actions bot assigned felixwang9817 Feb 15, 2022

kevjumba requested review from felixwang9817 and adchia February 15, 2022 17:56

felixwang9817 added ok-to-test size/S size/XS and removed size/S labels Feb 15, 2022

felixwang9817 requested changes Feb 15, 2022

View reviewed changes

sdk/python/feast/infra/online_stores/redis.py Show resolved Hide resolved

kevjumba added 3 commits February 15, 2022 10:04

oops removed something when i was copy pasting

926260b

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Test github workflow

b4e9ac0

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

lint

1281b2d

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

feast-ci-bot added size/L and removed size/XS labels Feb 16, 2022

kevjumba added 6 commits February 15, 2022 16:23

transfer to unit_tests

b404f4f

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Brute force implementation

f8dab19

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Brute force implementation add gcc

c05d2b2

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Brute force implementation add gcc

e53c75b

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Brute force implementation add gcc

eafc27d

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Brute force implementation add gcc

bbbb325

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

kevjumba added 15 commits February 16, 2022 11:53

Fix integration even more

c32ed21

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix integration even more

68c9b61

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix lint

caeb757

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

only run one test

8b9f2f1

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Do some more integration testing

1f060e0

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Do some more integration testing by adding bug to make sure no false …

931dc23

…positives Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Integration testing works

41f7c75

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Clean upu code

174af8e

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Add redis cluster script for starting a redis cluster

d818c0a

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Reset integration yml file

b2903f8

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

lint

0b5dabd

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Clean up

1a4a155

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix how to guide lint

52f239d

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

add fixtures and remove excess code

a21e13f

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

lint

9c59fcc

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

felixwang9817 changed the title ~~[WIP] Fix #2297 Materialize bug with RedisCluster~~ Fix materialize bug with RedisCluster Feb 17, 2022

feast-ci-bot removed the do-not-merge/work-in-progress label Feb 17, 2022

felixwang9817 approved these changes Feb 17, 2022

View reviewed changes

feast-ci-bot added lgtm approved labels Feb 17, 2022

kevjumba assigned adchia and unassigned felixwang9817 Feb 17, 2022

adchia approved these changes Feb 17, 2022

View reviewed changes

adchia merged commit 7fa9fe3 into feast-dev:master Feb 17, 2022

This was referenced Feb 19, 2022

Disable Redis cluster tests #2327

Merged

Redis Cluster implementation is broken #2328

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix materialize bug with RedisCluster #2311

Fix materialize bug with RedisCluster #2311

kevjumba commented Feb 15, 2022 •

edited

Loading

codecov-commenter commented Feb 15, 2022 •

edited

Loading

felixwang9817 left a comment

kevjumba commented Feb 17, 2022

adchia left a comment

feast-ci-bot commented Feb 17, 2022

Fix materialize bug with RedisCluster #2311

Fix materialize bug with RedisCluster #2311

Conversation

kevjumba commented Feb 15, 2022 • edited Loading

codecov-commenter commented Feb 15, 2022 • edited Loading

Codecov Report

felixwang9817 left a comment

Choose a reason for hiding this comment

kevjumba commented Feb 17, 2022

adchia left a comment

Choose a reason for hiding this comment

feast-ci-bot commented Feb 17, 2022

kevjumba commented Feb 15, 2022 •

edited

Loading

codecov-commenter commented Feb 15, 2022 •

edited

Loading