Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pyspark job for feature batch retrieval #1021

Merged
merged 6 commits into from
Oct 7, 2020
Merged

Pyspark job for feature batch retrieval #1021

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
afbf7cf
Pyspark job for feature batch retrieval
khorshuheng Sep 30, 2020
675a952
Add pyspark to ci requirements
khorshuheng Sep 30, 2020
dbe6192
Additional documentation and col mapping
khorshuheng Oct 1, 2020
30be3a5
Add Schema validation
khorshuheng Oct 2, 2020
7522b88
Improve test case and documentation
khorshuheng Oct 5, 2020
1a36737
Change max age to integer, filter source feature tables, tests for la…
khorshuheng Oct 6, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file added sdk/python/feast/pyspark/__init__.py
View file
Open in desktop
Empty file.
421 changes: 421 additions & 0 deletions sdk/python/feast/pyspark/historical_feature_retrieval_job.py
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions sdk/python/requirements-ci.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ pytest-lazy-fixture==0.6.3
pytest-mock
pytest-timeout
pytest-ordering==0.6.*
pyspark==3.*
pandas~=1.0.0
mock==2.0.0
pandavro==1.5.*
Expand Down
2 changes: 2 additions & 0 deletions sdk/python/requirements-dev.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,5 @@ flake8
black==19.10b0
boto3
moto
pyspark==3.*
pyspark-stubs==3.*
6 changes: 6 additions & 0 deletions sdk/python/tests/data/bookings.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
driver_id,event_timestamp,created_timestamp,completed_bookings
8001,2020-08-31T00:00:00.000,2020-08-31T00:00:00.000,200
8001,2020-09-01T00:00:00.000,2020-09-01T00:00:00.000,300
8002,2020-09-01T00:00:00.000,2020-09-01T00:00:00.000,600
8002,2020-09-01T00:00:00.000,2020-09-02T00:00:00.000,500
8003,2020-09-01T00:00:00.000,2020-09-02T00:00:00.000,700
6 changes: 6 additions & 0 deletions sdk/python/tests/data/column_mapping_test_entity.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
id,event_timestamp
1001,2020-09-02T00:00:00.000
Copy link
Member

@woop woop Oct 5, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we please add a few more rows, like 5-10. One row is asking for trouble. Same for rest

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even better to generate thousands to test how multiple partitions can affect result

Copy link
Member

@woop woop Oct 5, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, using code to generate data isn't a bad idea, as long as we avoid too much rand() without a seed.

1001,2020-09-03T00:00:00.000
2001,2020-09-04T00:00:00.000
2001,2020-09-04T00:00:00.000
3001,2020-09-04T00:00:00.000
6 changes: 6 additions & 0 deletions sdk/python/tests/data/column_mapping_test_feature.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
customer_id,total_bookings,datetime,created_datetime
1001,200,2020-09-02T00:00:00.000,2020-09-02T00:00:00.000
1001,400,2020-09-04T00:00:00.000,2020-09-02T00:00:00.000
2001,500,2020-09-03T00:00:00.000,2020-09-01T00:00:00.000
2001,600,2020-09-03T00:00:00.000,2020-09-02T00:00:00.000
3001,700,2020-09-03T00:00:00.000,2020-09-03T00:00:00.000
6 changes: 6 additions & 0 deletions sdk/python/tests/data/customer_driver_pairs.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
customer_id,driver_id,event_timestamp
1001,8001,2020-09-02T00:00:00.000
1001,8002,2020-09-02T00:00:00.000
1001,8002,2020-09-03T00:00:00.000
2001,8002,2020-09-03T00:00:00.000
2001,8002,2020-09-04T00:00:00.000
6 changes: 6 additions & 0 deletions sdk/python/tests/data/transactions.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
customer_id,event_timestamp,created_timestamp,daily_transactions
1001,2020-08-31T00:00:00.000,2020-09-01T00:00:00.000,50.0
1001,2020-09-01T00:00:00.000,2020-09-01T00:00:00.000,100.0
2001,2020-09-01T00:00:00.000,2020-08-31T00:00:00.000,80.0
2001,2020-09-01T00:00:00.000,2020-09-01T00:00:00.000,200.0
3001,2020-09-01T00:00:00.000,2020-09-01T00:00:00.000,300.0
Loading