Add support for file paths for providing entity rows during batch retrieval #375
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
voonhous
requested review from
davidheryanto,
pradithya,
thirteen37,
tims,
woop and
zhilingc
as code owners
|
/retest |
|
/lgtm |
woop
previously approved these changes
Dec 22, 2019
|
/retest test-end-to-end |
|
/retest |
voonhous
force-pushed
the
batch-retrieval-file-path-dev
branch
from
c97ffac
to
7a29d94
Compare
|
/lgtm |
|
/retest |
|
/approved |
woop
approved these changes
Dec 22, 2019
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: voonhous, woop The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR mirrors PR #365. The branch was deleted as i did not do a force-push but a delete-push to remote which resulted in PR#365 getting closed.
Description
Users should be able to provide large amounts of entity rows when retrieving batch features, but currently they are blocked by memory limits of pandas DataFrames.
Right now, for batch retrieval we already support Avro files as the format for sending entity rows, however this is only available on the Feast Serving API. The Python SDK hides this detail by doing
Entity_rows Pandas DF → .avro (local) → .avro (gcs) → BQ
This pull requests adds the ability for users to provide:
A pandas DataFrame with the "datetime" column
A local Avro file with the "event_timestamp" column
A gcs Avro file
A gcs wildcard path.
Examples:
entity_rows = [Pandas Dataframe]
entity_rows = subfolder/entities.avro
entity_rows = /data/subfolder/entities.avro
entity_rows = gs://food-recsys/folder/customer_entity_rows.avro
entity_rows = gs://food-recsys/folder/customer_entity_rows_*.avro
While datetime and event_timestamp are used interchangeably, there needs to be standardization within the SDK on which to use.
As of now:
datetime is enforced in Pandas DataFrame.
event_timestamp is enforced in local Avro file
No enforcement in files living in GCS. No validation will be done on GCS file paths.
Two additional tests have been ended to
bq_batch_retrieval.py. The two teststest_get_batch_features_with_fileandtest_get_batch_features_with_gs_pathwill run an end to end test respectively for all changes mentioned above.