Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FileSource.offline_write_batch doesn't work with S3 #3490

Closed
goodseog opened this issue Feb 12, 2023 · 1 comment
Closed

FileSource.offline_write_batch doesn't work with S3 #3490

goodseog opened this issue Feb 12, 2023 · 1 comment
Labels
kind/bug priority/p2 wontfix This will not be worked on

Comments

@goodseog
Copy link
Contributor

Expected Behavior

If FileSource.offline_write_batch, write correctly on S3 storage if use s3:// path.

Current Behavior

file_options = feature_view.batch_source.file_options
        filesystem, path = FileSource.create_filesystem_and_path(
            file_options.uri, file_options.s3_endpoint_override
        )

prev_table = pyarrow.parquet.read_table(path, memory_map=True)

When read prev_table, it doens't use "filesystem" kwargs

Steps to reproduce

Always happened

Specifications

  • Version: 0.29.0
  • Platform:
  • Subsystem:

Possible Solution

# default filesystem is None, so it works local & s3fs both
prev_table = pyarrow.parquet.read_table(path, filesystem=filesystem, memory_map=True)
@goodseog goodseog mentioned this issue Feb 12, 2023
Merged
goodseog added a commit to goodseog/feast that referenced this issue Feb 14, 2023
@goodseog
fix: Add filesystem kwargs when read prev_table on FileRetrievalJob (f…
6541206 
…east-dev#3490)

Signed-off-by: hyeongseok.yoo <goodseog.yoo@navercorp.com>
feast-ci-bot pushed a commit that referenced this issue Mar 2, 2023
@goodseog
fix: Add filesystem kwargs when read prev_table on FileRetrievalJob (… (
dca4745 
#3491)

fix: Add filesystem kwargs when read prev_table on FileRetrievalJob (#3490)

Signed-off-by: hyeongseok.yoo <goodseog.yoo@navercorp.com>
kevjumba pushed a commit that referenced this issue Mar 17, 2023
@feast-ci-bot
chore(release): release 0.30.0
5310280 
# [0.30.0](v0.29.0...v0.30.0) (2023-03-17)

### Bug Fixes

* Add description attribute to the Field.from_proto method ([#3469](#3469)) ([473f8d9](473f8d9))
* Add filesystem kwargs when read prev_table on FileRetrievalJob (… ([#3491](#3491)) ([dca4745](dca4745)), closes [#3490](#3490)
* Feature view `entities` from_proto type ([#3524](#3524)) ([57bbb61](57bbb61))
* Fix missing requests requirement after GCP requirement removed. Make BigQuerySource not require gcp extra ([2c85421](2c85421))
* Fix SQL Registry cache miss ([#3482](#3482)) ([3249b97](3249b97))
* Fixed path inside quickstart notebook ([#3456](#3456)) ([66edc32](66edc32))
* Improve BQ point-in-time joining scalability ([#3429](#3429)) ([ff66784](ff66784))
* Pin typeguard to 2.13.3 which is what we are currently using. ([#3542](#3542)) ([61f6fb0](61f6fb0))
* Protobuf lower bound to 3.20 to alert that Feast is incompatible with tensorflow ([#3476](#3476)) ([9ca59e3](9ca59e3))
* Spark kafka processor sorting ([#3479](#3479)) ([f2cbf43](f2cbf43))
* UI working behind base url ([#3514](#3514)) ([9a3fd98](9a3fd98))
* Update go dependencies ([#3512](#3512)) ([bada97c](bada97c))

### Features

* Add Rockset as an OnlineStore ([#3405](#3405)) ([fd91cda](fd91cda))
* Add Snowflake Registry ([#3363](#3363)) ([ec1e61d](ec1e61d))
* Adding query timeout to `to_df` and `to_arrow` retrieval methods ([#3505](#3505)) ([bab6644](bab6644))
* adds k8s config options to Bytewax materialization engine ([#3518](#3518)) ([1883f55](1883f55))
achals pushed a commit that referenced this issue Mar 24, 2023
@feast-ci-bot
chore(release): release 0.30.0
ebb36d5 
# [0.30.0](v0.29.0...v0.30.0) (2023-03-24)

### Bug Fixes

* Add description attribute to the Field.from_proto method ([#3469](#3469)) ([473f8d9](473f8d9))
* Add filesystem kwargs when read prev_table on FileRetrievalJob (… ([#3491](#3491)) ([dca4745](dca4745)), closes [#3490](#3490)
* Bytewax image pull secret config ([#3547](#3547)) ([d2d13b1](d2d13b1))
* Clean up Rockset Online Store for use ([#3549](#3549)) ([a76c6d0](a76c6d0))
* Feature view `entities` from_proto type ([#3524](#3524)) ([57bbb61](57bbb61))
* Fix missing requests requirement after GCP requirement removed. Make BigQuerySource not require gcp extra ([2c85421](2c85421))
* Fix SQL Registry cache miss ([#3482](#3482)) ([3249b97](3249b97))
* Fixed path inside quickstart notebook ([#3456](#3456)) ([66edc32](66edc32))
* Improve BQ point-in-time joining scalability ([#3429](#3429)) ([ff66784](ff66784))
* Pin typeguard to 2.13.3 which is what we are currently using. ([#3542](#3542)) ([61f6fb0](61f6fb0))
* Protobuf lower bound to 3.20 to alert that Feast is incompatible with tensorflow ([#3476](#3476)) ([9ca59e3](9ca59e3))
* Spark kafka processor sorting ([#3479](#3479)) ([f2cbf43](f2cbf43))
* UI working behind base url ([#3514](#3514)) ([9a3fd98](9a3fd98))
* Update go dependencies ([#3512](#3512)) ([bada97c](bada97c))

### Features

* Add Rockset as an OnlineStore ([#3405](#3405)) ([fd91cda](fd91cda))
* Add Snowflake Registry ([#3363](#3363)) ([ec1e61d](ec1e61d))
* Added SnowflakeConnection caching ([#3531](#3531)) ([f9f8df2](f9f8df2))
* Adding query timeout to `to_df` and `to_arrow` retrieval methods ([#3505](#3505)) ([bab6644](bab6644))
* adds k8s config options to Bytewax materialization engine ([#3518](#3518)) ([1883f55](1883f55))
@stale
Copy link

stale bot commented Aug 10, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix This will not be worked on label Aug 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug priority/p2 wontfix This will not be worked on
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@goodseog