Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[data][api] implement HudiDataSource #46273

Merged
merged 6 commits into from
Nov 19, 2024
Merged

[data][api] implement HudiDataSource #46273

merged 6 commits into from
Nov 19, 2024
+282 −1

Conversation

xushiyan
Copy link
Contributor

@xushiyan xushiyan commented Jun 26, 2024 •
edited
Loading

Why are these changes needed?

Support read from Hudi table into Ray dataset.

Related issue number

Closes #46272

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@xushiyan xushiyan force-pushed the add-hudi-datasource branch 5 times, most recently from 7bc3894 to 97f9de1 Compare June 28, 2024 18:11
@xushiyan xushiyan marked this pull request as ready for review June 28, 2024 18:20
@xushiyan xushiyan force-pushed the add-hudi-datasource branch from 97f9de1 to d4e8af6 Compare June 28, 2024 18:58
@anyscalesam anyscalesam added data Ray Data-related issues @external-author-action-required Alternate tag for PRs where the author doesn't have labeling permission. triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Jun 29, 2024
@xushiyan xushiyan force-pushed the add-hudi-datasource branch 3 times, most recently from e2f6704 to 557b887 Compare July 14, 2024 18:17
omatthew98
omatthew98 reviewed Jul 16, 2024
View reviewed changes
Copy link
Contributor

@omatthew98 omatthew98 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took a first pass and left some mostly nit comments. Overall looks good, let us know / re-request a review when it is ready for a re-review!

python/ray/data/_internal/datasource/hudi_datasource.py Outdated Show resolved Hide resolved
python/ray/data/_internal/datasource/hudi_datasource.py Outdated Show resolved Hide resolved
python/ray/data/_internal/datasource/hudi_datasource.py
return read_tasks

def estimate_inmemory_data_size(self) -> Optional[int]:
return None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any estimate that can be provided / returned here? Perhaps using the size_bytes from above? Maybe could cache that similar to what is done here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree to provide estimates here. However due to current impl, loading this info during init using HudiTable is not a lightweight operation, plus the size bytes are storage size without some translation to in-memory size. i've added a todo here to support this info through HudiTable API.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds reasonable!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xushiyan we can use Parquet metadata fetcher here (fine if we do as follow-up)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alexeykudinkin yes, will do follow up to enhance this part.

python/ray/data/tests/test_hudi.py Outdated Show resolved Hide resolved
@xushiyan xushiyan force-pushed the add-hudi-datasource branch from 557b887 to 177caab Compare July 26, 2024 21:00
@xushiyan xushiyan force-pushed the add-hudi-datasource branch 6 times, most recently from fe1c93a to b76c200 Compare August 19, 2024 05:47
@xushiyan xushiyan force-pushed the add-hudi-datasource branch 2 times, most recently from 0d5fb56 to f63fd83 Compare October 20, 2024 05:37
@xushiyan xushiyan force-pushed the add-hudi-datasource branch from c3dcb70 to fa3cc78 Compare November 6, 2024 00:07
@xushiyan xushiyan requested a review from srinathk10 as a code owner November 6, 2024 00:07
@xushiyan xushiyan force-pushed the add-hudi-datasource branch 3 times, most recently from 1a62d74 to cfb8406 Compare November 6, 2024 08:44
alexeykudinkin
alexeykudinkin reviewed Nov 16, 2024
View reviewed changes
Copy link
Contributor

@alexeykudinkin alexeykudinkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xushiyan LGTM, mostly minor comments

python/ray/data/_internal/datasource/hudi_datasource.py Outdated Show resolved Hide resolved
python/ray/data/_internal/datasource/hudi_datasource.py Outdated Show resolved Hide resolved
python/ray/data/_internal/datasource/hudi_datasource.py Outdated Show resolved Hide resolved
python/ray/data/_internal/datasource/hudi_datasource.py Outdated Show resolved Hide resolved
python/ray/data/_internal/datasource/hudi_datasource.py Outdated Show resolved Hide resolved
python/ray/data/_internal/datasource/hudi_datasource.py
return read_tasks

def estimate_inmemory_data_size(self) -> Optional[int]:
return None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xushiyan we can use Parquet metadata fetcher here (fine if we do as follow-up)

python/ray/data/read_api.py Outdated Show resolved Hide resolved
python/ray/data/tests/test_hudi.py Outdated Show resolved Hide resolved
python/requirements/ml/data-test-requirements.txt Show resolved Hide resolved
python/ray/data/tests/test_hudi.py Show resolved Hide resolved
@xushiyan xushiyan force-pushed the add-hudi-datasource branch 3 times, most recently from 83d856b to 9ffd524 Compare November 18, 2024 16:52
alexeykudinkin
alexeykudinkin approved these changes Nov 18, 2024
View reviewed changes
Copy link
Contributor

@alexeykudinkin alexeykudinkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xushiyan LGTM

python/ray/data/tests/test_hudi.py Outdated Show resolved Hide resolved
python/ray/data/tests/test_hudi.py Outdated Show resolved Hide resolved
python/ray/data/tests/test_hudi.py Outdated Show resolved Hide resolved
@xushiyan
[data][api] implement HudiDataSource
19d1df1 
@MicroCheck //python:ray/data/tests/test_hudi

Signed-off-by: Shiyan Xu <2701446+xushiyan@users.noreply.github.com>
@xushiyan
update hudi version
ea2fe5d 
Signed-off-by: Shiyan Xu <2701446+xushiyan@users.noreply.github.com>
@xushiyan
fix import
6d50a84 
Signed-off-by: Shiyan Xu <2701446+xushiyan@users.noreply.github.com>
@xushiyan
update version check
bde3179 
Signed-off-by: Shiyan Xu <2701446+xushiyan@users.noreply.github.com>
@xushiyan
address comments
656d10a 
Signed-off-by: Shiyan Xu <2701446+xushiyan@users.noreply.github.com>
@xushiyan
address comments rnd2
d723528 
Signed-off-by: Shiyan Xu <2701446+xushiyan@users.noreply.github.com>
@xushiyan xushiyan force-pushed the add-hudi-datasource branch from 9ffd524 to d723528 Compare November 18, 2024 22:57
@bveeramani bveeramani enabled auto-merge (squash) November 19, 2024 00:42
@bveeramani bveeramani merged commit 36682c1 into ray-project:master Nov 19, 2024
6 checks passed
@xushiyan xushiyan deleted the add-hudi-datasource branch November 19, 2024 03:08
dentiny pushed a commit to dentiny/ray that referenced this pull request Dec 7, 2024
@xushiyan @dentiny
[data][api] implement HudiDataSource (ray-project#46273)
6f9e769 
Support read from Hudi table into Ray dataset.

---------

Signed-off-by: Shiyan Xu <2701446+xushiyan@users.noreply.github.com>
Signed-off-by: hjiang <dentinyhao@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@scottjlee scottjlee scottjlee left review comments

@omatthew98 omatthew98 omatthew98 approved these changes

@alexeykudinkin alexeykudinkin alexeykudinkin approved these changes

@ericl ericl Awaiting requested review from ericl

@scv119 scv119 Awaiting requested review from scv119

@c21 c21 Awaiting requested review from c21

@amogkam amogkam Awaiting requested review from amogkam

@bveeramani bveeramani Awaiting requested review from bveeramani

@raulchen raulchen Awaiting requested review from raulchen

@stephanie-wang stephanie-wang Awaiting requested review from stephanie-wang

@srinathk10 srinathk10 Awaiting requested review from srinathk10

Assignees

@scottjlee scottjlee

@omatthew98 omatthew98

Labels
data Ray Data-related issues @external-author-action-required Alternate tag for PRs where the author doesn't have labeling permission. go add ONLY when ready to merge, run all tests triage Needs triage (eg: priority, bug/not-bug, and owning component)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Data] Support read Hudi table as a DataSource
7 participants
@xushiyan @aslonnie @alexeykudinkin @scottjlee @omatthew98 @bveeramani @anyscalesam