Skip to content

[data] dedupe schemas #55854

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
alexeykudinkin merged 15 commits into from
Aug 27, 2025
Merged

[data] dedupe schemas #55854

alexeykudinkin merged 15 commits into from
Aug 27, 2025

Conversation

@iamjustinhsu
Copy link
Contributor

@iamjustinhsu iamjustinhsu commented Aug 22, 2025 •
edited
Loading

Why are these changes needed?

results: TBD

  • 2180 columns, 1000 schemas deduping is 1 second

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@iamjustinhsu
[data] dedupe schemas
f5be8bc 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@iamjustinhsu
comment
0904bd6 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
alexeykudinkin
alexeykudinkin reviewed Aug 22, 2025
View reviewed changes
@iamjustinhsu
hash ArrowPythonObjecType
bbd1243 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@iamjustinhsu
done
635647a 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@iamjustinhsu
done
5d05869 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@iamjustinhsu iamjustinhsu added the go add ONLY when ready to merge, run all tests label Aug 25, 2025
@iamjustinhsu iamjustinhsu marked this pull request as ready for review August 25, 2025 22:24
@iamjustinhsu iamjustinhsu requested review from a team as code owners August 25, 2025 22:24
@iamjustinhsu
lint
62ac12f 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@ray-gardener ray-gardener bot added the data Ray Data-related issues label Aug 26, 2025
@iamjustinhsu
generalize
2bc7092 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
iamjustinhsu
iamjustinhsu commented Aug 26, 2025
View reviewed changes
alexeykudinkin
alexeykudinkin reviewed Aug 26, 2025
View reviewed changes
python/ray/data/_internal/arrow_ops/transform_pyarrow.py Outdated
Comment on lines 186 to 188
except Exception:
# Unsure if there are cases where schemas are NOT hashable
pass
Copy link
Contributor

@alexeykudinkin alexeykudinkin Aug 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't swallow exceptions

@iamjustinhsu
add warning + remove_metadata in RefBundle check
b84121d 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@iamjustinhsu
remove test
71d6eaa 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
alexeykudinkin
alexeykudinkin approved these changes Aug 27, 2025
View reviewed changes
@iamjustinhsu
print exception too
3a2b67c 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
alexeykudinkin
alexeykudinkin reviewed Aug 27, 2025
View reviewed changes
python/ray/data/_internal/execution/interfaces/ref_bundle.py
Comment on lines +68 to +69
if isinstance(self.schema, pa.lib.Schema):
self.schema = self.schema.remove_metadata()
Copy link
Contributor

@alexeykudinkin alexeykudinkin Aug 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iamjustinhsu on a second thought: why don't we do this when we convert from Pandas to Pyarrow, since we only do that in normalize_block_types essentially

Copy link
Contributor Author

@iamjustinhsu iamjustinhsu Aug 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i've seen it occur elsewhere as well: #55854 (comment). With the patch release not sure if would should try to find every area where this occurs

Copy link
Contributor

@alexeykudinkin alexeykudinkin Aug 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack. Add a comment and elaborate on why we're doing this

alexeykudinkin and others added 5 commits August 27, 2025 12:49
@alexeykudinkin
Tidying up
f42fb25 
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
@iamjustinhsu
lint
de0db90 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@iamjustinhsu
comment
5debf3b 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@iamjustinhsu
fix
1f7e0f4 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@iamjustinhsu
name
9439b0c 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@alexeykudinkin alexeykudinkin enabled auto-merge (squash) August 27, 2025 21:24
dayshah
dayshah approved these changes Aug 27, 2025
View reviewed changes
Copy link
Contributor

@dayshah dayshah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you update so core isn't a code owner on any of this @alexeykudinkin
trusting your approval here lol

@alexeykudinkin alexeykudinkin merged commit 4a6c264 into ray-project:master Aug 27, 2025
6 checks passed
omatthew98 pushed a commit to omatthew98/ray that referenced this pull request Aug 28, 2025
@iamjustinhsu @alexeykudinkin @omatthew98
[data] dedupe schemas (ray-project#55854)
fd26490 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
@omatthew98 omatthew98 mentioned this pull request Aug 28, 2025
Merged
8 tasks
@iamjustinhsu iamjustinhsu mentioned this pull request Aug 28, 2025
Closed
8 tasks
omatthew98 pushed a commit to omatthew98/ray that referenced this pull request Aug 28, 2025
@iamjustinhsu @alexeykudinkin @omatthew98
[data] dedupe schemas (ray-project#55854)
81ff179 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Signed-off-by: Matthew Owen <mowen@anyscale.com>
aslonnie pushed a commit that referenced this pull request Aug 28, 2025
@omatthew98 @iamjustinhsu @alexeykudinkin
[data] Cherry pick data fixes for 2.49.1 (#56058)
c057f1e 
Cherry pick two fixes for ray data (from
#55854 and
#55926).

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Signed-off-by: Matthew Owen <mowen@anyscale.com>
Co-authored-by: iamjustinhsu <140442892+iamjustinhsu@users.noreply.github.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
@iamjustinhsu iamjustinhsu deleted the jhsu/dedupe-schemas branch August 29, 2025 04:55
tohtana pushed a commit to tohtana/ray that referenced this pull request Aug 29, 2025
@iamjustinhsu @alexeykudinkin @tohtana
[data] dedupe schemas (ray-project#55854)
b98f3f7 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
tohtana pushed a commit to tohtana/ray that referenced this pull request Aug 29, 2025
@iamjustinhsu @alexeykudinkin @tohtana
[data] dedupe schemas (ray-project#55854)
2e6f894 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
gangsf pushed a commit to gangsf/ray that referenced this pull request Sep 2, 2025
@iamjustinhsu @alexeykudinkin
[data] dedupe schemas (ray-project#55854)
b8e9fd6 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Signed-off-by: Gang Zhao <gang@gang-JQ62HD2C37.local>
sampan-s-nayak pushed a commit to sampan-s-nayak/ray that referenced this pull request Sep 8, 2025
@iamjustinhsu @alexeykudinkin
[data] dedupe schemas (ray-project#55854)
5c80019 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Signed-off-by: sampan <sampan@anyscale.com>
jugalshah291 pushed a commit to jugalshah291/ray_fork that referenced this pull request Sep 11, 2025
@iamjustinhsu @alexeykudinkin @jugalshah291
[data] dedupe schemas (ray-project#55854)
3ac38da 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Signed-off-by: jugalshah291 <shah.jugal291@gmail.com>
dstrodtman pushed a commit to dstrodtman/ray that referenced this pull request Oct 6, 2025
@iamjustinhsu @alexeykudinkin @dstrodtman
[data] dedupe schemas (ray-project#55854)
271b116 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Signed-off-by: Douglas Strodtman <douglas@anyscale.com>
landscapepainter pushed a commit to landscapepainter/ray that referenced this pull request Nov 17, 2025
@iamjustinhsu @alexeykudinkin @landscapepainter
[data] dedupe schemas (ray-project#55854)
831a21a 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alexeykudinkin alexeykudinkin alexeykudinkin approved these changes

@dayshah dayshah dayshah approved these changes

Assignees

No one assigned

Labels

data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@iamjustinhsu @alexeykudinkin @dayshah