Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-44923: [MATLAB] Add IPC RecordBatchStreamReader MATLAB class #45068

Open
wants to merge 20 commits into
base: main
Choose a base branch
from

Conversation

kevingurney
Copy link
Member

@kevingurney kevingurney commented Dec 18, 2024

Rationale for this change

To enable support for the IPC Streaming format in the MATLAB interface, we should add a RecordBatchStreamReader class.

This is a followup to #44922

What changes are included in this PR?

  1. Added a new arrow.io.ipc.RecordBatchStreamReader MATLAB class.

Are these changes tested?

Yes.

  1. Added new MATLAB test suite arrow/matlab/test/arrow/io/ipc/tRecordBatchStreamReader.m.

Are there any user-facing changes?

Yes.

  1. Users can now create arrow.io.ipc.RecordBatchStreamReader objects to read RecordBatch objects incrementally from an Arrow IPC Stream file.

Notes

  1. Thank you @sgilmore10 for your help with this pull request!

Copy link
Member

@sgilmore10 sgilmore10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thanks for working on this :)

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Dec 18, 2024
@github-actions github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Dec 18, 2024
@github-actions github-actions bot removed the awaiting changes Awaiting changes label Dec 18, 2024
@github-actions github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Dec 18, 2024
@github-actions github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Dec 18, 2024
Co-authored-by: Sarah Gilmore <74676073+sgilmore10@users.noreply.github.com>
@github-actions github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Dec 18, 2024
@github-actions github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Dec 18, 2024
@kevingurney
Copy link
Member Author

Thanks for the review @sgilmore10! Sorry for all of the typos!

@kevingurney kevingurney marked this pull request as ready for review December 18, 2024 20:35
@kevingurney kevingurney requested a review from kou as a code owner December 18, 2024 20:35
Comment on lines +46 to +52
function tf = hasnext(obj)
tf = obj.Proxy.hasNextRecordBatch();
end

function tf = done(obj)
tf = ~obj.Proxy.hasNextRecordBatch();
end
Copy link
Member

@kou kou Dec 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not familiar with MATLAB but does MATLAB require those APIs for iterator (or something)?

If we don't require this, it may be better that we don't provide them... If we don't have them, we can simplify our implementation.

Copy link
Member Author

@kevingurney kevingurney Dec 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good question.

There is no specific API for iteration in MATLAB (i.e. there is no for each style iteration). So, we were trying to emulate this behavior using a "while has more data"-style loop. There is precedent for this pattern of iteration being used elsewhere in MATLAB.

Does that answer your question?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Yes.

One more question. Is while record_batch = reader.read()-style loop used in MATLAB too? If the style is also used in MATLAB, we may want to recommend the style for the MATLAB bindings.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, no - MATLAB does not support variable assignment like while record_batch = reader.read() inside of a while loop expression, either. If that syntax was supported, I agree it would be preferable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see!

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Dec 19, 2024
Copy link
Member

@kou kou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@github-actions github-actions bot added awaiting merge Awaiting merge and removed awaiting changes Awaiting changes labels Dec 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants