-
Notifications
You must be signed in to change notification settings - Fork 7
Fix that field nullability affects write #24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
} | ||
|
||
public void write(byte[] bytes) throws Exception { | ||
public void write(byte[] bytes, boolean needCheckSchema) throws Exception { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
check schema always?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
record_batch in #write_pandas is converted from table schema, so it is unnecessary to check it.
return true; | ||
} | ||
|
||
private boolean checkField(Field expected, Field actual) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
checkFieldIgnoreNullability
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
(cherry picked from commit 00af9ad)
Purpose
The case is:
pk
is non-null from Paimon;pk not null
strictly , sowith pa.RecordBatchStreamWriter(stream, self._arrow_schema) as writer
will throw error.Fix:
pa.RecordBatchStreamWriter
use data's schema, and let java codes to check field (ignore nullability)Tests
API and Format
Documentation