Allow any column name in property Parquet files #634
Conversation
| std::shared_ptr<arrow::Schema> schema = out->schema(); | ||
| if (schema->num_fields() != 1) { | ||
| return KATANA_ERROR( | ||
| tsuba::ErrorCode::InvalidArgument, "expected 1 field found {} instead", |
There was a problem hiding this comment.
RenameColumns() performs this same check, so we can just rely on that.
| return KATANA_ERROR( | ||
| tsuba::ErrorCode::InvalidArgument, "expected {} found {} instead", | ||
| expected_name, schema->field(0)->name()); | ||
| auto renamed = out->RenameColumns({expected_name}); |
There was a problem hiding this comment.
I'm not sure renaming is necessary. This way nobody outside of this function can even tell that the name in the Parquet file is not the same as in the metadata. Perhaps nobody looks at it anyway?
There was a problem hiding this comment.
Yeah, renaming is important since we rely on the arrow schema to name properties. thanks!
Sorry, I totally forgot about it! Let me know if you have an update on this. This is still kind of required for the LynxKite integration. Thanks! |
|
@tylershunt Daniel mentioned about this PR in today's meeting. Would you review and approve this? |
Hi,
With this change I'm able to use the same Parquet files in a new RDG version under a new name.
I'm not sure about the C++.
ValueOrDiesounds scary, but it's after theok()check. Is there a better way?