Add load/export support for Arrow IPC Stream #2625
Conversation
| @@ -25,7 +25,7 @@ function cutColumns(program, columns) { | |||
|
|
|||
| export function prepareProgram(format, program, columns) { | |||
| let p = cutColumns(program, columns) | |||
| if (format === "csv") p += " | fuse" | |||
| if (format === "csv" || format === "arrows") p += " | fuse" | |||
There was a problem hiding this comment.
Maybe add Parquet while you're here?
| if (format === "csv" || format === "arrows") p += " | fuse" | |
| if (format === "arrows" || format === "csv" || format === "parquet") p += " | fuse" |
There was a problem hiding this comment.
@nwt: As much as I'd love to, I've been holding off on adding support for Parquet export in the app. As described in #2591 and brimdata/super#4215, failures mid-export currently aren't being shown to the user and so there's a hazard of the user having incomplete output and not knowing about it. Granted, this seems to be a general problem with API interactions for both query and load and hence affects other formats as well (#2606, #2554). But in testing with various data I found it was way too easy to trigger the "union type unsupported" with Parquet export in particular, so that's been my rationale for holding it back. I know you've been doing more work with Parquet due to the new Arrow library so I've been waiting for the dust to settle on that before revisiting the topic, since I wondered if maybe the union support might magically appear. 😄 If you feel strongly about getting Parquet export in sooner I could surely change my tune since there's no perfect answers here. On the whole I see this topic of absent error messages during API interactions as a Zui v1.0 release blocker, so it will definitely not be forgotten as a priority.
There was a problem hiding this comment.
As much as I'd love to, I've been holding off on adding support for Parquet export in the app.
That's fine. I don't feel strongly about this.
I know you've been doing more work with Parquet due to the new Arrow library so I've been waiting for the dust to settle on that before revisiting the topic, since I wondered if maybe the union support might magically appear.
I'm afraid it won't since the Parquet format itself doesn't support unions.
brimdata/super#4252 added support for loading/querying in Arrow IPC Stream in Zed, so here we add support for it in the app as well.