-
Notifications
You must be signed in to change notification settings - Fork 84
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Apache Arrow as a recommended library and supported file attachment #223
Conversation
We're effectively pinned to these library versions, at least major if not minor versions, until we can version the standard library, correct? It feels like they are making steady, significant progress on this library and it's gone from v1 to v4 in the past year. All of the other libraries we've recently added I assumed were pretty stable, but I'm half worried Arrow could go from 4 to 7 before we can follow suit. |
We’re committed to backwards compatibility until we ship version pinning, yes. I don’t think we should block adding useful functionality on us shipping version pinning: I’d rather include a slightly out-of-date version of Apache Arrow in the box than nothing. |
Side note, but it does look like 5.0.0 is already planned per the package.json, but I can’t find any release notes, so I’m not really sure what’s different. In any case, I think we should still go ahead, but also redouble our efforts to ship version pinning. |
All Arrow packages are released every three months and every time there is a new major version. Note that the binary format is not changing. See https://arrow.apache.org/docs/format/Versioning.html for details. In the past few versions, the JS library hasn't changed much but for v5, we started some significant improvements to make the library leaner and more tree-shakeable. One breaking change we already added is apache/arrow#10277 (which you can work around easily by returning a |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can be updated to 4.0.1. But yeah ok, let's recommend Arrow!
Arrow 5.0.0 is out already, but since we plan on adding Arquero imminently, I figure we should stick to 4.0.1. |
The api hasn't changed much between 4 and 5. The biggest change is that tables don't extend data frame anymore (but data frames still extend tables). |
Is there a reason we can't do 5 then? Would it be incompatible with the current version Arquero? |
I don't think so but it would be good to confirm by updating arquero to v5. |
I will investigate upgrading to Arrow 5 at the time we add Arquero. |
Apache Arrow is exposed as
Arrow
in the standard library, and fileAttachment.arrow() returns a Promise to an Arrow.Table.