-
Notifications
You must be signed in to change notification settings - Fork 60
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Roundtrippability of special strings #196
Comments
Fixes apache/arrow-julia#196. This utilizes the new package extension feature of Julia 1.9 to add a conditional dependency on the ArrowTypes.jl package. With ArrowTypes.jl, it adds the necessary overloads to allow round- tripping of inline strings through the arrow format. Other language implementations will read them as normal strings, but in the Julia implementation, the additional type metadata signal that these strings were originally inline strings and can be deserialized as such. I'm explicitly not using the Requires.jl hack for backwards compat w/ older Julia versions because I like the idea of this being sort of a "beta" feature for users already using 1.9 to see if there are any unexpected issues that pop up for inline strings in the arrow format.
PR up to add support for InlineStrings round-tripping in arrow: JuliaStrings/InlineStrings.jl#66 |
* Add package extension to support InlineStrings in Arrow.jl Fixes apache/arrow-julia#196. This utilizes the new package extension feature of Julia 1.9 to add a conditional dependency on the ArrowTypes.jl package. With ArrowTypes.jl, it adds the necessary overloads to allow round- tripping of inline strings through the arrow format. Other language implementations will read them as normal strings, but in the Julia implementation, the additional type metadata signal that these strings were originally inline strings and can be deserialized as such. I'm explicitly not using the Requires.jl hack for backwards compat w/ older Julia versions because I like the idea of this being sort of a "beta" feature for users already using 1.9 to see if there are any unexpected issues that pop up for inline strings in the arrow format. * Only test package extension on 1.9
So if a Julia user doesn't want to depend on ArrowTypes.jl, they will get back normal String right? |
Correct (though it's whether the user has InlineStrings loaded or not, not ArrowTypes): julia> t = Arrow.Table("/Users/quinnj/.julia/dev/inlinestrings.arrow")
┌ Warning: unsupported ARROW:extension:name type: "JuliaLang.InlineStrings.InlineString7", arrow type = String
└ @ Arrow ~/.julia/dev/Arrow/src/eltypes.jl:53
Arrow.Table with 3 rows, 1 columns, and schema:
:x String |
I see... to certain degree I feel like this kind of support is a bit of rabbit hole, maybe InlineString is special enough to justify. But in large applications, it might be a footgun that users get different type in schema depending (invisible to user because some other dependency might load it) if another package is loaded or not. |
Btw Legolas.jl will error if |
As discussed on Slack, it would be nice if this would work:
The text was updated successfully, but these errors were encountered: