use of setmetadata! on generic tables basically requires calling Tables.columns on the input

[jrevels]

I thought there was an already issue for this, but couldn't find it:

julia> using Arrow, Tables

julia> t = Arrow.Table(Arrow.tobuffer((; x = [1,2,3], y = ["a", "b", "c"])))
Arrow.Table with 3 rows, 2 columns, and schema:
 :x  Int64
 :y  String

julia> Arrow.setmetadata!(t, Dict("key" => "value"))

julia> Arrow.getmetadata(t)
Dict{String,String} with 1 entry:
  "key" => "value"

# where did my metadata go?
julia> Arrow.getmetadata(Arrow.Table(Arrow.tobuffer(t))) isa Nothing
true

julia> t_cols = Tables.columns(t);

julia> Arrow.setmetadata!(t_cols, Dict("key" => "value"));

# but this works
julia> Arrow.getmetadata(Arrow.Table(Arrow.tobuffer(t_cols)))
Dict{String,String} with 1 entry:
  "key" => "value"

IIUC the reason for this is that Arrow.write(_, t) internally calls Tables.columns(t) and then tries to grab THAT object's metadata instead of the original t's metadata (IIRC the exact place where this happens is here; I'd have to double check though, it's been awhile since I tracked it down).

[haberdashPI]

If I understand this problem, what's ultimately needed is a change in Tables that provides support for metadata (e.g. by defining an interface for it that types can implement, or by transfering metadata to new objects upon a call to Table.columns etc... so that the new objects are also associated with the metadata in a global table).

[ericphanson]

I think an alternate solution would be #90 -- just removing the metadata cache altogether, and only support it at read and write time. I think something like table, metadata = Arrow.read("file.arrow") and Arrow.write("file.arrow", table, metadata) would be simple and clear, and avoid these issues. Personally, I don't think Arrow should be in the business of trying to persist metadata through various Julia types and should just stick to (de)-serialization since there isn't a fully established metadata story in Julia's data ecosystem in general.

[jrevels]

closed by #238