Large Memory footprint for processing String results

[pnorth1]

I'm on Julia v1.0.3 with ODBC.jl version v0.8.1, querying Snowflake cloud database.

I'm noticing that a large amount of memory is used in construction of my DataFrames that contain a String column.

I'm running a query that returns 1 String column with 300 rows, and seeing:
- Output from @time:
3.090998 seconds (1.29 k allocations: 4.688 GiB, 3.20% gc time)
- Info from varinfo():
df 7.704 KiB 300×1 DataFrame

In the database the column type is VARCHAR(16777216) even though the strings themselves are not more than 14 characters. Unfortunately we're unable to change the column type as we need to keep the ability to store arbitrary length strings there.

Is there anything we could try as a workaround or anything in ODBC.jl that could make memory usage more efficient/intelligent? Any help appreciated, thanks!

[pnorth1]

After poking around some more, if I select the same set of 300 rows from a column declared as VARCHAR(128), the memory allocated (and time spent) in DataFrame construction appears much more reasonable:
@time gives 0.436457 seconds (2.77 k allocations: 149.313 KiB).

[quinnj]

Should be fixed on master