TableTools.readCsv validation of column names happens after all the loading work has been done.

[jcferretti]

For a huge CSV file that means you may have been waiting for several minutes for the data to load to only then blow up because a column has a name that doesn't validate.

We should do column name validation earlier, before the bulk of the load work.

[jcferretti]

The particular case I hit this is interesting. By default, Pandas's dataframe.to_csv method writes the index of the dataframe alongside the columns (unless you explicitly ask the call to omit it adding index=False to the named arguments). The index is written as the first column, but with no column name, with the net result of the header looking like this:

,column_1,column_2,column_3

And DHC trips over the empty column name with the validation exception. Interestingly enough, DHE bard loaded the same file no problem.

Example:

cfs@erke 14:18:46 ~/dh/oss1/deephaven-core/data
$ head -n 10 /a1/tmp/workqueue-no-nulls-100m.csv 
,workqueue_id,user_id,timestamp
0,1,1001,2021-11-07 14:00:00.011463807+00:00
1,2,1003,2021-11-07 14:00:00.080748962+00:00
2,3,1002,2021-11-07 14:00:00.114976256+00:00
3,4,1004,2021-11-07 14:00:00.118275772+00:00
4,5,1003,2021-11-07 14:00:00.131173174+00:00
5,6,1001,2021-11-07 14:00:00.174901899+00:00
6,7,1001,2021-11-07 14:00:00.194499326+00:00

DHE makes up a column name of Column1 for the first column above.

[devinrsmith]

#1469 is relevant to empty column names

[devinrsmith]

I'm happy to add empty column name support for our read csv - but I'm a bit curious why pandas would choose to write out an empty column name by default. Is there a way we can dataframe.to_csv w/ a named index column?

[devinrsmith]

https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_csv.html#pandas.DataFrame.to_csv:

index_labelstr or sequence, or False, default None

    Column label for index column(s) if desired. If None is given, and header and index are True, then the index names are used. A sequence should be given if the object uses MultiIndex. If False do not print fields for index names. Use index_label=False for easier importing in R.

[jcferretti]

The particular file I was working with was obtained via

import pyarrow.parquet as pq
df = pq.read_table("/blah.parquet").to_pandas()
df.to_csv("/bluh.csv")

So the index for the dataframe that was output to CSV was what was created by the conversion from pyarrow table to dataframe. Which I think is likely to just be the default index that is created by dataframe contructors when one is not provided, namely, a RangeIndex(0..rows-1) with no name.

If I am right in that guess, it means that a default initialized dataframe, when no index is provided explicitly during construction, results in a dataframe with an index with no name. Which, when output to CSV, creates a corresponding column with empty name. Which one could argue is a pretty sad outcome, but maybe this is the world we live in.

[pete-petey]

@devinrsmith and @jcferretti to watch it.

[kosak]

Fixed in #1629