encoding/csv: unhelpful error location with badly quoted fields

[josharian]

I'm working with hundreds of 100,000+ line tsv files. They contain quotes in inappropriate places, so I am using LazyQuotes. However, somewhere in the guts of a few of the files, there are quotes at the beginning of a field, which breaks csv.Reader. That is #6287. When that happens, and LazyQuotes is on, the reported error location is at the very end of the file. See https://play.golang.org/p/y0jOObyBTV for an illustration of this.

However, if I turn LazyQuotes off, I get an error location on some other quotes that do no harm (with LazyQuotes on). I am left with an unreadable file that I cannot even fix manually, since there are some errors somewhere in hundreds of thousands of lines, but encoding/csv does not tell me where.

The error reported when LazyQuotes is on should probably at the beginning of the field in question if EOF is reached.

Also related: #3150.

I have some workarounds available. Just adding this to the list of encoding/csv quote issues. :)

[nussjustin]

For ErrFieldCount (which seems to be the problem child here) the column is always reported as 0 to make it clear that the whole record is the problem. I think the bigger problem here is the reported line. In the code from the playground link above, the Reader reads until EOF which is in line 5, but the record starts in line 3 and the other lines shouldn't even be part of the line. This could be solved by tracking the line a record started on and using this for errors instead or adding it as additional field to ParseError.

[nussjustin]

/cc @bradfitz @pborman

[gopherbot]

Change https://golang.org/cl/52830 mentions this issue: encoding/csv: report line start line in errors

[nussjustin]

I send a change which improves the issue described here by reporting the line where a record started, as proposed in my first comment. With this the Reader still reports column 0 for ErrFieldCount, but it should be much easier to find the problems in the files.

[nussjustin]

5d14ac7 was just merged, so with tip the code from the play ground example now reports

line 3, column 4: extraneous " in field
line 3, column 0: wrong number of fields in line

This seems much better to me, since it now reports the correct line. The column is still 0, but this is an explicit design choice and I think that in this case (ErrFieldCount) there isn't much value to the exact column when we already have the line.

[josharian]

@nussjustin seems good enough to me. Thanks!

[gopherbot]

Change https://golang.org/cl/72050 mentions this issue: encoding/csv: deprecate ParseError.Column

[gopherbot]

Change https://golang.org/cl/72150 mentions this issue: encoding/csv: simplify and optimize Reader