proposal: encoding/csv: add a way to limit bytes read per field/record

[nussjustin]

The csv.Reader has no way to limit the size of the fields read. In combination with LazyQuotes, this can lead to cases where a simple syntax error can make the Reader read everything until io.EOF into a single field.

Probably the simplest case (with LazyQuotes == true) is a quoted field, where there is some other rune between the closing quote and the comma (e.g. a,"b" ,c). In this case the second field will contain all bytes until either a single quote folowed by a comma or EOF is found. (See my comment in #8059)

This behaviour can lead to excessive memory usage or even OOM when reading broken CSV files.

To avoid this I propose to add a new, optional field to Reader to allow limiting the size of each field. When set the Reader would return an error as soon as it hits the limit.

Alternatively the limit could be per record instead. This could help especially with situations where FieldsPerRecord == -1, because there can be a different, basically unlimitted number of fields per record so a limit of 100 bytes per field doesn't help with e.g. missing new lines leading to records with many fields.

The new field would be specified as (as proposed by @bradfitz)

type Reader struct {
...
      // BytesPerField optionally specifies the maximum allowed number of bytes
      // per field. Zero means no limit.
      BytesPerField int

[nussjustin]

/cc @bradfitz @josharian

[josharian]

How about just hard-coding a limit, instead of adding API? I seriously doubt any reasonable csv file will contain a single field bigger than, say, 64k. We do something similar for, say, bufio.Scanner, with a note saying that those who find it problematic should switch to lower level reading.

[nussjustin]

That also works, but could break existing users (although I agree that there probably won't be any breaking code). In this case a per record limit would make even less sense (IMO) as CSV files can often have many fields (I have worked with files having hundreds or even thousands of fields per record) and then even small fields could trigger the limit.

In this case it's important to think about the limit. Too high and memory usage can still become "excessive", too low and existing programs could break. I would guess that in most cases when reading CSV the memory usage isn't really important or restricted and a limit of 32k or 64k would not be too high and still prevent most (if not all) problems related to the problem.

A different variant of this could make the Reader only limit fields when LazyQuotes is set and a lazy quote was found. This would still solve the problem for the "common" case and not break users that either don't set LazyQuotes to true or don't have CSV files that trigger this behaviour

[josharian]

A different variant of this could make the Reader only limit fields when LazyQuotes is set and a lazy quote was found.

This makes sense.

Related: #19019

[nussjustin]

I have a small patch ready that adds a limit for the LazyQuotes == true case when there is at least one lazy quote in a field, but although I like the idea I'm still not 100% sure this is the best approach.

I'd like to wait for a 3rd opinion on this.

[rsc]

It seems OK to have a limit of some kind, but someone needs to propose a concrete API.

[rsc]

On hold for concrete API.

[bradfitz]

We already have encoding/csv.Reader.FieldsPerRecord int, so I propose:

type Reader struct {
...
      // BytesPerField optionally specifies the maximum allowed number of bytes
      // per field. Zero means no limit.
      BytesPerField int