CSV sniffing falsely detects space as a delimiter

[ptokarski]

BPO	44677
Nosy	@rhettinger, @ptokarski
PRs	Do not allow to parse line breaks between quotes in sniffer regexes. #27256

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-07-19.17:40:33.684>
labels = ['3.8', 'type-bug', 'library']
title = 'CSV sniffing falsely detects space as a delimiter'
updated_at = <Date 2021-07-22.02:15:15.339>
user = 'https://github.com/ptokarski'

bugs.python.org fields:

activity = <Date 2021-07-22.02:15:15.339>
actor = 'rhettinger'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-07-19.17:40:33.684>
creator = 'pt12lol'
dependencies = []
files = []
hgrepos = []
issue_num = 44677
keywords = ['patch']
message_count = 4.0
messages = ['397821', '397859', '397860', '397974']
nosy_count = 3.0
nosy_names = ['rhettinger', 'python-dev', 'pt12lol']
pr_nums = ['27256']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44677'
versions = ['Python 3.8']

[ptokarski]

Let's consider the following CSV content: "a|b\nc| 'd\ne|' f". The real delimiter in this case is '|' character while ' ' is sniffed. Find verbose example attached.

Problem lays in csv.py file in the following code:

        matches = []
        for restr in (r'(?P<delim>[^\w\n"\'])(?P<space> ?)(?P<quote>["\']).*?(?P=quote)(?P=delim)', # ,".*?",
                      r'(?:^|\n)(?P<quote>["\']).*?(?P=quote)(?P<delim>[^\w\n"\'])(?P<space> ?)',   #  ".*?",
                      r'(?P<delim>[^\w\n"\'])(?P<space> ?)(?P<quote>["\']).*?(?P=quote)(?:$|\n)',   # ,".*?"
                      r'(?:^|\n)(?P<quote>["\']).*?(?P=quote)(?:$|\n)'):                            #  ".*?" (no delim, no space)
            regexp = re.compile(restr, re.DOTALL | re.MULTILINE)
            matches = regexp.findall(data)
            if matches:
                break

What makes matches non-empty and farther processing happens with delimiter falsely set to ' '.

[ptokarski]

Test sample:

import csv
from io import StringIO


def csv_text():
    return StringIO("a|b\nc| 'd\ne|' f")


with csv_text() as input_file:
    print('The following text is going to be parsed:')
    print(input_file.read())
    print()


with csv_text() as input_file:
    dialect_params = [
        'delimiter',
        'quotechar',
        'escapechar',
        'lineterminator',
        'quoting',
        'doublequote',
        'skipinitialspace'
    ]
    dialect = csv.Sniffer().sniff(input_file.read())
    print('The following dialect has been detected:')
    for dialect_param in dialect_params:
        print(f'- {dialect_param}: {repr(getattr(dialect, dialect_param))}')
    print()


with csv_text() as input_file:
    print('Parsed csv text:')
    for entry in csv.reader(input_file, dialect=dialect):
        print(f'- {entry}')
    print()

Actual output:

The following text is going to be parsed:
a|b
c| 'd
e|' f

The following dialect has been detected:
- delimiter: ' '
- quotechar: "'"
- escapechar: None
- lineterminator: '\r\n'
- quoting: 0
- doublequote: False
- skipinitialspace: False

Parsed csv text:
- ['a|b']
- ['c|', 'd\ne|', 'f']

Expected output:

The following text is going to be parsed:
a|b
c| 'd
e|' f

The following dialect has been detected:
- delimiter: '|'
- quotechar: '"'
- escapechar: None
- lineterminator: '\r\n'
- quoting: 0
- doublequote: False
- skipinitialspace: False

Parsed csv text:
- ['a', 'b']
- ['c', " 'd"]
- ['e', "' f"]

[ptokarski]

I think changing (?P<quote>["\']).*?(?P=quote) to (?P<quote>["\'])[^\n]*?(?P=quote) in all regexes does the trick, doesn't it?

[rhettinger]

Changing sniffer logic is risky because it risks breaking existing code that relies on the current predictions.

FWIW, in your example, the sniffer gets the desired result if given a delimiter hint:

>>> s = "a|b\nc| 'd\ne|' f"
>>> pprint.pp(dict(vars(Sniffer().sniff(s, '|'))))
{'__module__': 'csv',
 '_name': 'sniffed',
 'lineterminator': '\r\n',
 'quoting': 0,
 '__doc__': None,
 'doublequote': False,
 'delimiter': '|',
 'quotechar': "'",
 'skipinitialspace': False}