Real-world code snippets which libcst fails to parse

[Zac-HD]

@DRMacIver and I have been on a bit of a bughunt lately: the following cases are accepted by Python 3.10's compile() function, but cause libcst 1.0.0 to raise an exception:

import libcst

def compiles(source: str):
    try:
        compile(source, "<string>", "exec")
        return True
    except Exception:
        return False

def libcst_parses(source):
    try:
        libcst.parse_module(source)
        return True
    except Exception:
        return False

for s in [
    # Assignment expression while indexing
    '_[_:=0]',

    # Complex f-string expression
    'f"{_:{_:}{_}}"',

    # Expecting indentation?
    '(\n \\\n)',
    '(\n    """\n""")',
    'if _:\n    """\n)"""',
    'if _:\n    ("""\n""")',
    'if _:\n     """\n  """',
    'if _:\n        """\n    """ ',

    # (legally) missing whitespace around keywords
    'lambda*_:_',
    'lambda**_:_',
    '_ if 0else _',
    'with _ as():_',
    '(a:=()for _ in _)',

    '_ if _ else""if _ else _',
    '_ if _ else()if _ else _',
    '(_ if _ else 0for _ in _)',
    '(_ if _ else""for _ in _)',
    '(_ if _ else()for _ in _)',

    '(lambda:()for _ in _)',
    'def _():return()if _ else _',
]:
    assert compiles(s), repr(s)
    assert not libcst_native_parses(s), repr(s)

Note that each was extracted from code in the wild, so there's already more user impact than from (say) #446.

I was hoping to finish this before 1.0 was released, but you're too fast 💖

[zsol]

OK I've got a fix for some of these, and have a pretty good idea about how to fix the remaining whitespace validation issues (similar to the lambda situation).

I haven't looked into the indentation issues yet.

About to head out on vacation, so if anyone wants to jump in on these, feel free to 😝

[zsol]

With the stack at #939 and #940, all of these parse successfully 🎉

[DRMacIver]

BTW owing to the way we found these, it's likely that many of them are actually different bugs than the ones we found in the wild and we may have a bunch more for you once we've retested with #939 and #940. Will let you know when we do!

(It's a test-case reduction thing called "slippage": Automatically transforming a test case into a smaller one subject only to the condition that it still has a bug in it doesn't guarantee that it's the same bug. In particular I think probably these no-whitespace-between-tokens ones are something that was introduced by deleting the whitespace rather than whatever the original bug was)

[zsol]

Sounds good. Closing this for now, as all fixes have landed. Thanks for reporting!

[Zac-HD]

I installed libcst == 1.0.1, and reproduced the following examples:

handcrafted = [
    # looks like an attribute lookup, actually keyword + float
    '_ in.0',
    'while.0:_',
    '_ if _ else.0',

    # escaped indentation of some kind?
    'fr"\\\n"',
    'if _:\n _\n\\\nif _:\ _',

    # mixed tabs and spaces; many variations here
    'if _:\n if _:\n\t _',

    # various ways to look like you're calling a keyword
    '(_ if _ else _)in _',
    '_ if _ else(lambda:_)',
    '_ if(_ if _ else _) else _',
    '_ if _ else(_ if _ else _)',
    'def _(): return(lambda:_)',
    'def _(): return(_ if _ else _)',
]
for s in handcrafted:
    assert compiles(s), repr(s)
    assert not libcst_native_parses(s), repr(s)

[zsol]

Some of these trigger an existential crisis in me. Let me open separate issues. How do you find these?

[DRMacIver]

How do you find these?

I think part of this is that we're accidentally fuzzing libcst with our test-case reducer.

Short version is:

1. We've got a large corpus of Python code from real world datasets and are checking for examples in it that parse with compile but not libcst
2. We've got a test-case reducer (takes strings satisfying some condition, turns them into smaller strings that still satisfy that condition) we're using to try to turn those examples into small ones we can report.
3. One of the things that test-case reducer does is delete individual characters from the string.
4. Unfortunately that often causes it to "slip" off the original bug, where it finds a string that compiles but doesn't parse with libcst because Python turns out to have a lot of cases where you don't need a space.

Which is why we get monstrosities like 'while.0:_' - almost certainly nobody has ever deliberately written while.0, but there was some real world example they had written that triggers a bug, but when you delete characters from it turns into something exhibiting this issue.

I'll have a go at trying to write a more precise condition that tries to capture the original bug, but it's necessarily a bit tricky to do.

[DRMacIver]

To my total bafflement, it turns out that no this isn't actually slippage from our reducer and people really do write code like this. e.g. 0.15 if 20<x<30 else.2 is a real piece of code found in the wild that lead to one of these examples.