On Windows only (via GHA): TOML parsing error: "Couldn't read config file pyproject.toml: Unescaped '\' in a string"

[AndydeCleyre]

Describe the bug

I have a pyproject.toml which includes an entry:

install = "if [ $VIRTUAL_ENV ]; then pip install -r local-requirements.txt; else printf '%s\n' 'Please activate a venv first'; return 1; fi"

It's in a section unrelated to test running or coverage operations.

This seems fine on Linux and MacOS, but as I add a Windows 2022 runner to the coverage-invoking GitHub Actions workflow, which invokes the nox session:

nox.options.default_venv_backend = 'venv'
nox.options.reuse_existing_virtualenvs = True


@nox.session(python=['3.7', '3.8', '3.9', '3.10', '3.11'])
def test(session):
    """Run all tests."""
    session.install('.[test,toml]', 'coverage')
    session.run('coverage', 'run', '-p', '-m', 'ward', *session.posargs)

the following failure is produced:

Run nox -s test test_without_toml combine_coverage --force-color
  nox -s test test_without_toml combine_coverage --force-color
  shell: C:\Program Files\PowerShell\7\pwsh.EXE -command ". '{0}'"
nox > Running session test-3.7
nox > Creating virtual environment (venv) using python.exe in .nox\test-3-7
nox > python -m pip install '.[test,toml]' coverage
nox > coverage run -p -m ward
Couldn't read config file pyproject.toml: Unescaped '\' in a string (at line 62, column 21)

I've checked the TOML file with https://www.toml-lint.com/ and no problems were reported.

To Reproduce

1. Python versions: 3.7.9, 3.8.10, 3.9.13, 3.10.8
2. coverage==6.5.0; coverage debug sys fails with Couldn't read config file pyproject.toml: Unescaped '\' in a string (at line 62, column 21)
3. Package versions for the Python 3.7 run:

arrow==1.2.3
attrs==22.1.0
cattrs==22.2.0
click==8.1.3
click-completion==0.5.2
click-default-group==1.2.2
colorama==0.4.6
commonmark==0.9.1
coverage==6.5.0
cucumber-tag-expressions==4.1.0
exceptiongroup==1.0.0rc9
importlib-metadata==5.0.0
inform==1.27.0
Jinja2==3.1.2
MarkupSafe==2.1.1
nestedtext==3.4.0
nt2 @ file:///D:/a/nestedtextto/nestedtextto
pluggy==1.0.0
plumbum==1.8.0
pprintpp==0.4.0
Pygments==2.13.0
python-dateutil==2.8.2
pywin32==304
rich==12.6.0
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.7
shellingham==1.5.0
six==1.16.0
tomli==2.0.1
tomli-w==1.0.0
typing-extensions==4.4.0
ward==0.66.1b0
yamlpath==3.6.7
zipp==3.10.0

4. This repo at this commit exhibits the problem. At that commit, the problem is encountered before the regular coverage run, at the coverage debug sys invocation. At the next commit, the same problem is raised at the point of coverage run -p -m ward.
5. nox -s test (in a Windows environment) should do the trick.

Expected behavior
The TOML is parsed without issue and tests are run.

[AndydeCleyre]

I'm realizing this is probably a tomli issue, rather than belonging here.

[AndydeCleyre]

As per comments at hukkin/tomli#210, I was unable to reproduce using only tomli.

[hauntsaninja]

Like I mentioned on the tomli issue, I suspect this is related to the substitution stuff coverage is doing. Could you check the following snippet?

import os
import coverage.tomlconfig
import tomli

with open("pyproject.toml", "r", encoding="utf-8") as f:
    toml_text = f.read()
tomli.loads(toml_text)  # should pass
substituted_toml_text = coverage.tomlconfig.substitute_variables(toml_text, os.environ)
tomli.loads(substituted_toml_text)  # should fail

Could you also provide the value of the VIRTUAL_ENV environment variable on the systems you see the issue on?

[AndydeCleyre]

That snipped succeeds in the environment, when defined as:

@nox.session(python=['3.10'])
def tomli_check(session):
    """
    Temporary investigation for an issue with either tomli or coveragepy.

      - hukkin/tomli#210
      - nedbat/coveragepy#1481
    """
    code = dedent(
        '''
        import os
        import coverage.tomlconfig
        import tomli

        with open("pyproject.toml", "r", encoding="utf-8") as f:
            toml_text = f.read()
        tomli.loads(toml_text)  # should pass
        substituted_toml_text = coverage.tomlconfig.substitute_variables(toml_text, os.environ)
        tomli.loads(substituted_toml_text)  # should fail
        '''
    )
    session.install('tomli', 'coverage')
    session.run('pip', 'freeze')
    print(session.env)
    for name in ('python', 'python.exe'):
        if (Path(session.bin) / name).exists():
            interpreter = str(Path(session.bin) / name)
            break
    session.run(interpreter, '-c', code)

I'm seeing 'VIRTUAL_ENV': 'D:\\a\\nestedtextto\\nestedtextto\\.nox\\tomli_check-3-10'

[nedbat]

I'll have to figure out how to test this, but the theory about coverage.py's substitution is a good one. For other configuration files, we substitute each value individually. For toml we (incorrectly) perform substitution on the text of the entire toml file before parsing.

[nedbat]

Though, toml is typed, so values are retrieved as ints, bools, lists, etc, so we can't wait until we've read the value to do substitution... Hmm. Looking for clever ideas here.

[hauntsaninja]

I think it might be okay to substitute after loading the value? Maybe we can get away with just doing something like:

def substitute_variables_recursive(obj, variables):
    if isinstance(obj, dict):
        return {k: substitute_variables_recursive(v) for k, v in obj.items()}
    if isinstance(obj, list):
        return [substitute_variables_recursive(v) for v in obj]
    if isinstance(obj, str):
        return substitute_variables(obj, variables)
    return obj

Or is the concern backwards compatibility, e.g. that people had (invalid) "TOML" like x = $env that became valid TOML once substituted? If so, I think it might be okay to break... pyproject.toml should be valid TOML / many other tools that read from pyproject.toml would break if it wasn't.

[nedbat]

I'm not concerned about weird meta-toml like you showed :) The point wasn't to insert gobs of toml, it was to have substitution in strings before they were parsed as values. The recursive solution looks like it could work, thanks.

[nedbat]

The failure can be reproduced on any OS if a line references an environment variable:

[othersection]
something = "if [ $OTHER ]; then printf '%s\\n' 'Hi'; fi"

and OTHER has a backslash that isn't a valid escape:

export OTHER=hi\zebra

[nedbat]

I think I have it all fixed, can you give it a try?:

pip install git+https://github.com/nedbat/coveragepy@nedbat/bug1481-with-metacov

[AndydeCleyre]

Yes, that's working just fine alongside the funky TOML line in CI now, thanks!

[nedbat]

This is now fixed in commit b3a1d97

[nedbat]

This is now released as part of coverage 6.6.0b1.

[nedbat]

This is now released as part of coverage 7.0.0b1.