pyproject.toml 'include' directive parsing bug.

[RichardBruskiewich]

Highlighting here but elaborated a bit in #6094 (comment)

• Poetry version: 1.2.2
• Python version: 3.8.10 (Windows) / 3.10.6 (Ubuntu)
• OS version and name:
  • Cygwin under Windows 11
  • Ubuntu 22.04.1 LTS (GNU/Linux 5.15.74.2-microsoft-standard-WSL2 x86_64)
• pyproject.toml: https://gist.github.com/RichardBruskiewich/7c41ce4ea3b61224dcbf3080df04b9d0

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate (but slightly related to The include doc is missing the type key #6094 (comment)).
  • Now recognized to be a possible duplication of an unresolved 3 year old issue mixed types in array don't work uiri/toml#270
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Traceback (most recent call last):
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/__init__.py", line 185, in console_main
    code = main()
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/__init__.py", line 143, in main
    config = _prepareconfig(args, plugins)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/__init__.py", line 318, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/helpconfig.py", line 100, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1003, in pytest_cmdline_parse
    self.parse(args)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1283, in parse
    self._preparse(args, addopts=addopts)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1157, in _preparse
    self._initini(args)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1076, in _initini
    rootpath, inipath, inicfg = determine_setup(
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/findpaths.py", line 184, in determine_setup
    rootdir, inipath, inicfg = locate_config([ancestor])
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/findpaths.py", line 106, in locate_config
    ini_config = load_config_dict_from_file(p)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/findpaths.py", line 69, in load_config_dict_from_file
    config = toml.load(str(filepath))
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/toml/decoder.py", line 134, in load
    return loads(ffile.read(), _dict, decoder)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/toml/decoder.py", line 511, in loads
    ret = decoder.load_line(line, currentlevel, multikey,
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/toml/decoder.py", line 778, in load_line
    value, vtype = self.load_value(pair[1], strictly_valid)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/toml/decoder.py", line 880, in load_value
    return (self.load_array(v), "array")
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/toml/decoder.py", line 1002, in load_array
    a[b] = a[b] + ',' + a[b + 1]
IndexError: list index out of range

Issue

I added the following directive to a pyproject.toml file:

include = [
    "CHANGELOG.md",
    { path = "tests" },
    { path = "docs" }
]

Using my library, e.g. just running PyTests, triggers a IndexError: list index out of range exception reading in deep in the toml decoder.py module here:

                    while (not self.bounded_string(ab) or
                           (len(ab) > 2 and
                            ab[0] == ab[1] == ab[2] and
                            ab[-2] != ab[0] and
                            ab[-3] != ab[0])):
                        a[b] = a[b] + ',' + a[b + 1]  # <= site of error

but this error is not triggered either by

include = [
    "CHANGELOG.md"
]

alone, nor by

include = [
    { path = "tests" },
    { path = "docs" }
]

alone.

I guess I'll remove one or the other components of the include, but it would be helpful to know what's happening (and fix it!) or, at least, document the constraints of include usage accordingly.

[neersighted]

Please follow the issue template -- provide a complete report with versions, a reproducing pyproject.toml, and a step-by-step explanation of what you expect, what you get, and the commands you ran.

[neersighted]

Again, this is unlikely to be looked at if you don't follow the issue template. The template is there to avoid wasting reporters and contributors time; please fill it out so that people trying to reproduce clearly understand what versions and platform you are testing on. Similarly, the complete verbose output of the commands you are experiencing exceptions with is important, as it lets contributors ensure that they are indeed looking at the same symptom and cause as you.

I understand you think this is a trivial case, but again, we have the template for a reason, and it avoids wasting your time (because you've reported a non-bug, a fixed issue, or simply have provided a poor quality report no one is interested in looking at) as well.

[RichardBruskiewich]

@neersighted done

[dimbleby]

what command are you running to provoke this error? the stack that you've given doesn't appear to involve poetry at all, it's not clear why you've raised this bug against poetry rather than, say, pytest?

[dimbleby]

indeed this looks like a duplicate of uiri/toml#270 (which would have been the appropriate place to raise this)

[RichardBruskiewich]

1. It's a poetry TOML parsing error since tweaking the TOML file removes the runtime error. My understanding is that any aspect of the project code that triggers the TOML parsing will crash. Seems to me that the issue doesn't lie with the code I've written.

2. OK, fine... so I didn't stumble across the other relevant issue among the >500 other issues in your tracker. My sincere apologies. I can close my issue with this cross-reference?

I'm just a member of the bit herd of software engineers working globally who find poetry useful but occasionally vexing. Maybe just fix the bug? The "old" bug report dates to Nov 30, 2019. That seems kind of ancient...You wouldn't be getting my bug report now if it had been fixed 3 years ago...

[neersighted]

Let's slow down a second -- there's no need to get heated.

Following the issue template has revealed that this is in fact, not a Poetry issue. The pyproject.toml file is not exclusively Poetry's domain, and the issue you are hitting does not involve any Poetry code.

What you are encountering is a bug in the uiri/toml library, which does not like what is otherwise a valid construct (a list with mixed strings and tables). Poetry supports such a thing, and well-behaved TOML implementations will not object to it. Another tool you are using (I see pytest in the stack trace, so I'd start looking there) is using uiri/toml, which fails to parse a valid TOML document.

While Poetry is involved in the sense that you are editing a Poetry section of the pyproject.toml, it is not involved in the sense that the stack trace you have provided clearly shows no Poetry code is involved, and there is nothing that can be done in the Poetry project itself to resolve this.

You are correct that we still have too many issues, and that they often are unaddressed for longer than we would like. However, I would like to point out that we had 1500 issues as little as two months ago, and a very large effort is being put in to improve the signal-to-noise ratio of this tracker.

Having high-quality reports that follow the template is a key part of that effort; we can see that exhibited in this issue, where a well-formed issue report quickly revealed that this was confusion over where certain code lived, and was in fact an external issue. Anyone interested in this issue now knows where it needs to be fixed and where they can follow progress, and contributors are not spending time trying to chase down an issue without a stack trace (indeed, I was unable to reproduce with your pyproject.toml as no Poetry code is involved in your issue, and that was not obvious before the full report).

In any case, I'd finally like to remind you that both Poetry and uiri/toml are open-source, volunteer-led projects. I understand you're frustrated that an issue over there has gone unresolved for three years, but please keep in mind that these projects are worked on by individuals for no compensation in their free time, because they enjoy creating something others find useful. There is no warranty or contractual relationship -- you're free to report issues, or offer up code for others to use, but the project is under no obligation to do anything.

Please follow up with the project in question, but also remember to be understanding, and patient. If you are able and willing to contribute technical work like code towards a fix, it's often the fastest way to solve these issues -- though I will admit that the parser internals are likely complex enough that a regular contributor is a much more likely source for a fix.

[RichardBruskiewich]

@neersighted, I deeply appreciated your frank reply. I sincerely apologize for the consternation my own comments inspired on your part.

As a good colleague of mine remarked when I was once on the other (receiving) side of such comments: "...maybe so..and..so is just having a bad day on their (software engineering) side...". I hope that you can make such allowances for me today.

I was obviously mistaken in considering the error within the remit of poetry: the pyproject.toml file in my project was only created when I migrated the project to Poetry.. in fact, poetry created the initial file and I simply added to it following poetry documented additions. I guess I'd be forgiven for incorrectly assuming it was a poetry bug... I was inadvertently - not so blissfully - unaware that poetry and pytest share the same name for a configuration file. What an eye opener! Talk about (admirable!) standards reuse

I furthermore now also understand the point that there are multiple diverse implementations of TOML parsing, some more compliant than others with the expected constructs.... and that Poetry and Pytest each use different libraries to parse their TOML file! Talk about lack of reuse of standards, in terms of libraries!

On the issue of open source, I've long been a fan of, and contributor to, many such projects, so I do understand your angst with my harsh comments.

Again, please accept my apologies... and also, my gratitude for your thoughtful reply.

For now, I'll just avoid the offending construct in the TOML file in question since I don't actually enough professional or personal degrees of freedom at the moment to jump into the fray of uiri/toml to fix this bug... although all of the information gleaned here about the coding landscape for this was very helpful (I'm truly sorry that I didn't pick up on any of this earlier...)

[dimbleby]

uiri/toml is stagnant and gradually being eased out of the ecosystem - you wouldn't want to make fixes there even if you had the time and inclination to do so.

recent versions of pytest have moved to using tomli - pytest-dev/pytest#8857. (And what's more, recent versions of python have brought tomli into the standard library - as tomllib)

So the general situation with respect to toml-parsing libraries is improving. Probably you could avoid your original problem by upgrading to a recent pytest.

[RichardBruskiewich]

@dimbleby thank you for the head's up on this!

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.