bpo-40059: tomllib

[hukkin]

This adds a new standard library module, tomllib, for parsing TOML. The recently accepted PEP 680 -- tomllib is relevant here.

This PR has already seen some review in a PR under my personal fork: hukkin#2 (thanks to @encukou, @merwok, @hauntsaninja, @JelleZijlstra (I hope I'm not forgetting anyone)).

The implementation is based on Tomli which I plan to keep maintaining as a backport for Python versions 3.7, 3.8, 3.9 and 3.10, until finally Python 3.10 goes EOL.

Steps taken (converting tomli to tomllib)

• Move everything in tomli:src/tomli to Lib/tomllib. Exclude py.typed.

• Remove __version__ = ... line from Lib/tomllib/__init__.py

• Move everything in tomli:tests to Lib/test/test_tomllib. Exclude the following test data dirs recursively:

  • tomli:tests/data/invalid/_external/
  • tomli:tests/data/valid/_external/

• Create Lib/test/test_tomllib/__main__.py:

  import unittest
  
  from . import load_tests
  
  
  unittest.main()

• Add the following to Lib/test/test_tomllib/__init__.py:

  import os
  from test.support import load_package_tests
  
  def load_tests(*args):
      return load_package_tests(os.path.dirname(__file__), *args)

  Also change import tomli as tomllib to import tomllib.

• In cpython/Lib/tomllib/_parser.py replace __fp with fp and __s with s. Add the / to load and loads function signatures.

• Run make regen-stdlib-module-names

• Create Doc/library/tomllib.rst and reference it in Doc/library/fileformats.rst

edit: For reference, there's one more step – Add tomllib to Makefile.pre.in

https://bugs.python.org/issue40059

[hukkin]

A question: upstream (Tomli) is formatted with Black, using Black's defaults. This means a line length of 88. Should I reformat with line length at 79?

[hugovk]

This will need NEWS and "What's new" entries:

• https://devguide.python.org/committing/#updating-news-and-what-s-new-in-python

You can add this to https://github.com/python/cpython/blob/main/.github/CODEOWNERS

[mgorny]

Perhaps it'd make sense to include the tomli version used (or even the git commit hash), to make future syncing easier.

[hukkin]

Perhaps it'd make sense to include the tomli version used (or even the git commit hash), to make future syncing easier.

It was previously suggested that I remove Tomli version from the standard library port: hukkin#2 (comment)

I plan to include a migration guide in Tomli repository, also including commit hash that tracks what is currently in the stdlib.

[TeamSpen210]

Would it be good to mention in the docs why load() takes only binary files? The encoding requirement probably isn't obvious for first-time users.

[encukou]

I should get to the review this week or the next.

A question: upstream (Tomli) is formatted with Black, using Black's defaults. This means a line length of 88. Should I reformat with line length at 79?

No, I don't think that's worth it. And anyway, line length is not the only point where Black disagrees with PEP8 (starting with, like, the whole philosophy).
But future edits probably won't use Black.

Would it be good to mention in the docs why load() takes only binary files? The encoding requirement probably isn't obvious for first-time users.

Let's leave docs improvements to future PRs, so they get a proper discussion and don't delay this PR?

[encukou]

I have a few test nitpicks, but I'm happy to merge this!

[encukou]

MissingFile looks unnecessary. Does tomli need it?

[encukou]

Ah, I see it does.
For a poor man's xfail, you could you assert that p.stem is one of the expected failing cases.

[hukkin]

MissingFile looks unnecessary. Does tomli need it?

Yeah one of the two external test suites has a couple test cases where the expected data is missing.

For a poor man's xfail, you could you assert that p.stem is one of the expected failing cases.

I'm not sure I understand what you mean. Maybe you can show with the "Add a suggestion" feature?

[hauntsaninja]

I think something like:

if isinstance(expected, MissingFile):
    assert valid.stem in ("xfail_test_case1, "xfail_test_case2", ...)
    continue

[hukkin]

Oh yes, of course, thanks. Committed that.

[encukou]

For peace of mind, could you assert len(VALID_FILES) > 0, and same for INVALID_FILES?

[hukkin]

👍 Makes sense. Added the asserts.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @encukou for commit 2898cc3 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[hukkin]

It seems the failing CI job (AMD64 Arch Linux Usan PR) errors in other PRs too so should be unrelated to this PR.

[encukou]

The alpha 6 release is a bit bumpy and I don't want to destabilize it, so I'm holding off the merge until it's out.
I tested with all buildbots to see if there's an unforeseen platform-specific issue. It's common to see a few buildbots fail.

[encukou]

Let's get it in!