comments get deleted

[jugmac00]

 [testenv:mypy]
 deps =
     mypy
 commands =
-    # do not lint tests yet; waiting for pytest 6.0 release
     mypy --strict src {posargs}

and

 [pytest]
-# only collect intended test classes, e.g. TestSaving
-# do not collect e.g. TestingFileStorage
 python_classes = Test[A-Z]*

whereas the first "bug" is pretty nice, as pytest 6.0 got released already some time ago :-)

[jugmac00]

ConfigParser strips comments on reading, see https://github.com/python/cpython/blob/2ef5caa58febc8968e670e39e3d37cf8eef3cab8/Lib/configparser.py#L993-L1035

In another project, which uses GPG encrypted ini files for secrets, I also reported deleted comments, they switched from ConfigParser to ConfigUpdater
flyingcircusio/batou@b252f7a

https://pypi.org/project/ConfigUpdater/

While this approach seems ok for updating single values, this does not look like a lot of fun to implement for this project here 😱

That said, it looks like @asottile 's setup-cfg-fmt does not handle comments either.

[jugmac00]

As mentioned in #34, there are some more issues with comments.

What is your intention for comments? Support them? Delete them? Should tox-ini-fmt "cowardly refuse" to change a tox.ini which contains comments?

[gaborbernat]

Support them. TBD how though.

[hugovk]

Maybe also support (maximum one?) blank line?

 commands =
-    # Unit tests
     {envpython} -m pytest --cov tinytext --cov tests --cov-report xml {posargs}
-
-    # Test runs
     tinytext --version
     tinytext --help
     tinytext abcdef

[jugmac00]

This will be pretty interesting!

@gaborbernat maybe this is something for your next Twitch stream? I'd like to know your thoughts about the alternative ways to handle this.

Some random thoughts...

Imho, this problem breaks down into several smaller problems:

• read config file without stripping the comments
• parse values into a data structure which is aware of which key / entry goes with which comment
• re-assemble the values
• write back config file

I played a bit with ConfigUpdater - and it looks pretty promising...

a config like ...

[section]
# holla

key = value

# comment in section

will result in...

(Pdb++) cu["section"]._structure
[<Comment>, <Space>, <Option: key = value>, <Space>, <Comment>, <Space>]

Pretty nice!

Buuut, it breaks with @hugovk's https://github.com/hugovk/tinytext/blob/master/tox.ini

❯ python main.py 
Traceback (most recent call last):
  File "main.py", line 4, in <module>
    cu.read("tiny-tox.ini")
  File "/tmp/ConfigUpdater/venv/lib/python3.8/site-packages/configupdater/configupdater.py", line 618, in read
    self._read(fp, filename)
  File "/tmp/ConfigUpdater/venv/lib/python3.8/site-packages/configupdater/configupdater.py", line 826, in _read
    raise e
configparser.ParsingError: Source contains parsing errors: 'tiny-tox.ini'
        [line 11]: '    {envpython} -m pytest --cov tinytext --cov tests --cov-report xml {posargs}\n'
        [line 14]: '    tinytext --version\n'
        [line 15]: '    tinytext --help\n'
        [line 16]: '    tinytext abcdef\n'

Looks like ConfigUpdater fails when comments are between keys and values :-/

Which brings me back to the good ole ConfigParser- which - by default - strips comments (starting with # and ;).

But there is a workaround. When initializing ConfigParser, we could pass in an arbitrary comment-prefix - like "//" - then real comments, starting with # do not get stripped (or just pass in an empty list of prefixes).

e.g.

>>> list(list(cu.items())[2][1].items())
[('passenv', '\nFORCE_COLOR'), ('commands', '\n# Unit tests\n{envpython} -m pytest --cov tinytext --cov tests --cov-report xml {posargs}\n\n# Test runs\ntinytext --version\ntinytext --help\ntinytext abcdef'), ('commands_pre', '\n{envpython} -m pip install -r requirements.txt')]

Another problem

comments before the first section

# This is a simple example with comments.
[bug_tracker]
url = http://localhost:8080/bugs/
username = dhellmann
; You should not store passwords in plain text
; configuration files.
password = SECRET

When we fake the comment_prefixes to keep comments...

>>> c.read("motw.ini")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/configparser.py", line 697, in read
    self._read(fp, filename)
  File "/usr/lib/python3.8/configparser.py", line 1082, in _read
    raise MissingSectionHeaderError(fpname, lineno, line)
configparser.MissingSectionHeaderError: File contains no section headers.
file: 'motw.ini', line: 1
'# This is a simple example with comments.\n'

If comments before the first section need to be supported, that could mean e.g. that we need to read the file with e.g. open, memorize the comments before the first section, strip them, and pass the rest to the ConfigParser :-/

[gaborbernat]

But there is a workaround. When initializing ConfigParser, we could pass in an arbitrary comment-prefix - like "//" - then real comments, starting with # do not get stripped (or just pass in an empty list of prefixes).

This is promising. Can play around with something like this on my next stream. Thanks a ton for the in-depth research.

[hugovk]

Here's another example. First bit the same as before, second bit slightly different:

--- tox.ini

+++ tox.ini

@@ -10,10 +10,7 @@

     pandas
     tests
 commands =
-    # Unit tests
     {envpython} -m pytest --cov pypistats --cov tests --cov-report xml {posargs}
-
-    # Test runs
     pypistats --version
     pypistats --help
     pypistats recent --help
@@ -34,6 +31,5 @@

     {envpython} -m pip install -r requirements.txt

 [testenv:py39]
-# NumPy and pandas' dependency Cython doesn't yet support Python 3.9
 extras =
     tests

https://github.com/hugovk/pypistats/blob/master/tox.ini

[ssbarnea]

I was extremely happy to find this formatter until I observed that comments gets stripped, this being a deal breaker for most projects. I even doubt that I know a project without comments inside tox.ini, some of them extremely valuable. I really hope someone comes with a solution for this.

[gaborbernat]

I personally don't have comments in any of my projects, but that's probably just me. That being said PR's are very welcome to address this shortcoming 👍

[ssbarnea]

I bet you don't need comments because you know tox better than anyone else. Still, for bigger teams we often find the need to explain why we decided to use a specific option, so we avoid having someone else undo it by mistake. One recent example that I remember is using the trick of disabling tox install and use deps = --editable . in order to make the project that does not have setup.py work.

I wish I would have the time to do a PR on this but I am afraid that is not easy to implement or to find some free cycles. BTW, thanks again for pointing me to this project.

[pdecat]

One workaround that fits me is to put comment at the end of otherwise non empty lines, e.g.:

# tox-ini-fmt tox.ini
--- tox.ini

+++ tox.ini

@@ -26,7 +26,7 @@

     .venv,
     .virtualenv,
     __pycache__
-ignore= # See https://black.readthedocs.io/en/stable/faq.html#why-are-flake8-s-e203-and-w503-violated
+ignore = # See https://black.readthedocs.io/en/stable/faq.html#why-are-flake8-s-e203-and-w503-violated
     E203,
     W503
 enable-extensions =

Edit: this does not work as it breaks flake8 in this case, as comments must be on their own line as per configparser specification.

[ssbarnea]

Considering the age of this issue and the very limited time of all involved, I believe that is unlikely to be ever fixed, especially as we are more likely to switch config to pyproject.toml anyway.

I searched for other generic ini formatters but none is usable. The prettier plugin for ini files is far worse as it crashes on list entries from tox.ini.