Multi-line comments in Python are counted as code

[svisser]

For the following file example.py:

def main():
    """
    Docstring.
    """
    # This is a comment.
    pass

Tokei 5 reports the following:

-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 Python                  1            7            5            1            1
-------------------------------------------------------------------------------
 Total                   1            7            5            1            1
-------------------------------------------------------------------------------

whereas cloc 1.72 reports 4 comment, 2 code, 1 blank which I believe is correct as Python supports multi-line comments with """ and '''.

I tried to fix this by moving those comment characters in languages.json to "multi" and by testing it again using cargo build and ./target/debug/tokei example.py but that didn't make a difference.

Is this possibly due to the way it detects multi-line comments? I did notice that Python would be the first language in languages.json to use escaping when expressing multi-line comment strings but I don't know if that's related.

[XAMPPRocky]

Hello, thanks for filing the issue. I don't why they would be comments as they are str's. x = """Str""" is valid code, whereas x = # comment isn't. I know later in parsing the long str's are then attached as a docstring to the function in the AST, and reaching parity with that would require code too specific just for python.

[svisser]

@Aaronepower ah, thanks, that makes sense. I'm used to thinking of those strings early in a function as comments but yes they're indeed multi-line strings and not multi-line comments as part of the language.

To keep it generic, It may be that this approach also occurs in other programming languages but I'm not aware of any.

Feel free to close this issue as this clarifies why tokei counts the lines the way it does.

[flother]

I think @svisser is on the right track here and I don't think Tokei should count strings like this as Python code.

If the first statement in a Python module, class, method, or function is a string literal, it's considered a docstring and the statement becomes a special attribute on the class/method/whatever:

>>> def hello_world():
...     """Return the string 'Hello, world!'."""
...     return 'Hello, world!'
...
>>> hello_world.__doc__
"Return the string 'Hello, world!'."
>>> help(hello_world)
Help on function hello_world in module __main__:

hello_world()
    Return the string 'Hello, world!'.

Although technically docstrings are code I wouldn't expect them to be counted as such — they're really Python's version of Rust's doc comments.

[XAMPPRocky]

@flother Technically correct is the best kind of correct 😛. I might consider adding a compile or runtime flag that would change the behavior with the default being the current. Since it is a common practice.

[flother]

Thanks for reconsidering this.

Just to needlessly ram home my point: take a peek at re, the regular expression module from Python's standard library, and you can see how prevalent docstrings are. Tokei counts 306 lines of code, cloc just 173 — with 31 and 164 comments respectively.

It's a shame it would be a breaking change, because I'd love to see Tokei take these into account by default. I suppose that suggests I'd be in favour of a compile flag instead of a runtime flag, but I think I'd be more comfortable on the fence on that one.

[XAMPPRocky]

@flother Well I think the best option, is to include it in #67 whenever I figure out how to do it. 😅