gh-104683: Argument clinic: remove some unnecessary uses of self.next() in the DSLParser

[AlexWaygood]

Many state_foo methods on the DSLParser class in clinic.py are called only indirectly via the next() method:

cpython/Tools/clinic/clinic.py

Lines 4645 to 4652 in 407d7fd

	def next(
	self,
	state: StateKeeper,
	line: str | None = None
	) -> None:
	self.state = state
	if line is not None:
	self.state(line)

The reason for this is that the DSLParser parses argument-clinic input a line at a time; but state often persists across lines. (Example: a docstring can span multiple lines; by saving the "state" at the end of one line, the DSLParser is able to resume parsing with the same state when it begins parsing the next line, and therefore knows that it's in the middle of a docstring.)

However, there are two state_foo methods on the DSLParser class that are arguably badly named, and shouldn't be called via self.next(), even though they both currently are:

• state_modulename_name
• state_parameter_docstring_start

Both of these are examples of states that are impossible to span multiple lines. A modulename declaration cannot span multiple lines; nor can the start of a parameter docstring. As such, calling these methods via self.next() is needless indirection, and can be removed.

It is provable that both of these states cannot span multiple lines, due to the fact that neither state_modulename_name nor state_parameter_docstring_start ever exit early before setting self.state to something new (by calling self.next).

• Issue: Modernise code in Tools/clinic/ #104683

[AlexWaygood]

I renamed this function to distinguish it from the state_foo functions, all of which must only ever be called indirectly via self.next()

[erlend-aasland]

What do you think of parse_function_declaration?

Suggested change

	def parse_modulename_name(self, line: str) -> None:
	def parse_function_declaration(self, line: str) -> None:

[erlend-aasland]

I'm not sure (yet) this is a step in the right direction. By consolidating stages in the state machine, debugging it becomes harder. For example, if I add this debug print:

diff --git a/Tools/clinic/clinic.py b/Tools/clinic/clinic.py
index 6c5a2c7c85..b9d47d2e0e 100755
--- a/Tools/clinic/clinic.py
+++ b/Tools/clinic/clinic.py
@@ -4652,6 +4652,7 @@ def next(
             line: str | None = None
     ) -> None:
         self.state = state
+        print(f"--> {state.__name__:25} {line=!r}")
         if line is not None:
             self.state(line)
 

I get this:

$ cat test.c
/*[clinic input]
module m
m.func
    a: int
[clinic start generated code]*/

static PyObject *
m_func_impl(PyObject *module, int a)
/*[clinic end generated code: output=188ac0e0fa832273 input=e686625fcf7cad0e]*/
$ python3.12 Tools/clinic/clinic.py test.c
--> state_modulename_name     line='m.func'
--> state_parameters_start    line=None
--> state_parameter           line='    a: int'

However, with this PR, I now get:

$ python3.12 Tools/clinic/clinic.py test.c
--> state_parameters_start    line=None
--> state_parameter           line='    a: int'

[AlexWaygood]

Right, that's a very good point, and I think exposes that my mental model of the state machine here was subtly wrong. I think there may be another solution to the "But wait, these branches are unreachable!" problem I came up against in Argument-Clinic#14; I'll have a play around.