commands with optional arguments do not get parsed correctly

[underscore-j]

If a command has no optional arguments, then an arbitrary number of arguments given in curly braces will be treated as children of the command.

However, if the command has optional arguments (in []), then these, as well as any other arguments that come after, are not considered children of the command.

This appears to be because the rule for generic_command does not allow for optional arguments.

[underscore-j]

Of course, not everything given in square brackets after a command is actually an optional argument (this is especially true in math mode).
However, the same is also true for curly braces.

Perhaps it would be reasonable to square brackets as arguments if they are followed by a mandatory argument in curly braces? This should cover most uses of optional commands, without interfering (too much) with other uses of square brackets.

[pokey]

Not sure if this is the same issue, but this code:

\begin{lstlisting}[language=Python]
    aaa
\end{lstlisting}

parses as follows:

source_file [0:0-2:16]
  listing_environment [0:0-2:16]
    begin: begin [0:0-0:18]
      command: "\begin" [0:0-0:6]
      name: curly_group_text [0:6-0:18]
        "{" [0:6-0:7]
        text: text [0:7-0:17]
          word: word [0:7-0:17]
        "}" [0:17-0:18]
    code: source_code [0:18-2:0]
    end: end [2:0-2:16]
      command: "\end" [2:0-2:4]
      name: curly_group_text [2:4-2:16]
        "{" [2:4-2:5]
        text: text [2:5-2:15]
          word: word [2:5-2:15]
        "}" [2:15-2:16]

In particular, notice that the source_code node in the middle includes the [language=Python]

[jdujava]

Of course, not everything given in square brackets after a command is actually an optional argument (this is especially true in math mode).

Even though often in math mode there can be \foo[bar], where [bar] isn't optional argument of \foo as far as LaTeX is concerned, it is usually som kind of argument from mathematical point of view.

Thus, in my opinion the best result would be to always recognize square bracket group behind a command name as its argument. Thoughts?