Parsing incomplete/invalid inputs

[FSMaxB]

It would be useful to have the parser return parse trees for incomplete/invalid inputs like (1+1.

This isn't desirable as a default behaviour but as an optional behaviour that can be turned on and of.

Use case: Showing live previews of formulas and calculations from user input.

Is it possible to implement without having to rewrite the entire parser? And if so, is it a feature that fits in the scope of mathjs?

[josdejong]

Interesting idea. This could certainly be useful. Maybe instead of throwing an error when a bracket misses or the right hand side of an operator misses, the parser should add the error message to a separate error list (message and character index), and continue as good as possible. It may not be that hard to implement actually.

[FSMaxB]

ParenthesisNode would have to support incomplete parentheses (like only opening or only closing parentheses.

[josdejong]

Yes, and operators need to support a missing right hand side.

[hluedeke]

+1

This functionality would be GREAT when coupling with Tex for user input

[gwhitney]

See also the ideas in #2289.

[sancarn]

There is an implementation for a different parser mentioned in that thread, not sure how easy porting would be.

[gwhitney]

Right -- mathjs currently has a monolithic handwritten parser, as compared to the architecture you reference, which is more modular and distributed. That's a real plus -- Jos and I have discussed the fact that in a future more modular mathjs, it will be necessary for the parser to be extensible as additional modules are loaded. For example, you don't want to have a syntax like [1,2;4,5] for matrices if you haven't loaded any matrix modules, and conversely, if a Tuple module is added, it might want to add a read syntax for tuples like say <1, 2, 3> or something. So hard to say how difficult/easy it might be to incorporate a concept from a very differently organized parser into mathjs. But it's definitely something to keep in mind as mathjs faces potentially significant refactors to its parsing.

I like the "pros and cons" list in the README of the project you reference. In fact, I was just musing last night about how one might handle precedence in a modular setting. I thought about each operator specifying others it should have greater, less, or equal precedence to. But I worried about what happens as the collection of operators grows: some operators might not "know about" each other, and might list all the same comparative operators, but not mention each other, and then how would you know how to order their precedence? And I see that you came to regret the decision to try to specify precedence that way, so that definitely helps push my mindset against that approach. I am actually not sure what a good alternative is -- maybe just allowing precedence to be a floating point number between 0 and 1 inclusive, and hope that the assemblage can evolve toward pretty good precedences? Not sure what other approaches are out there... In a quick search, the only schemes I saw were numeric, either integer or floating point, and by order of declaration. I think the latter is likely not feasible for an extensible scheme where order of module loading might be difficult to control precisely.

[sancarn]

Discussed with TarVK. The best approach to precedence may just be a manual one:

createLanguage([
    [VariableFactory, GroupFactory],
    [NotFactory],
    [AndFactory],
    [OrFactory],
    [ImpliesFactory, BiImpliesFactory],
]);

Requires more boilerplate from the user when they want to customize things, but it's at least very direct and doesn't give these difficulties as described. Very much "hard coded" and means the user does kind of have to understand what precedence means, but module devs can provide a "base/example setup" of how precedence of modules should be ordered. It'd at least be a simple solution, and doesn't really have any pitfalls other than user friendliness.

[gwhitney]

The various concepts floated here and in #2289 sound a fair amount like the things that the Language Server Protocol provides; possibly creating a Language Server for the mathjs-expression-language (which I think still needs a name) is really the direction to think about in this era?

[dvd101x]

While doing some experiments defining mathjs mode for a text editor, this project was encountered for parsing text https://lezer.codemirror.net/
https://marijnhaverbeke.nl/blog/lezer.html

It seems that some of its features could be useful for the mathjs parser.

[gwhitney]

I agree many features there parallel suggestions in this discussion. I worry slightly that although mathjs certainly has to have an embedded parser for the expression language it supports, "editor services" or a Language Server for that expression language are a bit out of scope for the core of mathjs and should perhaps be in a companion module. Clearly that companion module will need a parser as well -- but are they the same parser, or perhaps different parsers produced from a common specification? I ask this in part because in one vision of the future of mathjs, the embedded parser should be generated dynamically as modules are loaded with new operations/types that come with their corresponding syntax. Lezer doesn't seem to be ideal for that: although it can generate a parser in memory, it's more designed for generating a parser module that is then later loaded, and it operates from one big specification string -- there's no way to add more rules to a generated parser as far as I can see. And Lezer has limitations on the structure of the syntax trees it produces (the documentation even describes them as non-abstract syntax trees).

Maybe the embedded parser of mathjs could be something more suited for building incrementally and for analyzing expressions and executing them, but wouldn't support error recovery and/or incremental re-parsing. And then maybe a fully-constructed mathjs instance could also emit a Lezer grammar specification (that would also be built along the way as modules are loaded), which could then be used by the companion module for fault-tolerant parsing for editors or other expression manipulation. Just thinking aloud.

[josdejong]

There are also other parser generators, we could do some experiments to see if they are usable for mathjs.