Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a way to turn a regular Parser into an IterParser so that chumsky can be used as a stream transformer #425

Open
zesterer opened this issue May 16, 2023 · 2 comments
Labels
api A problem with the design of an API feature

Comments

@zesterer
Copy link
Owner

zesterer commented May 16, 2023

Example APIs:

Parser::into_iter_parser(self) -> impl IterParser where O: IntoIterator
IterParser::chain(self, impl IterParser) -> impl IterParser
IterParser::repeated(self) -> impl IterParser // TODO: Conflict with <Repeated as Parser>::repeated?
@zesterer zesterer added the api A problem with the design of an API feature label May 16, 2023
@zesterer
Copy link
Owner Author

zesterer commented May 16, 2023

Existential crisis time: should IterParser and Parser be generalised? We could have:

trait Parser<I, O> {
    fn repeated() -> ...
    fn delimited_by(_, _) -> ...
    fn then(_) -> ...
    fn map(_) -> ...
    fn validate(_) -> ...
}

and then ValueParser and IterParser that extend this core trait for the following cases:

  • ValueParser: for parsers with exactly one output (an AST node, say)
  • IterParser: for parsers with any number of outputs (lexers, stream transformers, etc.)
trait ValueParser<I, O> {
    fn then_with_ctx(_) -> ...

    fn parse(_) -> Result<O, _>
    fn check(_) -> Result<O, _>
}
trait IterParser<I, O> {
    fn count() -> ...
    fn enumerate() -> ...
    fn collect() -> ...
    fn flatten() -> ...

    fn parse_iter(_) -> impl Iterator<Item = Result<O, _>>
    fn check_iter(_) -> impl Iterator<Item = Result<(), _>>
}

@cosmicexplorer
Copy link

Thinking about #399 (comment), it seems like we may be able to sidestep the issues with generalizing ParserIter by only enabling it for parsers which do not perform backtracking (which I believe should be true of most tokenizers). This doesn't address the larger issue, but it may be able to unblock many use cases for pipelined parsing as per #399. Would that be an acceptable interim solution?

@zesterer zesterer mentioned this issue Oct 10, 2023
35 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api A problem with the design of an API feature
Projects
None yet
Development

No branches or pull requests

2 participants