Implement complete_request

[SylvainCorlay]

Xeus-cling comprises a (fragile) implementation of auto-completion at https://github.com/jupyter-xeus/xeus-cling/blob/0.15.0/src/xinterpreter.cpp#L204-L238.

A more robust version should be implemented for xeus-cpp.

[jkshtj]

@SylvainCorlay I talked to @vgvassilev about picking up this issue and have been looking into it. I had a few questions that you could probably answer.

1. What about the xeus-cling implementation is fragile? It looks like it's calling a corresponding code-completion method on the interpreter and then formatting results, so it's not obvious to me outright what needs to be improved.

2. Is this going to involve any work on the clang-repl side?

[vgvassilev]

@jkshtj, https://discourse.llvm.org/t/rfc-flexible-lexer-buffering-for-handling-incomplete-input-in-interactive-c-c/64180/9 describes our plan and progress for how to implement continuation in clang-repl. However, the patches have not landed. I think we want some temporary solution where we could do rough brace counting to decide if something is complete or not.

A slightly more sophisticated implementation would involve creating a clang::Lexer to pre-lex the tokens as that might expand macro definitions but I do not think that would be necessary at that stage.

[jkshtj]

@jkshtj, https://discourse.llvm.org/t/rfc-flexible-lexer-buffering-for-handling-incomplete-input-in-interactive-c-c/64180/9 describes our plan and progress for how to implement continuation in clang-repl. However, the patches have not landed. I think we want some temporary solution where we could do rough brace counting to decide if something is complete or not.

A slightly more sophisticated implementation would involve creating a clang::Lexer to pre-lex the tokens as that might expand macro definitions but I do not think that would be necessary at that stage.

@vgvassilev perhaps I'm mistaken, but was your comment more so related to the is_complete_request implementation?

[vgvassilev]

Oh, yes, apologies. I’ve always meant is_complete_request. The code complete needs quite some support from the interpreter side. Probably we will have a GSoC contributor who will work on that on the clang-repl side and then we could implement the hooks here and resolve that issue. Do you mind working on the other one first?

[jkshtj]

Absolutely.

[jkshtj]

@vgvassilev just a quick heartbeat since I haven't posted much here.

I'm working on the solution to implement is_complete_request using the clang::Lexer. I've had basic setup done for while -- enough to have the clang::Lexer instance do the lexing for me, but I've been running into some issues with getting the lexer to expand macros while lexing.

I've been trying to get help to unblock myself through the Clang discord, but have recently also found some text that might be helpful in getting familiar with Clang and hacking with the lexer/preprocessor part.

Let me know if you think there are some other good resources that I could also consult or if I could be reaching out to specific individuals for faster help.

[vgvassilev]

We probably need an API to llvm/llvm-project@35b366a through CppInterOp.