Implement component name completion inside html! macro for IntelliJ Rust plugin

[vlad20012]

Description

These changes bring completion of component names inside an html! macro invocation:

Peek.2022-11-21.16-38.mp4

The behavior of html macro has been leaved absolutely unchanged in the case of usual usage during compilation.

How to check it out

You need CLion or IntelliJ IDE with Rust plugin version not lower than 0.4.184 (which includes intellij-rust/intellij-rust#9711)

[dependencies.yew]
git = "https://github.com/vlad20012/yew.git"
branch = "intellij-rust-completion"

use yew::html;

pub struct MyComponent;

fn main() {
    let _ = html! {
        <div>
            </*caret*/>
        </div>
    };
}

Implementation details

When a user performs code completion, IntelliJ Rust duplicates the current file, inserts a dummy identifier at the current caret position and then expands a macro invocation at the current caret position. After that, it maps the caret position to the expansion (based on spans) and then performs code completion on the expansion.

If the macro expansion fails (expands to compile_err!(...)), which is usual when a user is typing, IntelliJ Rust can't provide any completion.

yew::html! expansion fails in any cases of invalid syntax. For example, unmatched tag/component names:

html! {
    <MyC/*caret*/>
        
    </MyComponent>
}

In such cases html! expands to compile_err!(...), hence the user is left without completion :(

This PR is based on the fact that since intellij-rust/intellij-rust#9711 IntelliJ Rust sets these environment variables when invoking a proc macro during completion:
RUST_IDE_PROC_MACRO_COMPLETION=1
RUST_IDE_PROC_MACRO_COMPLETION_DUMMY_IDENTIFIER=actual dummy identifier

So, I changed the behavior of the parser if RUST_IDE_PROC_MACRO_COMPLETION env is set. Namely:

1. Ignore unclosed tags
2. Ignore a close tag without the open tag
3. Ignore unmatched open/close tag names

Note that this behavior only works when the macro is invoked from an IDE during completion. When an IDE invokes a macro in other cases, it doesn't set these envs and hence usual parser works.

Also, with these env vars, it is possible to make fully custom completion. For example, it's possible complete standard tag names like div (but this PR does not include this).

Inspired by rust-lang/rust-analyzer#7402 (comment) (thanks to @ogoffart & @matklad)

[matklad]

https://matklad.github.io/2022/10/24/actions-permissions.html

[github-actions]

Visit the preview URL for this PR (updated for commit a219e2f):

https://yew-rs-api--pr2972-intellij-rust-comple-a5oa472j.web.app

_{(expires Thu, 26 Jan 2023 15:58:56 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

[WorldSEnder]

Interesting approach. Before reviewing, can you perhaps clarify

[...] inserts a dummy identifier at the current caret position and then expands a macro invocation at the current caret position. [...] yew::html! expansion fails in any cases of invalid syntax

html! {
    <MyC/*caret*/>
        
    </MyComponent>
}

Would this be expanded as

html! {
    <MyC INTELLIJ_DUMMY_SYMBOL>
        
    </MyComponent>
}

or

html! {
    <INTELLIJ_DUMMY_SYMBOL>
        
    </MyComponent>
}

---

In rust-lang/rust-analyzer#7402 (comment) you mention the possibility to communicate "valid" expansions back to intellisense. Is this sort of thing possible for the IntelliJ plugin?

[github-actions]

Benchmark - SSR

Yew Master

Benchmark	Round	Min (ms)	Max (ms)	Mean (ms)	Standard Deviation
Baseline	10	300.342	301.398	300.861	0.355
Hello World	10	634.268	635.789	634.869	0.482
Function Router	10	2191.039	2203.514	2197.332	3.424
Concurrent Task	10	1008.250	1009.159	1008.697	0.266

Pull Request

Benchmark	Round	Min (ms)	Max (ms)	Mean (ms)	Standard Deviation
Baseline	10	336.001	337.464	336.863	0.485
Hello World	10	637.754	641.444	638.732	1.218
Function Router	10	2199.979	2218.087	2209.871	4.941
Concurrent Task	10	1008.102	1010.088	1009.033	0.548

[github-actions]

Size Comparison

examples	master (KB)	pull request (KB)	diff (KB)	diff (%)
async_clock	106.604	106.608	+0.004	+0.004%
boids	170.788	170.783	-0.005	-0.003%
communication_child_to_parent	90.832	90.836	+0.004	+0.004%
communication_grandchild_with_grandparent	105.160	105.162	+0.002	+0.002%
communication_grandparent_to_grandchild	100.991	100.996	+0.005	+0.005%
communication_parent_to_child	87.929	87.926	-0.003	-0.003%
contexts	107.695	107.698	+0.003	+0.003%
counter	85.730	85.731	+0.001	+0.001%
counter_functional	86.040	86.042	+0.002	+0.002%
dyn_create_destroy_apps	88.514	88.513	-0.001	-0.001%
file_upload	100.191	100.190	-0.001	-0.001%
function_memory_game	164.900	164.893	-0.008	-0.005%
function_router	349.510	349.522	+0.013	+0.004%
function_todomvc	159.864	159.866	+0.002	+0.001%
futures	224.654	224.655	+0.001	+0.000%
game_of_life	106.513	106.514	+0.001	+0.001%
immutable	181.951	181.960	+0.009	+0.005%
inner_html	82.118	82.119	+0.001	+0.001%
js_callback	111.944	111.939	-0.005	-0.004%
keyed_list	196.296	196.297	+0.001	+0.000%
mount_point	85.356	85.358	+0.002	+0.002%
nested_list	113.392	113.391	-0.001	-0.001%
node_refs	93.370	93.372	+0.002	+0.002%
password_strength	1549.012	1549.014	+0.002	+0.000%
portals	96.565	96.563	-0.002	-0.002%
router	319.657	319.657	0	0.000%
simple_ssr	150.973	150.968	-0.005	-0.003%
ssr_router	394.690	394.671	-0.020	-0.005%
suspense	109.129	109.135	+0.006	+0.005%
timer	88.633	88.633	0	0.000%
todomvc	141.097	141.097	0	0.000%
two_apps	86.351	86.353	+0.002	+0.002%
web_worker_fib	151.906	151.907	+0.001	+0.001%
webgl	84.908	84.907	-0.001	-0.001%

✅ None of the examples has changed their size significantly.

[vlad20012]

@WorldSEnder

html! {
    <MyC/*caret*/>
        
    </MyComponent>
}

Guessed wrong :D
It will be expanded like this:

html! {
    <MyCINTELLIJ_DUMMY_SYMBOL>
        
    </MyComponent>
}

In rust-lang/rust-analyzer#7402 (comment) you mention the possibility to communicate "valid" expansions back to intellisense. Is this sort of thing possible for the IntelliJ plugin?

Of course! If a macro expands normally and correctly sets spans for output tokens, IntelliJ Rust looks through the macro call at the expansion and provide some IDE features there (like highlighting and, of course, completion). It the case of yew::html!, completion works in Rust code blocks (inside {} braces) right now without this PR.

[ranile]

Great to see this. There's something I would like to clarify: what about the closing tag? If you try using a component in JSX (in a .jsx or .tsx file in IntelliJ), it automatically completes the closing tag when > is typed. If a suggestion is selected, even the > is added automatically. What's the equivalent here (if any)?

Same goes for editing. If I start typing in opening tag, it automatically matches what's in the closing tag. In Rust, those two are separate Idents with their own spans (needed for error reporting). I assume that's not supported, right?

---

https://matklad.github.io/2022/10/24/actions-permissions.html

I didn't even know that option existed. @siku2 can you make the change?

[siku2]

@hamza1311 done!

[vlad20012]

@hamza1311, well, at the moment I don't see how such advanced features could be achieved using such an approach, unfortunately. We likely need a special IntelliJ plugin with yew integration.

[ranile]

Exactly. Writing plugins specifically for proc macros would need to be supported by IntelliJ Rust

[ogoffart]

@hamza1311

well, at the moment I don't see how such advanced features could be achieved using such an approach, unfortunately. We likely need a special IntelliJ plugin with yew integration.

I realize this is a bit off-topic for this issue in this repository, but the ideally, ide integration shouldn't not need to have special plugin for each possible proc-maco. What I was discussing with matklat that lead to rust-lang/rust-analyzer#7402 (comment) was specifically that the yew html! macro, and when run through an ide, would look for the MyCINTELLIJ_DUMMY_SYMBOL (or whatever its name is, probably something more neutral, such as like MyC__IDE_INTEGRATION_COMPLETION) and then could expand its macro to the list of available token, so the proc macro output would be like __ide_integration_completion![ "MyComponent", "MyController", "MyC" ]
and the yew's html macro implementation would also be able to recognize the torken in the closing tag ( </__IDE_INTEGRATION_COMPLETION> ) and reply with __ide_integration_completion![ "MyComponent"]

Of course, the whole protocol needs to be well defined so we can do completion items, highlight, gotodefinition, and so on.

But for sure the first thing we can do now as macro authors is to be a bit more resilient to errors in macro implementation, and still try to produce the most possible rust code before throwing a compilation_error.

[vlad20012]

@ogoffart The features that you describe can be done in the scope of this PR, I guess (even without a special __ide_integration_completion thing that an IDE have to understand).
But not the features that @hamza1311 explains in this comment, because these features are not about completion.

[vlad20012]

@ogoffart Let me explain. With this PR, when you invoke completion with such a source:

html! {
    <MyC/*caret*/>
}

the macro will see this input:

<MyCINTELLIJ_DUMMY_SYMBOL>

Since the macro understands it's invoked during a completion inside an IDE, it uses a less strict parser and eventually expands to something where MyCINTELLIJ_DUMMY_SYMBOL is used as a type:

{
    ...
    let _ = |_: MyCINTELLIJ_DUMMY_SYMBOL| {};
    ...
}

Of course, the macro sets correct span to the token in the output.
Then, the IDE sees the token in the output and performs completion in that context. This is why we can complete items that are out of scope of the macro:

You don't need __ide_integration_completion approach for this, and, furthermore, such behavior is unachievable with __ide_integration_completion approach because a macro can't know what names are defined outside of the macro (MyComponent1, MyComponent2) and hence it can't provide completion for them.

Let's take a look at a more complex example with auto-import:

Peek.2022-11-23.14-58.mp4

This is unachievable with __ide_integration_completion not only because the macro don't know anything about MyComponent1 but also because applying completion must lead to the import insertion outside of the macro!

Such tricks works well with closing tag just because they're correctly mapped with expansion as well:

What is not done in this PR is completion for standard html elements:

html! {
    </*caret*/>
}

You'd likely want div, span and other tags completion here. This is where __ide_integration_completion approach may seem useful, but actually it looks like there is a better approach. If a macro understands that it's invoked during a completion inside an IDE (it is) and that a dummy identifier is placed to the tag name position (it can figure it out by seeking for dummy identifier name), it can expand to something like this:

{
    struct div;
    struct span;
    ...
    let _ = |_: MyCINTELLIJ_DUMMY_SYMBOL| {};
    ...
}

And the IDE will pick these definitions up and show them in the completion list just because these names are now in the scope. No need for special __ide_integration_completion syntax! This is not done in this PR, but it seems easy achievable with the same approach.

---

What @hamza1311 suggests is automatic insertion of the corresponding closing tag when a user types >:

Peek.2022-11-23.15-11.mp4

Or automatically adjust a closing tag when editing the open one:

Peek.2022-11-23.15-14.mp4

Completion is just not involved here. So I think the only way to fulfill these feature is implementing a special plugin

[ranile]

Would it make sense to move is_ide_completion call to root of the macro where syn::Error is converted to compile_err!? That way all the cases of compilation failure will covered, while avoiding adding the if conditional at every error return

[vlad20012]

Hmm, but what can we do with syn::Error except converting it to compile_err!? Here we skip returning with an error and continuing the parsing. If we return with an error, the parsing is abandoned, so we can do nothing except returning compile_err!, I guess

[ranile]

The macro needs to expand to actual results (even if incomplete)? I was thinking maybe we could return an empty TokenStream and such so the output doesn't return a compile error. Html::default() is valid return type for every macro (the return type must be Html - more accurately VNode - for it to pass type checking) . Perhaps that can be returned?

[vlad20012]

The macro needs to expand to actual results (even if incomplete)?

Yes, definitely. If not for this, then no changes in yew would be required at all.

[ranile]

Sorry, it took me so long to properly try this out. It works well. Great work!

I did run into issues with error reporting though. See intellij-rust/intellij-rust#9898

Side note: I'm not sure how IntelliJ is able to figure out the prop definitions for components with props, considering it's impossible for the macro to know about the props struct. It would be great if it could auto complete too. You can try it out by jumping to definition of seed here:
https://github.com/vlad20012/yew/blob/5355b65ff5f9747cbad801d4b337a5ac7a94d0f4/examples/function_router/src/app.rs#L92

[vlad20012]

I'm sorry for being so late with the response. I was on vacation without a PC 😅

I'm not sure how IntelliJ is able to figure out the prop definitions for components with props, considering it's impossible for the macro to know about the props struct. It would be great if it could auto complete too.

I'm going to try making it in a separate PR right after this. I think in this case the macro should expand to a completely different code that puts the dummy identifier in such specific context where the completion will work as expected.

struct FooComponent {
    attribute: i32
}

...

html! {
   <FooComponent attr/*caret*/>
}

In this case, we could expand the macro to

let _ = FooComponent {
    attr/*caret*/: ()
}

Then, when an IDE will invoke completion in such expanded fragment, it will show only fields of the FooComponent struct in the list

[ranile]

Then, when an IDE will invoke completion in such expanded fragment, it will show only fields of the FooComponent struct in the list

FYI: this would need to show fields of FooComponent::Properties. Using FooComponent is wrong as it's not supposed to be constructed directly, instead Component::create function is used

[futursolo]

Thank you for the contribution.

I am going to follow @hamza1311's review and approve this as I do not use IntelliJ.
CI all passes so I assume it will not have impacts when this feature is not enabled.