Better GitHub Code Search for R with tree-sitter-r

[DavisVaughan]

Select Topic Area

Question

Body

Hi GitHub Code Search team!

I'm an employee of Posit, best known for RStudio and our collection of R packages like dplyr, tidyr, and the rest of the tidyverse.

We'd really love to see improved code search for R, like how in these two C examples:

• You can click on a function call and it will tell you the definition site within the project
• You can search for a function name and it will bring up the definition site within the project

We've written a tree-sitter grammar that we are pretty happy with. It's current at the next branch but that will be moving to main soon. https://github.com/r-lib/tree-sitter-r/tree/next

I imagine I probably need to write a tags.scm for you guys? i.e. https://tree-sitter.github.io/tree-sitter/code-navigation-systems. We do have highlights.scm already.

Is there anything else you need? Once we have tags.scm, what next?

And in general I'm curious about the process of how things are kept up to date. Like, if we make an update or improvement in tree-sitter-r, how could that flow into your Code Search? Would we manually ping you about it to get things synced up?

I've seen https://github.com/orgs/community/discussions/10934 by @dcreager and @colinwm, are there any other community guides that have been developed since then?

Thanks in advance!

[look]

@DavisVaughan thanks for the suggestion. We'd be happy to add code navigation support for R. You are correct that having a tags.scm will make this much easier for us. Once that is done and the code is in your mainline, please let us know. The next step for us is to take your tags.scm and adapt it to our system (our pipeline requires certain names for the language elements returned by the query). Then the next time we rebuild our code search index, R symbols will be included.

[DavisVaughan]

Not too late! It looks like top level Search now includes R functions!! That is one of the major things I was going to show, so thank you so much!

The other thing I was potentially going to show (but I can leave it out if not possible to get it up in time) is clicking on an R function reference, and then using the Symbols pane (which pops up for other languages) to jump straight to definition, but that still isn't working in vetiver-r. Any ideas? (i.e. when I click on vetiver_write_plumber() in the video below, was hoping the symbols pane would pop up). Either way, thank you so much for the extra time you're putting in here!

Screen.Recording.2024-07-08.at.10.06.34.PM.mov

[DavisVaughan]

FYI my talk is actually this Thursday

[look]

@DavisVaughan I looked into this a little. I think there are a couple of problems. In the file you demo in your video, clicking works for some items but not others (for example, it works pretty well for function arguments). This is because the calls don't have the right CSS class to be considered a symbol. I think we need to upgrade our syntax highlighter to fix that.

I also found that functions with . in the name don't work well (for example, vetiver_create_description.default in prepare.R). This is because the syntax highlighter treats the whole text as a symbol, and the regex that we use to determine a symbol excludes symbols with . in them. I'm not sure what the correct behavior in this case is.

We will work on improving this but unfortunately it won't be ready for your talk tomorrow.

[DavisVaughan]

clicking works for some items but not others (for example, it works pretty well for function arguments). This is because the calls don't have the right CSS class to be considered a symbol. I think we need to upgrade our syntax highlighter to fix that.

Let me know if there is anything I can do to help!

---

the regex that we use to determine a symbol excludes symbols with . in them. I'm not sure what the correct behavior in this case is.

In R, in general you can have a . in the name of a variable, like

> x.y <- 1
> x.y
[1] 1

In the case of

vetiver_create_description.default <- function(model) {

this is a special case of using the . where this is a default S3 method for the generic function vetiver_create_description(). Similarly you could define vetiver_create_description.foo <- function(model) { as the S3 method for vetiver_create_description() specific to model objects of class "foo".

That's only considered an S3 method because vetiver_create_description() itself is defined as a generic function. In general you can even use a . in a function name, like:

data.frame <- function(..., stringsAsFactors = FALSE) {

which is a function called data.frame(), not an S3 method for data() on objects of class "frame".

This is obviously a confusing part of R, and I think it is fairly hard to figure out statically, so I think what I would do would be to unconditionally allow . as part of a symbol name, so data.frame and vetiver_create_description.default would be considered distinct symbols.

---

unfortunately it won't be ready for your talk tomorrow

That is ok! I have the videos I need for the talk.

But just FYI, the day after you told me that search for vetiver-r was working, it...stopped working 😆 . Luckily I snagged a video before it stopped working, but here it is today (when it worked it would show a function called vetiver_model that you can jump to), not sure what's up with that:

Screen.Recording.2024-07-11.at.9.58.45.AM.mov

[DavisVaughan]

@look searching for functions using the search box has been working beautifully and looks to be rolled out across all R projects on GitHub now, thanks so much for your help!

My talk went very well, and people really really loved hearing about this. It even got a round of applause in the middle of the talk!

Is there any update regarding this?

This is because the calls don't have the right CSS class to be considered a symbol. I think we need to upgrade our syntax highlighter to fix that.

Would love to be able to click on a reference to a function call and jump straight to the definition in the project