Enable a tree-sitter playground inside Emacs

[shackra]

This PR enables developers to execute tree-sitter queries and see what parts of source code they identify in a target buffer by highlighting. This extends the available code for debugging tree-sitter-query and friends.

[shackra]

Hope this doesn't come as unnecessary. This PR is also helping me grasp the available bindings to tackle #4

[ubolonton]

I think this is basically the start of a major mode for the query language, so it can be named tree-sitter-query-mode.

[shackra]

was working on this, doing some testing, and the query I was passing made the bindings to panic:

Debugger entered--Lisp error: (rust-panic "called `Result::unwrap()` on an `Err` value: Synta...")
  ts--make-query(#<user-ptr ptr=0x556817714970 finalizer=0x7f1ddb96ab00> "91\n40\n99\n97\n108\n108\n95\n101\n120\n112\n114\n101\n115\n115...")
  (let ((source (cond ((sequencep patterns) (mapconcat #'(lambda ... ...) patterns "\n")) ((stringp patterns) patterns) (t (format "%S" patterns))))) (ts--make-query language source))
  ts-make-query(#<user-ptr ptr=0x556817714970 finalizer=0x7f1ddb96ab00> "[(call_expression function: (identifier) @function...")
  (let* ((query (ts-make-query tree-sitter-language patterns)) (root-node (ts-root-node tree-sitter-tree)) (matches (ts-query-captures query root-node nil nil))) (if (> (length matches) 0) (seq-do #'(lambda (match) (tree-sitter-query--highlight-node match)) matches) (message "[ERR] no matches found or invalid query")))
  (save-current-buffer (set-buffer tree-sitter-query--target-buffer) (remove-text-properties (point-min) (point-max) '(font-lock-face tree-sitter-query-match)) (let* ((query (ts-make-query tree-sitter-language patterns)) (root-node (ts-root-node tree-sitter-tree)) (matches (ts-query-captures query root-node nil nil))) (if (> (length matches) 0) (seq-do #'(lambda (match) (tree-sitter-query--highlight-node match)) matches) (message "[ERR] no matches found or invalid query"))))
  tree-sitter-query--eval-query("[(call_expression function: (identifier) @function...")
  (condition-case err (tree-sitter-query--eval-query pattern) ((debug error) (message "Error: %S" err) nil))
  (let ((pattern (buffer-substring-no-properties (point-min) (point-max)))) (condition-case err (tree-sitter-query--eval-query pattern) ((debug error) (message "Error: %S" err) nil)))
  (progn (let ((pattern (buffer-substring-no-properties (point-min) (point-max)))) (condition-case err (tree-sitter-query--eval-query pattern) ((debug error) (message "Error: %S" err) nil))))
  (if (or (eq this-command 'self-insert-command) (eq this-command 'insert-buffer)) (progn (let ((pattern (buffer-substring-no-properties (point-min) (point-max)))) (condition-case err (tree-sitter-query--eval-query pattern) ((debug error) (message "Error: %S" err) nil)))))
  tree-sitter-query--after-post-command()

maybe I should rebase and try again.

[shackra]

okay, did that and the error is almost the same except that I can see what query I was using:

Debugger entered--Lisp error: (rust-panic "called `Result::unwrap()` on an `Err` value: Syntax(1, \"[(call_expression function: (identifier) @function)]\\n^\")")
  ts--make-query(#<user-ptr ptr=0x7f55e37fe1e0 finalizer=0x7f55e161ac90> "[(call_expression function: (identifier) @function)]")
  (let ((source (cond ((stringp patterns) patterns) ((sequencep patterns) (mapconcat #'(lambda (p) (format "%S" p)) patterns "\n")) (t (format "%S" patterns))))) (ts--make-query language source))
  ts-make-query(#<user-ptr ptr=0x7f55e37fe1e0 finalizer=0x7f55e161ac90> "[(call_expression function: (identifier) @function)]")
  (let* ((query (ts-make-query tree-sitter-language patterns)) (root-node (ts-root-node tree-sitter-tree)) (matches (ts-query-captures query root-node nil nil))) (if (> (length matches) 0) (seq-do #'(lambda (match) (tree-sitter-query--highlight-node (elt match 1))) matches) (message "[ERR] no matches found or invalid query")))
  (save-current-buffer (set-buffer tree-sitter-query--target-buffer) (remove-text-properties (point-min) (point-max) '(face tree-sitter-query-match)) (let* ((query (ts-make-query tree-sitter-language patterns)) (root-node (ts-root-node tree-sitter-tree)) (matches (ts-query-captures query root-node nil nil))) (if (> (length matches) 0) (seq-do #'(lambda (match) (tree-sitter-query--highlight-node (elt match 1))) matches) (message "[ERR] no matches found or invalid query"))))
  tree-sitter-query--eval-query("[(call_expression function: (identifier) @function)]")
  (condition-case err (tree-sitter-query--eval-query pattern) ((debug error) (message "Error: %S" err) nil))
  (let ((pattern (buffer-substring-no-properties (point-min) (point-max)))) (condition-case err (tree-sitter-query--eval-query pattern) ((debug error) (message "Error: %S" err) nil)))
  (progn (let ((pattern (buffer-substring-no-properties (point-min) (point-max)))) (condition-case err (tree-sitter-query--eval-query pattern) ((debug error) (message "Error: %S" err) nil))))
  (if (or (eq this-command 'self-insert-command) (eq this-command 'insert-buffer)) (progn (let ((pattern (buffer-substring-no-properties (point-min) (point-max)))) (condition-case err (tree-sitter-query--eval-query pattern) ((debug error) (message "Error: %S" err) nil)))))
  tree-sitter-query--after-post-command()

I don't really understand why the bindings fail that critically 🤔

[ubolonton]

That means the query has a syntax error.

;; [] are just the container for the patterns.
[(call_expression function: (identifier) @function)
 (impl_item type: (*) @type)]

;; They themselves should not be in the string representation.
"(call_expression function: (identifier) @function)
 (impl_item type: (*) @type)"

The error reporting is bad, as described in #22.

[shackra]

so avoiding the square brackets as container for the query should suffice to workaround this issue, interesting.

[ubolonton]

Just a heads-up: I reorganized the directory structure a bit, moving all Lisp code to lisp/.

I also reworked the types in #26.

[shackra]

@ubolonton good mornin'

Can you review my tree-sitter-query--eval-query function? suddenly tree-sitter-query--highlight-node is being passed something that triggers an exception:

Debugger entered--Lisp error: (wrong-type-argument listp #<user-ptr ptr=0x56555c81b680 finalizer=0x7efbf1e83950>)
  elt(("name" . #<user-ptr ptr=0x56555c81b680 finalizer=0x7efbf1e83950>) 1)
  (tree-sitter-query--highlight-node (elt match 1))
  (closure ((matches . [("name" . #<user-ptr ptr=0x56555c81b680 finalizer=0x7efbf1e83950>)]) (root-node . #<user-ptr ptr=0x5655658aef10 finalizer=0x7efbf1e83950>) (query . #<user-ptr ptr=0x56555c8d1950 finalizer=0x7efbf1e83980>) (patterns . "(function_item (identifier) @name)") t) (match) (tree-sitter-query--highlight-node (elt match 1)))(("name" . #<user-ptr ptr=0x56555c81b680 finalizer=0x7efbf1e83950>))
  mapc((closure ((matches . [("name" . #<user-ptr ptr=0x56555c81b680 finalizer=0x7efbf1e83950>)]) (root-node . #<user-ptr ptr=0x5655658aef10 finalizer=0x7efbf1e83950>) (query . #<user-ptr ptr=0x56555c8d1950 finalizer=0x7efbf1e83980>) (patterns . "(function_item (identifier) @name)") t) (match) (tree-sitter-query--highlight-node (elt match 1))) [("name" . #<user-ptr ptr=0x56555c81b680 finalizer=0x7efbf1e83950>)])
  seq-do((closure ((matches . [("name" . #<user-ptr ptr=0x56555c81b680 finalizer=0x7efbf1e83950>)]) (root-node . #<user-ptr ptr=0x5655658aef10 finalizer=0x7efbf1e83950>) (query . #<user-ptr ptr=0x56555c8d1950 finalizer=0x7efbf1e83980>) (patterns . "(function_item (identifier) @name)") t) (match) (tree-sitter-query--highlight-node (elt match 1))) [("name" . #<user-ptr ptr=0x56555c81b680 finalizer=0x7efbf1e83950>)])
  (if (> (length matches) 0) (seq-do #'(lambda (match) (tree-sitter-query--highlight-node (elt match 1))) matches) (message "[ERR] no matches found or invalid query"))
  (let* ((query (ts-make-query tree-sitter-language patterns)) (root-node (ts-root-node tree-sitter-tree)) (matches (ts-query-captures query root-node nil nil))) (if (> (length matches) 0) (seq-do #'(lambda (match) (tree-sitter-query--highlight-node (elt match 1))) matches) (message "[ERR] no matches found or invalid query")))
  (save-current-buffer (set-buffer tree-sitter-query--target-buffer) (remove-text-properties (point-min) (point-max) '(face tree-sitter-query-match)) (let* ((query (ts-make-query tree-sitter-language patterns)) (root-node (ts-root-node tree-sitter-tree)) (matches (ts-query-captures query root-node nil nil))) (if (> (length matches) 0) (seq-do #'(lambda (match) (tree-sitter-query--highlight-node (elt match 1))) matches) (message "[ERR] no matches found or invalid query"))))
  tree-sitter-query--eval-query("(function_item (identifier) @name)")
  (condition-case err (tree-sitter-query--eval-query pattern) ((debug error) (message "Error: %S" err) nil))
  (let ((pattern (buffer-substring-no-properties (point-min) (point-max)))) (condition-case err (tree-sitter-query--eval-query pattern) ((debug error) (message "Error: %S" err) nil)))
  tree-sitter-query--after-change(1 35 0)
  insert-for-yank-1("(function_item (identifier) @name)")
  insert-for-yank("(function_item (identifier) @name)")
  yank-pop(0)
  counsel-yank-pop-action("(function_item (identifier) @name)")
  #f(compiled-function (x) #<bytecode -0x1d98955800ac363>)("(function_item (identifier) @name)")
  ivy-call()
  #f(compiled-function (arg1 arg2 &rest rest) "Read a string in the minibuffer, with completion.\n\nPROMPT is a string, normally ending in a colon and a space.\n`ivy-count-format' is prepended to PROMPT during completion.\n\nCOLLECTION is either a list of strings, a function, an alist, or\na hash table, supplied for `minibuffer-completion-table'.\n\nPREDICATE is applied to filter out the COLLECTION immediately.\nThis argument is for compatibility with `completing-read'.\n\nWhen REQUIRE-MATCH is non-nil, only members of COLLECTION can be\nselected.\n\nIf INITIAL-INPUT is non-nil, then insert that input in the\nminibuffer initially.\n\nHISTORY is a name of a variable to hold the completion session\nhistory.\n\nKEYMAP is composed with `ivy-minibuffer-map'.\n\nPRESELECT, when non-nil, determines which one of the candidates\nmatching INITIAL-INPUT to select initially.  An integer stands\nfor the position of the desired candidate in the collection,\ncounting from zero.  Otherwise, use the first occurrence of\nPRESELECT in the collection.  Comparison is first done with\n`equal'.  If that fails, and when applicable, match PRESELECT as\na regular expression.\n\nDEF is for compatibility with `completing-read'.\n\nUPDATE-FN is called each time the candidate list is re-displayed.\n\nWhen SORT is non-nil, `ivy-sort-functions-alist' determines how\nto sort candidates before displaying them.\n\nACTION is a function to call after selecting a candidate.\nIt takes one argument, the selected candidate. If COLLECTION is\nan alist, the argument is a cons cell, otherwise it's a string.\n\nMULTI-ACTION, when non-nil, is called instead of ACTION when\nthere are marked candidates. It takes the list of candidates as\nits only argument. When it's nil, ACTION is called on each marked\ncandidate.\n\nUNWIND is a function of no arguments to call before exiting.\n\nRE-BUILDER is a function transforming input text into a regex\npattern.\n\nMATCHER is a function which can override how candidates are\nfiltered based on user input.  It takes a regex pattern and a\nlist of candidates, and returns the list of matching candidates.\n\nDYNAMIC-COLLECTION is a boolean specifying whether the list of\ncandidates is updated after each input by calling COLLECTION.\n\nEXTRA-PROPS can be used to store collection-specific\nsession-specific data.\n\nCALLER is a symbol to uniquely identify the caller to `ivy-read'.\nIt is used, along with COLLECTION, to determine which\ncustomizations apply to the current completion session." #<bytecode 0x777e6436501b47>)("kill-ring: " ("(function_item (identifier) @name)" #("(add-to-list 'load-path \"/home/jorge/code/emacs-tr..." 0 1 (face (rainbow-delimiters-depth-1-face) fontified t) 1 24 (fontified t) 24 65 (face font-lock-string-face fontified t) 65 66 (face (rainbow-delimiters-depth-1-face) fontified t)) #("(add-to-list 'load-path \\\"$(PWD)/lisp\\\")" 0 28 (face (makefile-shell font-lock-string-face) fontified t) 28 31 (face (font-lock-variable-name-face makefile-shell font-lock-string-face) fontified t) 31 40 (face (makefile-shell font-lock-string-face) fontified t)) "(require 'tree-sitter-langs)" #("require-language" 0 16 (fontified t)) "https://www.youtube.com/watch?v=AQpIY5YCWys") :require-match t :preselect #("(add-to-list 'load-path \"/home/jorge/code/emacs-tr..." 0 1 (face (rainbow-delimiters-depth-1-face) fontified t) 1 24 (fontified t) 24 65 (face font-lock-string-face fontified t) 65 66 (face (rainbow-delimiters-depth-1-face) fontified t)) :action counsel-yank-pop-action :caller counsel-yank-pop :sort t)
  apply(#f(compiled-function (arg1 arg2 &rest rest) "Read a string in the minibuffer, with completion.\n\nPROMPT is a string, normally ending in a colon and a space.\n`ivy-count-format' is prepended to PROMPT during completion.\n\nCOLLECTION is either a list of strings, a function, an alist, or\na hash table, supplied for `minibuffer-completion-table'.\n\nPREDICATE is applied to filter out the COLLECTION immediately.\nThis argument is for compatibility with `completing-read'.\n\nWhen REQUIRE-MATCH is non-nil, only members of COLLECTION can be\nselected.\n\nIf INITIAL-INPUT is non-nil, then insert that input in the\nminibuffer initially.\n\nHISTORY is a name of a variable to hold the completion session\nhistory.\n\nKEYMAP is composed with `ivy-minibuffer-map'.\n\nPRESELECT, when non-nil, determines which one of the candidates\nmatching INITIAL-INPUT to select initially.  An integer stands\nfor the position of the desired candidate in the collection,\ncounting from zero.  Otherwise, use the first occurrence of\nPRESELECT in the collection.  Comparison is first done with\n`equal'.  If that fails, and when applicable, match PRESELECT as\na regular expression.\n\nDEF is for compatibility with `completing-read'.\n\nUPDATE-FN is called each time the candidate list is re-displayed.\n\nWhen SORT is non-nil, `ivy-sort-functions-alist' determines how\nto sort candidates before displaying them.\n\nACTION is a function to call after selecting a candidate.\nIt takes one argument, the selected candidate. If COLLECTION is\nan alist, the argument is a cons cell, otherwise it's a string.\n\nMULTI-ACTION, when non-nil, is called instead of ACTION when\nthere are marked candidates. It takes the list of candidates as\nits only argument. When it's nil, ACTION is called on each marked\ncandidate.\n\nUNWIND is a function of no arguments to call before exiting.\n\nRE-BUILDER is a function transforming input text into a regex\npattern.\n\nMATCHER is a function which can override how candidates are\nfiltered based on user input.  It takes a regex pattern and a\nlist of candidates, and returns the list of matching candidates.\n\nDYNAMIC-COLLECTION is a boolean specifying whether the list of\ncandidates is updated after each input by calling COLLECTION.\n\nEXTRA-PROPS can be used to store collection-specific\nsession-specific data.\n\nCALLER is a symbol to uniquely identify the caller to `ivy-read'.\nIt is used, along with COLLECTION, to determine which\ncustomizations apply to the current completion session." #<bytecode 0x777e6436501b47>) ("kill-ring: " ("(function_item (identifier) @name)" #("(add-to-list 'load-path \"/home/jorge/code/emacs-tr..." 0 1 (face (rainbow-delimiters-depth-1-face) fontified t) 1 24 (fontified t) 24 65 (face font-lock-string-face fontified t) 65 66 (face (rainbow-delimiters-depth-1-face) fontified t)) #("(add-to-list 'load-path \\\"$(PWD)/lisp\\\")" 0 28 (face (makefile-shell font-lock-string-face) fontified t) 28 31 (face (font-lock-variable-name-face makefile-shell font-lock-string-face) fontified t) 31 40 (face (makefile-shell font-lock-string-face) fontified t)) "(require 'tree-sitter-langs)" #("require-language" 0 16 (fontified t)) "https://www.youtube.com/watch?v=AQpIY5YCWys") :require-match t :preselect #("(add-to-list 'load-path \"/home/jorge/code/emacs-tr..." 0 1 (face (rainbow-delimiters-depth-1-face) fontified t) 1 24 (fontified t) 24 65 (face font-lock-string-face fontified t) 65 66 (face (rainbow-delimiters-depth-1-face) fontified t)) :action counsel-yank-pop-action :caller counsel-yank-pop :sort t))
  ivy-read("kill-ring: " ("(function_item (identifier) @name)" #("(add-to-list 'load-path \"/home/jorge/code/emacs-tr..." 0 1 (face (rainbow-delimiters-depth-1-face) fontified t) 1 24 (fontified t) 24 65 (face font-lock-string-face fontified t) 65 66 (face (rainbow-delimiters-depth-1-face) fontified t)) #("(add-to-list 'load-path \\\"$(PWD)/lisp\\\")" 0 28 (face (makefile-shell font-lock-string-face) fontified t) 28 31 (face (font-lock-variable-name-face makefile-shell font-lock-string-face) fontified t) 31 40 (face (makefile-shell font-lock-string-face) fontified t)) "(require 'tree-sitter-langs)" #("require-language" 0 16 (fontified t)) "https://www.youtube.com/watch?v=AQpIY5YCWys") :require-match t :preselect #("(add-to-list 'load-path \"/home/jorge/code/emacs-tr..." 0 1 (face (rainbow-delimiters-depth-1-face) fontified t) 1 24 (fontified t) 24 65 (face font-lock-string-face fontified t) 65 66 (face (rainbow-delimiters-depth-1-face) fontified t)) :action counsel-yank-pop-action :caller counsel-yank-pop)
  counsel-yank-pop(nil)
  funcall-interactively(counsel-yank-pop nil)
  call-interactively(counsel-yank-pop nil nil)
  command-execute(counsel-yank-pop)

[shackra]

Okay I was able to fix this #15 (comment)

now, I'm struggling with font-lock face property additions for the matches, I asked on StackExchange for help https://emacs.stackexchange.com/q/56143/690

after this is solved and you do a review, this thing can be merged.

[shackra]

Yes, it works, we have a tree-sitter playground inside Emacs. Please review the branch and notify me of any corrections you would like to happen

[Alexander-Miller]

@shackra You have for a review on reddit, here's my 2c.

[shackra]

@shackra You have for a review on reddit, here's my 2c.

saw my post today but no comments, maybe you mispoke... I will address your feedback

[Alexander-Miller]

I missed a word. I meant to say you asked for a review on reddit, so here's mine.

[shackra]

@Alexander-Miller made changes, you can re-review if you want (not sure why the button for requesting this does not work)

[Alexander-Miller]

made changes, you can re-review if you want

Done.

(not sure why the button for requesting this does not work)

Probably because I didn't start a review. (Why am I even able to start one? This is not my repo.)

[shackra]

Typo fixed.

[shackra]

@ubolonton this PR is ready to be reviewed by you!

[ubolonton]

This is pretty cool. I added some comments for improvement.

I have further suggestions to improve usability, but let's focus on getting a basic version merged first, and discuss those later.