Cannot transpose keywords

[manuel-uberti]

Expected behavior

Given a .clj file with the following content:

{:a :b}

With point between :a and :b:

{:a| :b}

If I hit C-M-t, I get:

{:b :a}

Just like clojure-mode behaves.

Actual behavior

Given the previous scenario, when I hit C-M-t, I get the message:

transpose-subr-1: Don’t have two things to transpose

Steps to reproduce the problem

Provided clojure-ts-mode is installed (I installed via NonGNU ELPA):

• emacs -Q
• M-x package-initialize RET
• C-x C-f /tmp/test.clj RET
• M-x clojure-ts-mode RET (thanks @sogaiu for noticing I missed this step)
• Write {:a :b}
• Place cursor between :a and :b
• Hit C-M-t

Environment & Version information

clojure-ts-mode version

0.1.4

Emacs version

GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, cairo version 1.16.0) of 2023-08-29

Operating system

Ubuntu 22.04.3

[manuel-uberti]

FTR, the same problem appears with elements with vectors and lists.

[sogaiu]

In my case transposition does occur without issues with the following bits:

• clojure-ts-mode (d630cd6)
• Emacs 29.1.50 (built from source, not sure of commit, but around 2023-08-09)
• Ubuntu 22.04.3

I'll see if I can reproduce with bits closer to the original report.

[sogaiu]

Still no reproduction (transposition happens) using:

• clojure-ts-mode (0.1.4 from Non-GNU Elpa -- used M-x package-install)
• Emacs 29.1.50 (built from source, f26622b2 2023-08-28)
• emacs -Q
• M-x package-initialize
• C-x C-f /tmp/test.clj RET
• Put {:a :b} as buffer content
• M-x clojure-ts-mode (not doing this leaves me in fundamental mode, though no issue there either)
• Place cursor between :a and :b
• Enter C-M-t

[manuel-uberti]

Can you try building Emacs master instead of 29, by any chance?

[sogaiu]

Yes.

That seems to be sufficient to reproduce, I got:

transpose-subr-1: Don't have two things to transpose

[sogaiu]

I compared the source of transpose-sexps from Emacs 29.x and 30.y.

29.x:

(defun transpose-sexps (arg &optional interactive)
  "Like \\[transpose-chars] (`transpose-chars'), but applies to sexps.
Unlike `transpose-words', point must be between the two sexps and not
in the middle of a sexp to be transposed.
With non-zero prefix arg ARG, effect is to take the sexp before point
and drag it forward past ARG other sexps (backward if ARG is negative).
If ARG is zero, the sexps ending at or after point and at or after mark
are interchanged.
If INTERACTIVE is non-nil, as it is interactively,
report errors as appropriate for this kind of usage."
  (interactive "*p\nd")
  (if interactive
      (condition-case nil
          (transpose-sexps arg nil)
        (scan-error (user-error "Not between two complete sexps")))
    (transpose-subr
     (lambda (arg)
       ;; Here we should try to simulate the behavior of
       ;; (cons (progn (forward-sexp x) (point))
       ;;       (progn (forward-sexp (- x)) (point)))
       ;; Except that we don't want to rely on the second forward-sexp
       ;; putting us back to where we want to be, since forward-sexp-function
       ;; might do funny things like infix-precedence.
       (if (if (> arg 0)
	       (looking-at "\\sw\\|\\s_")
	     (and (not (bobp))
		  (save-excursion
                    (forward-char -1)
                    (looking-at "\\sw\\|\\s_"))))
	   ;; Jumping over a symbol.  We might be inside it, mind you.
	   (progn (funcall (if (> arg 0)
			       'skip-syntax-backward 'skip-syntax-forward)
			   "w_")
		  (cons (save-excursion (forward-sexp arg) (point)) (point)))
         ;; Otherwise, we're between sexps.  Take a step back before jumping
         ;; to make sure we'll obey the same precedence no matter which
         ;; direction we're going.
         (funcall (if (> arg 0) 'skip-syntax-backward 'skip-syntax-forward)
                  " .")
         (cons (save-excursion (forward-sexp arg) (point))
	       (progn (while (or (forward-comment (if (> arg 0) 1 -1))
			         (not (zerop (funcall (if (> arg 0)
							  'skip-syntax-forward
						        'skip-syntax-backward)
						      ".")))))
		      (point)))))
     arg 'special)))

30.y:

(defun transpose-sexps (arg &optional interactive)
  "Like \\[transpose-chars] (`transpose-chars'), but applies to sexps.          
Unlike `transpose-words', point must be between the two sexps and not           
in the middle of a sexp to be transposed.                                       
With non-zero prefix arg ARG, effect is to take the sexp before point           
and drag it forward past ARG other sexps (backward if ARG is negative).         
If ARG is zero, the sexps ending at or after point and at or after mark         
are interchanged.                                                               
If INTERACTIVE is non-nil, as it is interactively,                              
report errors as appropriate for this kind of usage."
  (interactive "*p\nd")
  (if interactive
      (condition-case nil
          (transpose-sexps arg nil)
        (scan-error (user-error "Not between two complete sexps")))
    (transpose-subr transpose-sexps-function arg 'special)))

Looks like some factoring has happened.

I tried using edebug with both versions and it seems that what is happening in transpose-subr differs.

Specifically, in 29.x, these lines:

      (setq pos1 (funcall aux -1))
      (setq pos2 (funcall aux arg))

give non-nil results, while the corresponding lines in 30.y both return nil.

FWIW, aux appears to be treesit-transpose-sexps in 30.y, and something that shows #f(compiled-function (arg) #<bytecode 0x.....>) in 29.x.

So far this doesn't seem like a bug in clojure-ts-mode, but who knows what further investigation might yield :)

[manuel-uberti]

Well, thanks a lot really for going through all this. It may be worth reporting an Emacs bug, so I'll try to do that and point to this discussion.

[sogaiu]

I was mistaken in my last report -- aux was not treesit-transpose-sexps for 29.x -- that doesn't seem to exist in Emacs 29.x. My bad.

I've amended that above in my previous comment.

[sogaiu]

I tried stepping through treesit-transpose-sexps and the vague sense I get is that it may be assuming things that don't hold for some parts of tree-sitter-clojure (the tree-sitter grammar that clojure-ts-mode currently uses).

Note that transposition does seem to work for integers.

Using tree-sitter-clojure's terms, transposition works for num_lit (numbers), but not kwd_lit.

Currently, kwd_lit is defined this way:

    kwd_lit: $ =>
    choice($._kwd_leading_slash,
           $._kwd_just_slash,
           $._kwd_qualified,
           $._kwd_unqualified),

    // (namespace :/usr/bin/env) ; => ""
    // (name :/usr/bin/env) ; => "usr/bin/env"
    _kwd_leading_slash: $ =>
    seq(field('marker', $._kwd_marker),
        field('delimiter', NS_DELIMITER),
        field('name', alias(KEYWORD_NAMESPACED_BODY, $.kwd_name))),

    // (namespace :/) ;=> nil
    // (name :/) ;=> "/"
    _kwd_just_slash: $ =>
    seq(field('marker', $._kwd_marker),
        field('name', alias(NS_DELIMITER, $.kwd_name))),

    _kwd_qualified: $ =>
    prec(2, seq(field('marker', $._kwd_marker),
                field('namespace', alias(KEYWORD_NO_SIGIL, $.kwd_ns)),
                field('delimiter', NS_DELIMITER),
                field('name', alias(KEYWORD_NAMESPACED_BODY, $.kwd_name)))),

    _kwd_unqualified: $ =>
    prec(1, seq(field('marker', $._kwd_marker),
                field('name', alias(KEYWORD_NO_SIGIL, $.kwd_name)))),

    _kwd_marker: $ =>
    choice(KEYWORD_MARK, AUTO_RESOLVE_MARK),

The definition I currently see for treesit-transpose-sexps is:

 1835 (defun treesit-transpose-sexps (&optional arg)
 1836   "Tree-sitter `transpose-sexps' function.                                           
 1837 ARG is the same as in `transpose-sexps'.                                             
 1838                                                                                      
 1839 Locate the node closest to POINT, and transpose that node with                       
 1840 its sibling node ARG nodes away.                                                     
 1841                                                                                      
 1842 Return a pair of positions as described by                                           
 1843 `transpose-sexps-function' for use in `transpose-subr' and                           
 1844 friends."
 1845   (let* ((parent (treesit-node-parent (treesit-node-at (point))))
 1846          (child (treesit-node-child parent 0 t)))
 1847     (named-let loop ((prev child)
 1848                      (next (treesit-node-next-sibling child t)))
 1849       (when (and prev next)
 1850         (if (< (point) (treesit-node-end next))
 1851             (if (= arg -1)
 1852                 (cons (treesit-node-start prev)
 1853                       (treesit-node-end prev))
 1854               (when-let ((n (treesit-node-child
 1855                              parent (+ arg (treesit-node-index prev t)) t)))
 1856                 (cons (treesit-node-end n)
 1857                       (treesit-node-start n))))
 1858           (loop (treesit-node-next-sibling prev t)
 1859                 (treesit-node-next-sibling next t)))))))

On line 1845, (treesit-node-at (point)) yields a node for kwd_name, and its parent is kwd_lit.

On line 1846, (treesit-node-child parent 0 t) yields kwd_name.

This logic is not likely to work with tree-sitter-clojure as it is currently implemented.

I think what one likely wants (if possible) is to identify kwd_lit at point and then map_lit as the parent.

Below is an invocation and output of running tree-sitter parse on the sample buffer content:

$ tree-sitter parse samples/transpose-keywords.clj
(source [0, 0] - [1, 0]
  (map_lit [0, 0] - [0, 7]
    value: (kwd_lit [0, 1] - [0, 3]
      name: (kwd_name [0, 2] - [0, 3]))
    value: (kwd_lit [0, 4] - [0, 6]
      name: (kwd_name [0, 5] - [0, 6]))))

It's very likely that transposition will not work with symbols either. Yes, I just checked and the same thing happens there too.

[manuel-uberti]

I see, thanks again. IIUC, it does not make sense to report this as an Emacs bug then.

Put otherwise, is there something to adjust in tree-sitter-clojure?

[sogaiu]

I am not sure about it not being an Emacs bug yet actually.

I'm not sure the treesit-transpose-sexps implementation is reasonable. It seems to be assuming that (treesit-node-at (point)) will yield an appropriate node for what it wants to accomplish. For tree-sitter-clojure's case, you might say it is looking too deeply when trying to handle keywords and symbols.

IIUC, there are over 300 tree-sitter grammars and I wonder whether tree-sitter-clojure is the only one that has this sort of characteristic.

The reason keywords and symbols are defined the way they are is to support the ability to access "structure" that both keywords and symbols have in Clojure (the namespace part and the name part). I'm not sure what kind of adjustment could (or should) be made on tree-sitter-clojure's side, but there hasn't been much time for thoughts to arise :)

May be @dannyfreeman will have some relevant thoughts.

[manuel-uberti]

Sorry if I am being too pushy, this is not a blocking/urgent problem by all means. I've been a happy user of this mode for a while, this won't change my opinion. :)

[sogaiu]

No worries!

Thanks for opening this issue -- I think it's often better to learn of these things earlier rather than later :)

[dannyfreeman]

I do not believe this would be an issue if tree-sitter would have properly parsed keyword names and namespaces into fields of anonymous nodes instead of forcing us to resort to named nodes. Either way here we are. Funny enough you can transpose keyword namespace and names by running transpose-sexps in the middle of a namespaced keyword :my-ns|/the-name where | is the cursor.

I'm not sure we can fix this on our end without dropping some features in the grammar, but we can use the original transpose-sexp function while tree-sitter is active.

@manuel-uberti do you mind trying out the latest change on the main branch and see if that fixes your issue?