Investigate cursor blocks

[dbalatero]

From #62, there is a fair amount of discussion from @yoricfr about cursor blocks.

This might be possible to do, but we'd need:

• an obj-c module
• a reference to the focused UITextView

https://stackoverflow.com/questions/36311582/caret-cursor-color-and-size
http://programming.jugglershu.net/wp/?p=765

[dbalatero]

@yoricfr This is an example of a built .so library that bridges Obj-C and Lua together:

https://github.com/asmagill/hs._asm.axuielement

[dbalatero]

This is how to get a focused UI element, which might have a UITextView pointer under the hood:
https://www.hammerspoon.org/docs/hs.uielement.html#focusedElement

[yoricfr]

What's remarkable with XVim2: the block adjust with the font and char's width.

Nova (constant block width)

Terminal (constant block width)

Xcode + XVim (variable block width!)

Edit: SublimeText handles it pretty well too, with a sleek transparency effect:

[yoricfr]

@yoricfr This is an example of a built .so library that bridges Obj-C and Lua together:
https://github.com/asmagill/hs._asm.axuielement
This is how to get a focused UI element, which might have a UITextView pointer under the hood:
https://www.hammerspoon.org/docs/hs.uielement.html#focusedElement

@dbalatero Thanks for the links. I feel like trying to build the simplest Mac application with a NSTextView in it. Then I'll do some experimenting with overriding the drawInsertionPoint() function and see what it does.
Like your link to JugglerShu or this link to christianTietze.
If I get a block caret, then I'll try to figure out the Lua integration in Hammerspoon.

[dbalatero]

That sounds great, thanks for your energy on this!

Yeah, the order of operations seems like:

• Get a simple Mac application with TextView and see how the API works
• Figure out how to call into a .so module from Lua
• Write a function that takes a UITextView and swaps its insertion point
• Wrap that in a nice Lua interface

Ownership

One big thing to figure out:

It's one thing for an application to manage its own UITextView caret style.

What we want is to reach into other applications and mess around with its caret style from the Hammerspoon process.

Is this considered OK by the APIs? Or is it securely isolated?

[dbalatero]

https://github.com/XVimProject/XVim/blob/53c652421ca8ad63a2380a5cad316b1997999b8a/Documents/Developers/DevNotes.txt#L118-L246

Here's some dev notes on it, which is great. It looks like it's patching NSTextView though to achieve this.

This Stackoverflow post seems to indicate that you can do it per-application, but not system wide: https://superuser.com/questions/429464/change-the-width-and-color-of-mac-os-x-text-caret-cursor

[dbalatero]

@yoricfr I did an experiment, and was able to draw a box over the next character.

Downsides:

• You'd have to figure out how to move the box as the cursor position changes.
• The normal line cursor still blinks
• What background color would you choose to work in all cases?

To try it, bind this code to a hot key, put your cursor somewhere, and fire the hotkey. You should see a box.

local ax = require("hs.axuielement")

currentAxElement = function()
  local systemElement = ax.systemWideElement()
  return systemElement:attributeValue("AXFocusedUIElement")
end

hs.hotkey.bind(super, 'e', function()
  local currentElement = currentAxElement()

  -- Get the current selection
  local range = currentElement:attributeValue("AXSelectedTextRange")

  -- Get the range for the next character after the blinking cursor
  local caretRange = {
    location = range.location,
    length = 1,
  }

  -- get the { h, w, x, y } bounding box for the next character's range
  local bounds = currentElement:parameterizedAttributeValue("AXBoundsForRange", caretRange)

  -- draw a black rectangle in the bounding box with 20% opacity
  local canvas = hs.canvas.new(bounds)

  canvas:insertElement(
    {
      type = 'rectangle',
      action = 'fill',
      fillColor = { red = 0, green = 0, blue = 0, alpha = 0.2 },
      frame = { x = "0%", y = "0%", h = "100%", w = "100%", },
      withShadow = false
    },
    1
  )

  canvas:level('overlay')
  canvas:show()
end)

[dbalatero]

I did another pass at this. This time, we redraw the cursor at a 60fps rate, so it captures any movement.

intervalTimer = nil

hs.hotkey.bind(super, 'e', function()
  -- draw a black rectangle in the bounding box with 20% opacity
  local canvas = hs.canvas.new({ x = 0, y = 0, h = 1, w = 1 })

  canvas:level('overlay')
  canvas:insertElement(
    {
      type = 'rectangle',
      action = 'fill',
      fillColor = { red = 0, green = 0, blue = 0, alpha = 0.2 },
      frame = { x = "0%", y = "0%", h = "100%", w = "100%", },
      withShadow = false
    },
    1
  )

  local repositionCursor = function()
    local currentElement = currentAxElement()

    -- Get the current selection
    local range = currentElement:attributeValue("AXSelectedTextRange")

    -- Last visible char
    local visibleRange = currentElement:attributeValue("AXVisibleCharacterRange")
    local lastVisibleIndex = visibleRange.length + visibleRange.location

    if range.location == lastVisibleIndex then
      -- hide the caret if we're at the end of the text box
      canvas:hide()
    else
      -- Get the range for the next character after the blinking cursor
      local caretRange = {
        location = range.location,
        length = 1,
      }

      -- get the { h, w, x, y } bounding box for the next character's range
      local bounds = currentElement:parameterizedAttributeValue("AXBoundsForRange", caretRange)

      -- move the position and resize
      canvas:topLeft({ x = bounds.x, y = bounds.y })
      canvas:size({ h = bounds.h, w = bounds.w })

      -- show if not shown
      canvas:show()
    end
  end

  repositionCursor()

  refresh = 1 / 60 -- 60fps
  intervalTimer = hs.timer.doEvery(refresh, repositionCursor)
end)

[dbalatero]

Ok I had another insane idea - what if we covered the blinking cursor with the background color of the text field?

Here's what drawing a "cursor cover" looks like, when I make it obvious and red:

To get the background color of the UITextInput, you can take a screenshot and get the color at a pixel:

local screenshotOfTextInput = hs.screen.mainScreen():snapshot(currentElement:attributeValue("AXFrame"))
local color = screenshotOfTextInput:colorAt({ x = 10, y = 10 })

If we set the color to that instead, you get:

A GIF of it in action:

This is probably not perfect:

• I have to set the width of the cursor cover to 2px to cover sub-pixel offsets of the cursor
• It probably fails for large fonts/large cursors? I don't know.

Final lua code (for now I just paste it in ~/.hammerspoon/init.lua as a proof of concept):

currentAxElement = function()
  local systemElement = ax.systemWideElement()
  return systemElement:attributeValue("AXFocusedUIElement")
end

intervalTimer = nil

hs.hotkey.bind(super, 'e', function()
  -- draw a black rectangle in the bounding box with 20% opacity
  local canvas = hs.canvas.new({ x = 0, y = 0, h = 1, w = 1 })

  canvas:level('overlay')

  -- block caret
  canvas:insertElement(
    {
      type = 'rectangle',
      action = 'fill',
      fillColor = { red = 0, green = 0, blue = 0, alpha = 0.2 },
      frame = { x = "0%", y = "0%", h = "100%", w = "100%", },
      withShadow = false
    },
    1
  )

  -- cursor disabler
  local cursorDisableCanvas = hs.canvas.new({ x = 0, y = 0, h = 1, w = 1 })

  cursorDisableCanvas:insertElement(
    {
      type = 'rectangle',
      action = 'fill',
      fillColor = { red = 255, green = 0, blue = 0, alpha = 1 },
      frame = { x = "0%", y = "0%", h = "100%", w = "100%", },
      withShadow = false
    },
    1
  )

  local repositionCursor = function()
    local currentElement = currentAxElement()

    -- Get the background color
    local screenshotOfTextInput = hs.screen.mainScreen():snapshot(currentElement:attributeValue("AXFrame"))
    local color = screenshotOfTextInput:colorAt({ x = 10, y = 10 })

    -- Get the current selection
    local range = currentElement:attributeValue("AXSelectedTextRange")

    -- Last visible char
    local visibleRange = currentElement:attributeValue("AXVisibleCharacterRange")
    local lastVisibleIndex = visibleRange.length + visibleRange.location

    if range.location == lastVisibleIndex then
      -- hide the caret if we're at the end of the text box
      canvas:hide()
      cursorDisableCanvas:hide()
    else
      -- Get the range for the next character after the blinking cursor
      local caretRange = {
        location = range.location,
        length = 1,
      }

      -- get the { h, w, x, y } bounding box for the next character's range
      local bounds = currentElement:parameterizedAttributeValue("AXBoundsForRange", caretRange)

      -- move the position and resize
      canvas:topLeft({ x = bounds.x, y = bounds.y })
      canvas:size({ h = bounds.h, w = bounds.w })

      -- show if not shown
      canvas:show()

      -- disable the cursor
      cursorDisableCanvas:topLeft({ x = bounds.x - 1, y = bounds.y })
      cursorDisableCanvas:size({ h = bounds.h, w = 2 })
      cursorDisableCanvas:elementAttribute(1, 'fillColor', color)

      cursorDisableCanvas:show()
    end
  end

  repositionCursor()

  refresh = 1 / 60 -- 60fps
  intervalTimer = hs.timer.doEvery(refresh, repositionCursor)
end)

[yoricfr]

@dbalatero How on earth do you come up with these ideas?
Really, you're taking a snapshot to determine the background colour, then color the cursor to make the caret disappear?!
And this is being done 60 times per second with a bunch of other stuff in the repositionCursor() function, how can it be that fast?!

I've copied your proof of concept in my init.lua file and it looks like magic. No matter the font and text size, the block's width is adjusting accordingly.

I don't know how you came up with calls like attributeValue("AXFrame") and parameterizedAttributeValue("AXBoundsForRange", caretRange). I didn't even know it was possible.

---

About the simplest Mac application, I simply added a NSTextView programmatically after the code automatically generated by Xcode for the window. MyTextView is a sub-class of NSTextView so we can override the drawInsertionPoint function:

import Cocoa
import SwiftUI

@main
class AppDelegate: NSObject, NSApplicationDelegate {

    var window: NSWindow!

    func applicationDidFinishLaunching(_ aNotification: Notification) {
        
        // Create the window and set the content view.
        window = NSWindow(
            contentRect: NSRect(x: 0, y: 0, width: 640, height: 480),
            styleMask: [.titled, .closable, .miniaturizable, .resizable, .fullSizeContentView],
            backing: .buffered, defer: false)
        window.center()
        window.makeKeyAndOrderFront(nil)
        
        // Code added to add a NSTextView
        // Instead of using NSTextView, we use our own class that is a sub-class of NSTextView 
        let ed = MyTextView(frame: NSMakeRect(20, 30, 360, 280))
        ed.font = NSFont(name:"Chalkduster", size:20)
        ed.string = "Chalkduster"
        ed.isEditable = true
        ed.isSelectable = true
        window.contentView!.addSubview(ed)
    }

    func applicationWillTerminate(_ aNotification: Notification) {
        // Insert code here to tear down your application
    }
}

// Our customised NSTextView
class MyTextView: NSTextView {
    var caretSize: CGFloat = 10

    open override func drawInsertionPoint(in rect: NSRect, color: NSColor, turnedOn flag: Bool) {
        var rect = rect
        let customColor = NSColor(red: 1.0, green: 0.3, blue: 0.1, alpha: 0.5)
        rect.size.width = caretSize
        super.drawInsertionPoint(in: rect, color: customColor, turnedOn: flag)
    }

    open override func setNeedsDisplay(_ rect: NSRect, avoidAdditionalLayout flag: Bool) {
        var rect = rect
        rect.size.width += caretSize
        super.setNeedsDisplay(rect, avoidAdditionalLayout: flag)
    }
}

I tried to play a little with boundingRect to get the exact box of the next letter, without success so far.
But anyway, as you said, customising the caret from a test application is the easy part. The question is wether it is possible or not to override the drawInsertionPoint of any frontmost application.

I have no clue. For now, I am in awe with your solution.

[dbalatero]

How on earth do you come up with these ideas?

For one, the AX documentation is really bad so I made a debug helper:

function debugElement(currentElement)
  local role = currentElement:attributeValue("AXRole")

  if role == "AXTextField" or role == "AXTextArea" or role == "AXComboBox" then
    logger.i("Currently in text field")
    logger.i(inspect(currentElement:parameterizedAttributeNames()))
    logger.i("attributes:")
    logger.i("-----------")

    local attributes = currentElement:allAttributeValues()

    local names = {}
    for name in pairs(attributes) do table.insert(names, name) end
    table.sort(names)

    for _, name in ipairs(names) do
      logger.i("  " .. name .. ": " .. inspect(attributes[name]))
    end

    logger.i("action names:")
    local names = currentElement:actionNames()
    logger.i(inspect(names))
    logger.i("action descriptions:")
    logger.i("--------------------")

    for _, name in ipairs(names) do
      logger.i("  " .. name .. ": " .. currentElement:actionDescription(name))
    end
  else
    logger.i("Role = " .. role)
  end
end

hs.hotkey.bind(super, 'd', function()
  local systemElement = ax.systemWideElement()
  local currentElement = systemElement:attributeValue("AXFocusedUIElement")

  debugElement(currentElement)
end

This dumps out all the parameterized attributes and simple attributes, then I just stare at it and look for interesting combinations.

As far as the screenshot thing, I think I just googled for "hammerspoon color at pixel" and ended up on this issue: Hammerspoon/hammerspoon#1559

which linked to here: Hammerspoon/hammerspoon#1868

and ended up with that idea…

And this is being done 60 times per second with a bunch of other stuff in the repositionCursor() function, how can it be that fast?!

I'm not sure actually. I guess all I know is that I intend for it to run every 1/60th of a second, but it could actually be taking longer than 1 / 60 = 16.66ms to run the function. I guess I'd actually need to time it.

---

To get the solution I have actually beyond a proof of concept, some extra things might need to be done:

• cache the pixel color as long as we stay inside the field, so we don't take a screenshot 60x/second (I think fields have an AXIdentifier we could use for uniqueness? not sure.)
• disable the block cursor follow when we're in a field that doesn't have accessibility
• only enable it when the various Vim modes are enabled

[yoricfr]

the AX documentation is really bad so I made a debug helper:

That is brilliant, thanks for sharing.
It's like being given a new pair of gleamy eyes in the darkness.

Up to now I was exploring with :

for k,v in pairs(currentElement) do
    hs.printf("%s - %s", k, v)
end

which obviously don't dig into the table pointers:

2020-11-02 13:23:17: AXParent - hs.axuielement: AXScrollArea (0x6000031fe638)
2020-11-02 13:23:17: AXRole - AXTextArea
2020-11-02 13:23:17: AXWindow - hs.axuielement: AXWindow (0x6000031da7f8)
2020-11-02 13:23:17: AXSharedCharacterRange - table: 0x6000031db5c0
2020-11-02 13:23:17: AXFrame - table: 0x6000031ea600
2020-11-02 13:23:17: AXSelectedTextRanges - table: 0x6000031d8900
2020-11-02 13:23:17: AXChildrenInNavigationOrder - table: 0x6000031ea7c0
2020-11-02 13:23:17: AXSelectedText - 
2020-11-02 13:23:17: AXTopLevelUIElement - hs.axuielement: AXWindow (0x6000031d93b8)
2020-11-02 13:23:17: AXIdentifier - First Text View
2020-11-02 13:23:17: AXTextInputMarkedRange - table: 0x6000031eb680
2020-11-02 13:23:17: AXSize - table: 0x6000031e8c80
2020-11-02 13:23:17: AXInsertionPointLineNumber - 2
2020-11-02 13:23:17: AXChildren - table: 0x6000031eb7c0
2020-11-02 13:23:17: AXHelp - nil
2020-11-02 13:23:17: AXValue - Chalkduster
2020-11-02 13:23:17: AXRoleDescription - text entry area
2020-11-02 13:23:17: AXFocused - true
2020-11-02 13:23:17: AXSharedTextUIElements - table: 0x6000031dad40
2020-11-02 13:23:17: AXVisibleCharacterRange - table: 0x6000031e9300
2020-11-02 13:23:17: AXPosition - table: 0x6000031dbc40
2020-11-02 13:23:17: AXSelectedTextRange - table: 0x6000031eb400
2020-11-02 13:23:17: AXNumberOfCharacters - 27

To get the solution I have actually beyond a proof of concept, some extra things might need to be done:

• cache the pixel color as long as we stay inside the field, so we don't take a screenshot 60x/second (I think fields have an AXIdentifier we could use for uniqueness? not sure.)
• disable the block cursor follow when we're in a field that doesn't have accessibility
• only enable it when the various Vim modes are enabled

All good points.

---

I'm not sure actually. I guess all I know is that I intend for it to run every 1/60th of a second, but it could actually be taking longer than 1 / 60 = 16.66ms to run the function. I guess I'd actually need to time it.

I ran a few tests, and it obviously depends on how big the FocusedUIElement is: from 300 snap/sec (small text area) to 1/sec (large text area)

hs.hotkey.bind({}, 'd', function()
  local systemElement = ax.systemWideElement()
  local currentElement = systemElement:attributeValue("AXFocusedUIElement")
  start = hs.timer.absoluteTime()
  nbSnapShot = 0
  -- how many loops in 1 sec?
  while hs.timer.absoluteTime() - start < 1E9 do
    local screenshotOfTextInput = hs.screen.mainScreen():snapshot(currentElement:attributeValue("AXFrame"))
    local color = screenshotOfTextInput:colorAt({ x = 10, y = 10 })
    nbSnapShot++
  end
  hs.printf("%s snapshots", nbSnapShot)
end)

I also noticed snapshots are 4 times larger (supposedly through the retina effect). For example my console's width is 550px but its snapshot is 2250px.
screenshotOfTextInput:saveToFile("~/Desktop/snapshot.jpg")

If we take a smaller portion of the UIElement, it get faster:
local screenshotOfTextInput = hs.screen.mainScreen():snapshot({ x = 0, y = 0, w = 10, h = 10}) but caching the color as you said is even better.

[dbalatero]

Ah thanks so much for doing the timing! That's super helpful.

I think taking a smaller portion screenshot is the way to go then!

edit: The other possibility is to just leave the cursor blinking for now.

I think the next step I'll take is to get this cursor functionality behind a beta config flag and merge it to master.

First priority probably should be knocking down the issues you found in the UTF8 thread though, and wrapping that up – that work I think is higher impact than the cosmetic cursor in here (as cool as it is!)

[dbalatero]

Ack, if we do this screen capture thing, Hammerspoon needs to ask the user for screen recording permissions. This like it would generate a lot of FUD from the user around security, and might be a bridge too far.

[yoricfr]

First priority probably should be knocking down the issues you found in the UTF8 thread though, and wrapping that up – that work I think is higher impact than the cosmetic cursor in here (as cool as it is!)

I totally rally your point.
The cursor thing was a detour while discussing the cosmetic frustration when reaching a character via F or T.
I would never have imagined it would bring us this far.

Ack, if we do this screen capture thing, Hammerspoon needs to ask the user for screen recording permissions. This like it would generate a lot of FUD from the user around security, and might be a bridge too far.

Yes I got this same permission request.
After seeing your trick in action, about taking screenshots silently in the background, and being that fast, I was thinking about spyware: imagine how easy it would be to spy someone's screen without him knowing. So I guess we can't blame the OS for requesting such permission.
It's not shocking to me. But clearly, taking screenshot to hide the cursor is a mind-blowing, "out-of-the-normal" trick!

[dbalatero]

@yoricfr I added the cursor overlay behind a beta flag in master (also this is the first feature to get a beta flag).

If you get the latest master, you can add vim:enableBetaFeature('block_cursor_overlay') to your init.lua to turn it on.

A few things I notice so far:

• Doesn't draw correctly in Chrome <textarea> (like this one)
• Doesn't draw correctly in Chrome menubar

edit: Chrome doesn't seem to support AXBoundsForRange very well. https://groups.google.com/a/chromium.org/g/chromium-accessibility/c/eB34iqVFAu8

Using the Accessibility API reminds me of writing cross-browser JS, but it's even less consistent somehow.

[dbalatero]

@yoricfr How is it feeling with the beta for you?

[dbalatero]

@yoricfr ping again?

[godbout]

have you considered selecting one character for the block cursor vibe?

Screen.Recording.2021-05-21.at.20.12.17.mp4

works pretty well in my case. some issues depending on the weather, but probably related to the hardcore Accessibility API.

[godbout]

btw the video is a bit laggy but in the real world it's flawless, on a 10 year old iMac.

(P.S.: and yeah the up is not implemented yet :D AX Strategy is much harder than expected if you want to do it flawlessly. many edge cases. and not even talking about handling smileys.)

[godbout]

😬️

[dbalatero]

Oh sorry, what did you need @godbout ?

[godbout]

ah, it's me. sorry!
i was just showcasing what i was able to achieve, regarding my earlier comment about selecting one character as a block cursor. apologies if that ended up being some noise to you.

have you considered selecting one character for the block cursor vibe?

Screen.Recording.2021-05-21.at.20.12.17.mp4

works pretty well in my case. some issues depending on the weather, but probably related to the hardcore Accessibility API.