Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Pixel Coordinates in Mouse Events #873

Open
bbb651 opened this issue Mar 29, 2024 · 7 comments · May be fixed by #929
Open

Support Pixel Coordinates in Mouse Events #873

bbb651 opened this issue Mar 29, 2024 · 7 comments · May be fixed by #929

Comments

@bbb651
Copy link

bbb651 commented Mar 29, 2024

Is your feature request related to a problem? Please describe.
Some terminals offer the ability to report pixel coordinates of mouse events along with the character coordinate. I want to make a UI based on the kitty image protocol (also blocked by #834) where pixel coordinates of events are important.

Describe the solution you'd like
I would like crossterm to provide access to these modes and add the pixel coordinates to MouseEvent.

Describe alternatives you've considered if any

Additional context
I found this website describing the SGR-Pixels (1016) mode.

@bbb651
Copy link
Author

bbb651 commented Mar 29, 2024

I just realized Event::Resize is also missing pixel sizes (there's a variety of xterm sequences for that, also there's TIOCGWINSZ with the ioctl syscall and SIGWINCH signal on unix). Are these not implemented due platform support? Also is this library for/against implementing image protocols such as the kitty image protocol, iTerm image protocol, sixel, etc? (they are relatively niche in terms of usage and platform support, and are relatively complex and require a cargo feature to avoid image crate dependencies, but I feel like there's a gap in the rust ecosystem for high level terminal image handling).

@bogzbonny
Copy link

bump! I would find this useful for the same reason. Basic image interaction in the terminal is an inhibiting feature for making some of my applications fully terminal based

@joshka
Copy link
Collaborator

joshka commented Sep 2, 2024

Rather than bumping consider doing the work to work out / design how something like this should be implemented.

Some constraints to keep in mind would be:

  • There are a large amount of dependents on crossterm (https://github.com/crossterm-rs/crossterm/network/dependents says 34500 repos), changing a fundamental type like the Resize / Mouse events would be problematic and should be approached with care
  • Think about how to implement this behavior in a way that works with terminals that support it and terminals that don't
  • Think particularly about various operating systems

Regarding image proto stuff, have you considered using ratatui and ratatui-image? I know that this goes one step further than just raw crossterm, but if your goal is writing working software then this might be a valid approach for your problem.

@bogzbonny
Copy link

bogzbonny commented Sep 3, 2024

Yeah I spent a bunch of time going through SGR-Pixels (1016)... the best that I could make out is that, to simply switch to 1016 would replace character positions altogether with pixel positions, making that an optional feature I think would be actually pretty straight-forward however not nearly as useful as having both pixel and character positions (IMO, thoughts?).

Considering that some terminals may not support pixels mouse responses, I imagine that it makes sense to add this under a feature flag (wezterm does, ref).
One open question I haven't been able to figure out is - how to get the the height/width of a character from the console to then be able to back calculate the character positions from the pixel dimensions (as to be able to provide pixel and character coordinates). Once this is figured out, everything else seems pretty straight forward. Also I suppose the MouseEvent type would be modified and have a couple extra fields (for pixel coordinates) with this feature flag on.

One more complication which I just thought of is if it would be required to somehow query the terminal for pixel compatibility and only provide the character output if pixel output is not available (even with the feature-flag on).

More relevant discussions: dankamongmen/notcurses#2326


edit: from that same document bbb651 posted, cells in pixels are CSI 16 t aka echo -e "\e[16t"

@bogzbonny
Copy link

@joshka btw implementing kitty image protocol from scratch isn't that difficult (I don't know about sixels though, I haven't tried), the need for supporting cursor location in pixels is so that one could hypothetically interact with an image while in the terminal - this logic is much more coupled with the inner workings of crossterm

@joshka
Copy link
Collaborator

joshka commented Sep 6, 2024

One open question I haven't been able to figure out is - how to get the the height/width of a character from the console to then be able to back calculate the character positions from the pixel dimensions (as to be able to provide pixel and character coordinates). Once this is figured out, everything else seems pretty straight forward. Also I suppose the MouseEvent type would be modified and have a couple extra fields (for pixel coordinates) with this feature flag on.

That part should be simple. Call terminal::window_size() then divide width by columns, height by rows.

@bogzbonny bogzbonny linked a pull request Sep 22, 2024 that will close this issue
@o-sdn-o
Copy link

o-sdn-o commented Sep 24, 2024

Maybe this will be useful. To get rid of the dependence on the cell size, it is enough to use floating point coordinates with the following format:

fractional_coordinates = integer_cellular_coordinates + coordinates_inside_the_cell / cell_size

This approach allows you to easily convert coordinates to pixel coordinates as well as to cell coordinates back and forth. At the same time, the console application does not need to know the cell size at all. The console application simply uses fractional coordinates as it needs.

I use this in vtm when the console application tracks mouse drags and reports back to the terminal if it is necessary to move the terminal GUI window at the pixel level, xlink: directvt/vtm#571 (comment), protocol draft: https://github.com/directvt/vtm/blob/master/doc/vt-input-mode.md#mouse

You can play with this by moving the vtm GUI windows on Windows (run vtm.exe, vtm.exe -r term or vtm -r test).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants