[Feature Request] Add an intelligent screen ruler with image edge detection

[martinszeltins]

Feature Description

Microsoft has this really cool PowerToys screen ruler with smart image edge detection to measure distance between objects on the screen. It looks really nice. Unfortunately there is no such tool available for Linux. This would be a really powerful feature if implemented inside Flameshot.

It also seems like a relatively simple feature to add using tools like OpenCV to detect edges in an image and measure distance between them in pixels.

[mmahmoudian]

The opencv have been already discussed. There are some challenges to it. For example many use dark theme and dark background and this makes it very hard to detect the edges of the window.

In addition to the point above, I understand that it is cool to have such feature, but I cannot see how that fits into a screenshot tool. Perhaps it fits more into a tool like kruler and Gnome Screen Ruler. Would you please elaborate on the usecase if you believe it is a good fit for Flameshot?

[borgmanJeremy]

@mmahmoudian What do you think about a measurement mode? I've been considering adding a way to measure angles so maybe we consider an entire measurement mode.

[mmahmoudian]

@borgmanJeremy I myself use Flameshot in daily basis for usecases that have nothing to do with taking screenshot. For instance I use it a lot for live annotation during presentation. But I do this because the other tools are lacking essential features. For instance Gromit-MPX lacks the ability to draw straight lines. I also see that Kruler which I also use regularly is lacking the feature to measure angles or stick to edges of windows.

I believe certain features like measurement are super useful and Flameshot has some features that can even enhance the experience of such feature (e.g the pixel selection zoom). But I think such feature are out of the scope of a screenshot tool.

I'm not necessarily against it. I'm just concerned that how would it affect the load time and snappiness of Flameshot. Basically we should avoid creating a bloatware. It feels like it is a good addition as a plugin rather than a native feature. It can even be a plugin that is installed by default, but the user should be able to disable it.

[martinszeltins]

@borgmanJeremy I myself use Flameshot in daily basis for usecases that have nothing to do with taking screenshot. For instance I use it a lot for live annotation during presentation. But I do this because the other tools are lacking essential features. For instance Gromit-MPX lacks the ability to draw straight lines. I also see that Kruler which I also use regularly is lacking the feature to measure angles or stick to edges of windows.

I believe certain features like measurement are super useful and Flameshot has some features that can even enhance the experience of such feature (e.g the pixel selection zoom). But I think such feature are out of the scope of a screenshot tool.

I'm not necessarily against it. I'm just concerned that how would it affect the load time and snappiness of Flameshot. Basically we should avoid creating a bloatware. It feels like it is a good addition as a plugin rather than a native feature. It can even be a plugin that is installed by default, but the user should be able to disable it.

@mmahmoudian yes, the plugin idea sounds really good. I don't know if Flameshot currently has plugin support or not.

I also use Flameshot for more than a screenshot tool because it has all these great extra features. And it is really nice that I can simply press "PrtScr" on my keyboard and I can take a screenshot, draw on it, measure it etc. It is so convenient.

[mmahmoudian]

@martinszeltins

I don't know if Flameshot currently has plugin support or not.

No we don't have the plugin system yet, but we are sold on the idea already and many feature requests are already marked with the Plugin label so that we keep a good track of what can/should be implemented as a plugin. Of course, it would be nice if you can go through them and up-vote those you like.

Regarding the implementation of plugin system, perhaps @borgmanJeremy and @veracioux can write a more detailed comment about the technical ch a challenges and the plan. You can follow and participate in the discussions here:
#2529

[martinszeltins]

The opencv have been already discussed. There are some challenges to it. For example many use dark theme and dark background and this makes it very hard to detect the edges of the window.

@mmahmoudian I don't know how Microsoft did it. Maybe with a different library. But as you can see it works really good even in dark mode.

I don't know how to add the "Plugin" tag to this thread, perhaps someone can add it.

[mmahmoudian]

@martinszeltins

I don't know how Microsoft did it. Maybe with a different library. But as you can see it works really good even in dark mode.

Maybe they are using some API to query the window manager for window location and dimensions. Windows is not opensource and many things cannot be confirmed. In Xorg we can to some extent do this, but in Wayland it is not at the moment possible.

I don't know how to add the "Plugin" tag to this thread, perhaps someone can add it.

I can add the label, but I would like to discuss this with @borgmanJeremy as he might like it to be added as a native feature.

Regardless of the points above, would you please elaborate on the use-case of such feature and explain if this is something that you would need from a screenshot tool or it is a handy feature that you would like to have but is not directly related to screenshot.

[martinszeltins]

@mmahmoudian I don't think this has anything to to with a window or a window manager. You take a screenshot of your desktop (which is an image). Then you use OpenCV or some other library to detect edges in that image (screenshot). Then as you move your mouse over your screenshot it draws distance lines between the detected edges and shows the distance in pixels.

At the end you are working with a simple image which should not depend on Xorg or Wayland or anything OS specific.

That is how I see it, please correct me if I'm wrong.

[mmahmoudian]

@martinszeltins thank you for the brief explanation, but I already know how in theory opencv works :) I have also worked enough with edge detection algorithms in my machine learning projects to know the limitations as well. Good news is that we are dealing with rectangles, the bad news is that window decorations and compositors are not uniformly doing the same thing. But these are besides the point. My question was about providing a use-case that such tool can enhance the experience of a screenshot tool.

[martinszeltins]

Would you please elaborate on the use-case of such feature and explain if this is something that you would need from a screenshot tool or it is a handy feature that you would like to have but is not directly related to screenshot.

@mmahmoudian Yes, so the reason I believe that this should be part of a "screenshot tool" is that it follows a very natural workflow. Currently you can take a screnshot and you have drawing, selection size indicator, blur etc. We already have "selection size indicator" so naturally this can be extended to show the "size" between edges in the screnshot image.

Also, this would make it much easier and be more productive if I did not need to open another tool. Essentially if this was a seperate program it would do what Flameshot does + measure distance between edges in the screenshot. So it does not make sense to create another tool that can 1. Take a screenshot 2. As user moves their mouse around the taken screenshot it shows the distance between the edges in the screenshot. It would be a waste of time and resources to create a completely new tool since 50% of the functionality already exists in Flameshot.

[martinszeltins]

I did a quick mockup of what the feature could look like:

[martinszeltins]

The opencv have been already discussed. There are some challenges to it. For example many use dark theme and dark background and this makes it very hard to detect the edges of the window.

@mmahmoudian There are some tricks that can be used to make edge detection better. One trick is to invert the dark image, then increase the contrast, remove the noise with gaussian filter and lower the brightness. And then OpenCV will detect edges much much better. Perhaps this is the same technique that Microsoft uses in their tool.

[martinszeltins]

So I played around a little bit with OpenCV and it seems to be doing a really good job. It can be improved even more by applying some clever tricks like removing noise etc.

import cv2 as cv

image = cv.imread("screenshot2.png")
gray_image = cv.cvtColor(image, cv.COLOR_BGR2GRAY)
canny_edges = cv.Canny(gray_image, 60, 60)
cv.imshow("Canny Edges", canny_edges)
cv.waitKey(0)

The only thing left to do would be to draw a line in 4 direction from the current mouse cursor.

1. Draw a line to the left of mouse cursor position to the closest edge (to the left)
2. Draw a line to the right of mouse cursor position to the closest edge ( to the right)
3. Draw a line from mouse position to the top to the closest edge (at the top)
4. Draw a line from mouse position to the bottom to the closest edge(at the bottom)
5. Show distance in pixels between edges.

[borgmanJeremy]

I also have a test branch and it’s working okay. This is a much easier problem than what was last attempted with opencv (window boundary detection).

I have a bit more of a blur on mine to reduce the effect of text.

it’s going to need a lot of testing across different resolutions and screen scaling .

[martinszeltins]

I also have a test branch and it’s working okay. This is a much easier problem than what was last attempted with opencv (window boundary detection).

I have a bit more of a blur on mine to reduce the effect of text.

it’s going to need a lot of testing across different resolutions and screen scaling .

@borgmanJeremy Maybe it's possible to get a prototype ready to play with it and test it?

[borgmanJeremy]

Yup, my time is pretty constrained but this is a fun feature to play with. I’ll try to have my branch pushed later today.

[kunaltyagi]

@martinszeltins Instead of using constant parameters in the canny detection, you can use a auto-canny approach, where the median of the grayscale image is reduced and increased by a factor (say 0.3) to make a single parameter canny.

This technique works despite the darkness/brightness of the image and the factor parameter can often be fixed after some experimentation with a greater confidence than the 2 different values required for classic canny

[martinszeltins]

I found this Chrome extension that does exactly what this feature request is all about, it's called Dimensions and it works very very well: https://chrome.google.com/webstore/detail/dimensions/baocaagndhipibgklemoalmkljaimfdj (of course this works only inside a browser)

I don't know if this extension uses OpenCV but one could examine the source code of this extension and maybe get some ideas. It seems like they have accomplished this feature using pure Javascript.