Visual recognition not finding elements / capture Sikuli visual automation debug info

[yolibernal]

I have trouble to make the visual automation work. I installed Sikuli as described in the instructions (I think) and when starting a flow Sikuli seems to start in the background.

But when trying to interact with an element there is an error:

ERROR - cannot find example.png on screen

When using the SikuliX IDE in tagui.sikuli to interact with the element it works without a problem.

[kensoh]

Hi Jonas @yolibernal thanks for raising this!

Can you share more which operating system are you using? If using Windows, is the display magnification % set to recommended %? Can you also paste the contents of the log files from tagui.sikuli folder here? There should be a default log file for Mac and another one for Windows.

I'm using macOS (High Sierra) from 2014 MacBook Air, Windows 10 and CentOS Linux without issue. Recently a couple of users running newer Macs have this problem but we can't figure out a way to replicate on other laptops.

[yolibernal]

Thanks for the fast answer!

I'm using a macOS (High Sierra) MacBook from 2015.
The logs contain:

running SikuliX: tagui.sikuli/SikuliX.app/Contents/Java/sikulix.jar -r tagui.sikuli
[error] RunTimeINIT: ***** BE AWARE: Running on Java 9 - Please report problems *****
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.python.core.PySystemState (file:/usr/local/lib/node_modules/tagui/src/tagui.sikuli/SikuliX.app/Contents/Java/sikulix.jar) to method java.io.Console.encoding()
WARNING: Please consider reporting this to the maintainers of org.python.core.PySystemState
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[tagui] START  - listening for inputs

[tagui] INPUT  - [1] dclick /Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png
[tagui] ACTION - dclick /Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png
[tagui] OUTPUT - [1] ERROR

[tagui] FINISH - stopped listening

[kensoh]

Hi Jonas,

This will be hard and take some time to troubleshoot because sikuli's functions such as click and doubleClick returns either 1 / 0 to denote success / error. Following will help to capture more details to see what could be the reason for not able to click on the image -

1. To get more verbose logs, change the following line in tagui/src/tagui from

tagui.sikuli/runsikulix -r tagui.sikuli >> tagui.sikuli/tagui.log 2>&1 &

to

tagui.sikuli/runsikulix -r tagui.sikuli -d 3 >> tagui.sikuli/tagui.log 2>&1 &

2. Then try a TagUI script like the following and see what is the output log?

if present('/Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png')
echo 'icon present'

// issue sikuli command directly using vision step
vision click('/Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png')

[yolibernal]

The logs show

running SikuliX: tagui.sikuli/SikuliX.app/Contents/Java/sikulix.jar -r tagui.sikuli -d 3
[error] RunTimeINIT: ***** BE AWARE: Running on Java 9 - Please report problems *****
[debug] RunTimeINIT: loadOptions: check: /usr/local/lib/node_modules/tagui/src
[debug] RunTimeINIT: loadOptions: check: /Users/jonasbernhard
[debug] RunTimeINIT: loadOptions: check: /Users/jonasbernhard/Library/Application Support/Sikulix/SikulixStore
[debug] RunTimeINIT: global init: entering as: IDE
[debug] RunTimeINIT: Accessing: GraphicsEnvironment.getLocalGraphicsEnvironment()
[debug] RunTimeINIT: Accessing: GraphicsEnvironment.getLocalGraphicsEnvironment().getScreenDevices()
[debug] RunTimeINIT: ScreenDevice 0 has (0,0) --- will be primary Screen(0)
[debug] RunTimeINIT: Monitor 0: (0, 0) 1440 x 900
[debug] RunTimeINIT: runs as sikulix.jar in: /usr/local/lib/node_modules/tagui/src/tagui.sikuli/SikuliX.app/Contents/Java
[debug] RunTimeINIT: exists libs folder at: /Users/jonasbernhard/Library/Application Support/Sikulix/SikulixLibs_201803080837
***** show environment for IDE (build 201803080837)
user.home: /Users/jonasbernhard
user.dir (work dir): /usr/local/lib/node_modules/tagui/src
user.name: jonasbernhard
java.io.tmpdir: /var/folders/s7/9h_905cj5hg8yqrtb66kk54w0000gn/T
running 64Bit on Mac OSX (10.13.5) as application .app (not from /Applications folder)
java 9-64 version 9.0.1+11 vm 9.0.1+11 class 53.0 arch x86_64
app data folder: /Users/jonasbernhard/Library/Application Support/Sikulix
libs folder: /Users/jonasbernhard/Library/Application Support/Sikulix/SikulixLibs_201803080837
executing jar: /usr/local/lib/node_modules/tagui/src/tagui.sikuli/SikuliX.app/Contents/Java/sikulix.jar
[error] RunTimeIDE: *** Java 9: skipped: dumpClassPath()
***** show environment end
[debug] RunTimeIDE: global init: leaving
[debug] RunTimeIDE: initIDEbefore: entering
[debug] RunTimeIDE: initIDEbefore: leaving
[debug] RunTimeIDE: initAPI: entering
[debug] RunTimeIDE: resourceList: enter
[debug] RunTimeIDE: resourceLocation: (class org.sikuli.ide.SikuliIDE) /Lib/sikuli
[debug] RunTimeIDE: resourceList: having jar: jar:file:/usr/local/lib/node_modules/tagui/src/tagui.sikuli/SikuliX.app/Contents/Java/sikulix.jar!/Lib/sikuli
[debug] RunTimeIDE: files exported: 7 from: Lib/sikuli to:
 /Users/jonasbernhard/Library/Application Support/Sikulix/Lib/sikuli
[debug] RunTimeIDE: initAPI: leaving
[debug] IDE: Switching to ScriptRunner with option -r, -t or -i
[debug] Runner: commandline: -r tagui.sikuli -d 3
[debug] --- Sikuli parameters ---
[debug] 1: -r
[debug] 2: tagui.sikuli
[debug] 3: -d
[debug] 4: 3
[debug] ScriptingSupport: givenScriptName:
tagui.sikuli
[debug] ScriptingSupport: Trying to run script:
/usr/local/lib/node_modules/tagui/src/tagui.sikuli/tagui.py
[debug] ScriptingSupport: initScriptingSupport: enter
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.python.core.PySystemState (file:/usr/local/lib/node_modules/tagui/src/tagui.sikuli/SikuliX.app/Contents/Java/sikulix.jar) to method java.io.Console.encoding()
WARNING: Please consider reporting this to the maintainers of org.python.core.PySystemState
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[debug] JythonSupport: init: starting
[debug] JythonSupport: init: success
[debug] JythonSupport: added as Jython::sys.path[0]:
/Users/jonasbernhard/Library/Application Support/Sikulix/Lib/site-packages
[debug] JythonSupport: ***** Jython sys.path
 0: /Users/jonasbernhard/Library/Application Support/Sikulix/Lib/site-packages
 1: /Users/jonasbernhard/Library/Application Support/Sikulix/Lib
 2: /usr/local/lib/node_modules/tagui/src/tagui.sikuli/SikuliX.app/Contents/Java/Lib
 3: /usr/local/lib/node_modules/tagui/src/tagui.sikuli/SikuliX.app/Contents/Java/sikulix.jar/Lib
 4: __classpath__
 5: __pyclasspath__/
[debug] JythonSupport: ***** Jython sys.path end
[debug] Jython: sikuli: Sikuli: starting init
[debug] RunTimeIDE: loadLib: libVisionProxy.dylib
[debug] Mouse: init start
[debug] Mouse: init end
[debug] RunTimeIDE: loadLib: libMacUtil.dylib
[debug] Jython: requested to use as default region: S(0)[0,0 1440x900]
[debug] Jython: sikuli: Sikuli: ending init
[debug] JythonScriptRunner: running Jython 2.7.1 
[debug] ScriptingSupport: initScriptingSupport: added: jython
[debug] ScriptingSupport: initScriptingSupport: exit with defaultrunner: jython (py)
[debug] ImagePath: new BundlePath:
/usr/local/lib/node_modules/tagui/src/tagui.sikuli/
[debug] Jython: requested to use as default region: S(0)[0,0 1440x900]
[debug] JythonScriptRunner: runPython: running script: 
/usr/local/lib/node_modules/tagui/src/tagui.sikuli
[tagui] START  - listening for inputs

[tagui] INPUT  - [1] present /Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png
[tagui] ACTION - present /Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png
[debug] Image: loaded: /Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png (file:/Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png)
[debug] Image: cached: /Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png (158 KB) (# 1 KB 158 -- 0 % of 64 MB)
[debug] Region: exists: waiting 1,0 secs for /Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png to appear in S(0)[0,0 1440x900]
[debug] Region: handleFindFailed: /Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png
[debug] Region: exists: /Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png did not appear [1026 msec]
[tagui] OUTPUT - [1] ERROR

[tagui] INPUT  - [2] vision click('/Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png')
[tagui] ACTION - click('/Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png')
[debug] Image: reused: /Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png (file:/Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png)
[debug] Region: wait: waiting 10,0 secs for /Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png to appear in S(0)[0,0 1440x900]
[debug] Region: handleFindFailed: /Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png
[debug] Region: wait: /Users/jonasbernhard/Code/RPA/tag-ui/visual-automation/o_image_icon.png did not appear [10329 msec]
[tagui] OUTPUT - [2] SUCCESS

[tagui] FINISH - stopped listening
[debug] RunTimeIDE: final cleanup
[debug] FileManager: deleteFileOrFolder:
/var/folders/s7/9h_905cj5hg8yqrtb66kk54w0000gn/T/Sikulix_1595436093

[yolibernal]

I just tested the script with an image taken with the Sikuli IDE and it worked fine.
Before I was using a screenshot selection taken with macOS Grab, which produces an image with a much higher resolution than the image taken with the IDE.

Might it be that the handleFindFailed occurs due to timeout caused by the higher resolution? I imagine that the algorithm would take much longer to execute in that case.
Or Grab changes the image slightly when saving.

[kensoh]

Hi @yolibernal can you try taking the same icon using 1. macOS Grab, 2. cmd+shit+4 and 3. sikuli IDE, and share the image dimensions (and dpi if available)?

I googled and saw some threads online talking about retina display on macbook which can give screenshots larger than how it appears on the screen. That could be a reason as sikuli visual matching won't work if the size of the image versus how it looks on the screen differs too much.

Also, for taking screenshot with the sikuli ide, how do you retrieve the image to use it in tagui? Is it you check the path of the image and copy it out to be used in tagui script?

[yolibernal]

The dimension of the images are:

• Grab selection: 190x198 with 144 pixels/inch
• Grab screenshot: 2880 × 1800 with 144 pixels/inch
• Grab cropped screenshot: 194x208 with 144 pixels/inch
• Sikuli IDE selection: 127x118 with 72 pixels/inch

I made some further tests and the issue, in fact, seems to be with Grab. When importing the Grab images in the Sikuli IDE the visual recognition did not work again (both Grab selection and cropped Grab screenshot). But when displaying one of the images on the screen the script was able to find them, so they seem to not be a valid copy of the displayed screen sections.

Also, for taking screenshot with the sikuli ide, how do you retrieve the image to use it in tagui? Is it you check the path of the image and copy it out to be used in tagui script?

When selecting them in the Sikuli IDE they get saved in the package where the scripted is saved (packageName.sikuli). From there I copied them out.

So, all in all, it looks like the issue is neither with TagUI nor with Sikuli. But thanks for your help!

[kensoh]

Thanks Jonas! This will be a recurring problem for other users with retina displays. I'll continue looking to see if there's an easier way for users to capture images that are of the 'correct' resolutions for Sikuli visual engine to use for automation (instead of capturing image files using Sikuli IDE and copying out).

[kensoh]

adding some sikuli debug logs on working laptops for reference

Windows - tagui_tk.log
macOS -tagui_mac.log

[kensoh]

This part of Jonas' observation is interesting - that when the image captured by macOS Grab tool is shown in preview, the image recognition works on that image, but does not work on the actual UI element. Since this is an issue with the image capturing, shall close the issue for now, but will add on if there are better ways to capture images accurately for automation! Also, going to add a commit which will capture more detailed Sikuli debug logs, for easier troubleshooting in future.

But when displaying one of the images on the screen the script was able to find them, so they seem to not be a valid copy of the displayed screen sections.

[pythiantech]

Facing the same issue even when using Sikuli IDE. I am on macOS Catalina, MacBook Pro 2017.

[kensoh]

Check that you have Java JDK or OpenJDK 64-bit - https://github.com/kelaberetiv/TagUI#visual-automation. Then check whether it is because Catalina is blocking TagUI due to the new security restrictions -

tebelorg/RPA-Python#79
tebelorg/RPA-Python#78

[githubuser789]

Hi kensoh,

I'm recently started working on sikuli along with java code. I'm trying to click on image on my windows system but getting Findfailed exception. I kept images under one folder and I have given folder path to click on image. I run app on admin mode still no luck. Could you please help me on this.

note : resoultion is on recommend resolution

System configuration:
Windows 10 Enterprise edition

[kensoh]

Sorry @githubuser789 I was away the past few months - tebelorg/RPA-Python#144

For SikuliX issues such as using it in your Java IDE, please raise issues to SikuliX GitHub page. This project is specifically for issues for TagUI related to visual automation using SikuliX, and not using SikuliX in Java. So I don't have answers for that.

[khuntia]

I had the same issue, came to know that its actually a screen resolution issue in modern screens. I have 2560x1440 screen. When i installed sikulix ide, and took a snapshot from it, only 1/4th of my screen was captured. This is where i understood that it simply cant see such high resolution. I changed my resolution to 1280x800 and everything worked perfectly fine, even direct screen capture, no need to capture from sikulix ide just change resolution to older figures. I havent deep dived the reason for this, but I mostly think it is to do with some assumption by sikulix on max screen resolution i guess. Will update here once i have time to deep dive.
Abhi

[kensoh]

Hi Abhi, thanks for sharing your findings I appreciate it! I hear before a friend was saying some Retina display the resolution density is higher, so snapshots captured with the defaulting image capture tool may be different from what computer sees.

This is the issue with details on this Retina display issue and solution. User changed the PPI from 144 to 72 to get it working.

#896 (comment)

[kensoh]

The solution for macOS Retina display problem will be adding this line at the start of your automation workflow -

vision Settings.AlwaysResize = 0.5

Solution first found at RPA for Python issue - tebelorg/RPA-Python#170 (comment)