keyboard [enter] , click OK.png not working - see working example from Marcelo

[maheswarGR]

@kensoh Using windows tagui 6
Need to click OK or cancel button in alert window. Tried multiple ways
keyboard [enter]
click OK.png
vision click("OK.png")
not working

Also tried taking screenshot using sikuli and placed the image in C:\tagui\flows\samples.. not working

I have placed images in C:\tagui\flows\samples

[kensoh]

Hi @maheswarGR the image has to be in the same folder as your .tag file, if you are doing click OK.png.

Can you try changing your Windows display zoom % to the recommended % or 100% to see if that helps?

Also, can you share the log file in tagui\src\tagui.sikuli folder to see if there are clues there?

[kensoh]

CC @ruthtxh

[kensoh]

Lastly, make sure you have installed Java (for eg Open Corretto), at least 64-bit version 8 is needed
See more here and the link to the Corretto installer - https://tagui.readthedocs.io/en/latest/setup.html

[maheswarGR]

yes image.png and tag files are in same path C:\tagui\flows\samples
its not working from wait 2 sec line
log file tagui.sikuli is empty
How to change windows display zoom ?

[kensoh]

Have you tried installing Corretto from above link?

For changing Windows display zoom, you can google to find tutorial or video to show the steps.

[kensoh]

You can also join our weekly Zoom call in 40 minutes time to look into this

[kensoh]

Here is the URL - #914

[maheswarGR]

yes windows zoom is previously 100% and i tried changing to recommended % also ..didnt work

I am using java version

java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)
Java HotSpot(TM) 64-Bit Server VM (build 25.0-b70, mixed mode)

will this support tagui vision click?

[kensoh]

Yes this version is ok. Can you share the logs here in tagui\src\tagui.sikuli folder to see if there are clues there?

[maheswarGR]

there are many files in it
expect 1st file , all are empty
After alert box appears in ui page, next line in tag file is not reading even.
the control is not coming back to the tagui from chrome when the alert is displayed

all tag and image.png is located in same path

[kensoh]

Oh I see, I know why now. The reason is because until user manually clicks the OK button, the control does not return back to TagUI. The solution is using visual automation to click on the button that triggers this popup, then clicking OK. For eg

click delete.png
click ok.png

Please try something like above and tell me what do you see in the logs, thanks!

[maheswarGR]

I tried . Until user manually clicks the OK button, the control does not return back to TagUI...
"click ok.png" in tag file is not read.
the control is not coming back to the tagui from chrome when the alert is displayed

I have attached log files for your reference..

My tag file :

http://3.132.253.47/dmapim/publisher/login.html?publisherName=coe

wait 15s
type //[@id="username_view"] as admin
type //[@id="password"] as Thbs1234!
click //*[@id="container"]/div/center/form/div/div[4]/div/button
wait 20s

click //*[@id="menuAccordion"]/div[1]/a/span
click //td[text()='RPA-PoC-Blueprism']//following::td[@id='delete']/i

click ok.png

logFiles.zip

NOTE : tried click image.png ..works until alert window appears.

[kensoh]

Hi @maheswarGR , as mentioned above, use click delete.png to use visual automation instead of click //td[text()='RPA-PoC-Blueprism']//following::td[@id='delete']/i because click //td[text()='RPA-PoC-Blueprism']//following::td[@id='delete']/i will hang TagUI process and not return control until user clicks OK.

[maheswarGR]

yes i tried .but facing a problem there.
delete.png -

but it is clicking on close browser . How can i avoid this ?

My page :
My page has multiple delete images. Need to click on RPA-PoC-Blueprism delete image.

Modified tag file :
............
wait 20s
click //*[@id="menuAccordion"]/div[1]/a/span

click delete.png
click ok.png

[kensoh]

Oh yes it looks very similar. Is the delete to be click on always the first row record? If the row to click on is always different it will be hard. Or is it always at the same (x,y) position on the webpage? Thinking of other ways to click on the correct X.

[marcelocecin]

hello guys, another way would be to get the position of the icon next to it (pencil) and add the width and height between them to send the mouse click command
something similar to this:

hover pencil.png
x = mouse_x() + 10
y = mouse_y() + 100
click (`x`,`y`)

[maheswarGR]

HI @marcelocecin @kensoh ..
It is not same (x,y) position on the webpage everytime ..requirement is to delete with respect to server name (i.e.,RPA-PoC-Blueprism)..so tried using xpath ..

Following script worked out:
//click //td[text()='RPA-PoC-Blueprism']//following::td[@id='delete']/i

click (1855,615)
wait 5s
hover cancel.png
click ok.png

1. Can you kindly brief reason for why control is not returning (hangs) to TAGUI process after
   > click td[text()='RPA-PoC-Blueprism']//following::td[@id='delete']/i

2. Is there any other approach to make click with respect to server name (i.e.,RPA-PoC-Blueprism) or alternate way of using xpath without hanging TAGUI process?

[kensoh]

1. The reason is this keyboard [enter] , click OK.png not working - see working example from Marcelo #1171 (comment) - Chrome by design does not return control to TagUI when there is a popup dialog.

2. Maybe you can try click Autostub using ocr by using OCR to search for a portion of the server name

[marcelocecin]

try
click //td[contains(text(),"RPA PoC with Blueprism")]/../td[9]
or
click (//td[contains(text(),"RPA PoC with Blueprism")]/../td)[9]

[maheswarGR]

sikuli screnshot - how to include restricted region in tag script
@marcelocein
@kensoh hi, just took a screenshot of an icon and selected restricted region using sikuli ..After saving it has created an png and python source file ..

1. how can i include the restricted region in tag script commands i.e., in click delete.png
2. will hover xpath wont work ?
   hover //*[@id="gatewayHistoryDetailsTab"]/div[1]/i[3] not working
3. how to click image.png within a particular region i.e.,(300,400)-(500,550)

[marcelocecin]

if you want help about python, the right thing is to create an issue in the repository https://github.com/tebelorg/RPA-Python
I tested here and it worked, follow the script:

http://3.132.253.47/dmapim/publisher/login.html?publisherName=coe
wait 5
if present('//*[@id="username_view"]')
  type //*[@id="username_view"] as admin
  type //*[@id="password"] as Thbs1234![enter]
  wait 5
  if present('Backends')
    click Backends
    wait 5
    click (//td[contains(text(),"RPA PoC with Blueprism")]/../td)[9]

[kensoh]

Oh nice, thanks Marcelo for sharing this!

Hi @maheswarGR , for your question 1 and 3 they seems to be the same question, I've replied here at the new issue you've created - #1174

For the second question on hover, you can try double-checking in Chrome web inspector to see if that XPath really exists. You can also check your Chrome is at 100% zoom. Otherwise TagUI will hover at the wrong position.

[maheswarGR]

if you want help about python, the right thing is to create an issue in the repository https://github.com/tebelorg/RPA-Python I tested here and it worked, follow the script:

http://3.132.253.47/dmapim/publisher/login.html?publisherName=coe
wait 5
if present('//*[@id="username_view"]')
  type //*[@id="username_view"] as admin
  type //*[@id="password"] as Thbs1234![enter]
  wait 5
  if present('Backends')
    click Backends
    wait 5
    click (//td[contains(text(),"RPA PoC with Blueprism")]/../td)[9]

@marcelocecin yes that worked ..we faced issue only after that alert rises, couldn't click on ok or cancel button..it hanged ..so we done last click with position (click (1829,331) and followed to click ok.png...Position may keep changing in future ..so asked of another way to achieve it without position

[maheswarGR]

Oh nice, thanks Marcelo for sharing this!

Hi @maheswarGR , for your question 1 and 3 they seems to be the same question, I've replied here at the new issue you've created - #1174

For the second question on hover, you can try double-checking in Chrome web inspector to see if that XPath really exists. You can also check your Chrome is at 100% zoom. Otherwise TagUI will hover at the wrong position.

@kensoh xpath exists and chrome is at 100% zoom .. i tried click with xpath it works..hover xpath doesn't work ..
I tried hover xpath for various existing xpaths - nothing works ..but same click xpaths works

[kensoh]

I see.. In that case, it sounds like the web element does not react to the hover event from TagUI.

You can try using visual automation to move the mouse to that location to hover. Make sure to click on Chrome browser first to bring it to the foreground and let it be in focus.

[maheswarGR]

yes .. below one works .
hover (1800,200)
hover delete.png

BUT why hover xpath doesn't work ?

[marcelocecin]

you can use keyboard commands: enter to confirm the operation or esc to cancel
don't forget to put a wait before, reason: to allow time for the confirmation screen to appear

http://3.132.253.47/dmapim/publisher/login.html?publisherName=coe
wait 5
if present('//*[@id="username_view"]')
  type //*[@id="username_view"] as admin
  type //*[@id="password"] as Thbs1234![enter]
  wait 5
  if present('Backends')
    click Backends
    wait 5
    click (//td[contains(text(),"RPA PoC with Blueprism")]/../td)[9]
    wait 5
    keyboard [enter]

[kensoh]

Adding, if zoom is already at 100%, and hover doesn't work, it means that the web element can't receive the hover event sent from TagUI. So you'll have to find other means to get to the outcome you want.

[ruthtxh]

Closing this issue due to inactivity, feel free to reopen.

[Turbolento]

@kensoh

Adding, if zoom is already at 100%, and hover doesn't work, it means that the web element can't receive the hover event sent from TagUI. So you'll have to find other means to get to the outcome you want.

Hi, bro, I have the same problem — hover xpath doesn't work , but click xpath works.
About " find other means", I don't quite understand it, I look forward to your more specific solution.

[kensoh]

Copying @ruthtxh as I've left AI Singapore and she is taking over TagUI project

[ruthtxh]

Hi @Turbolento perhaps you can try using coordinates or visual automation (screenshot image) and see if hover works for these?

hover (100,100)

hover image.png

where you can save the image.png to the same folder as your flow file

[Turbolento]

@ruthtxh Thank you very much for your enthusiastic reply.
However, in some special scenarios, using coordinates or images for positioning may be an unstable way.
So, if someone could solve this problem, I think it would be a big improvement for the tagui project.

[kensoh]

You can try to give a simple replication example of TagUI script why hover does not work, then @ruthtxh can have a look. So far, users have not feed back that hover step does not work. It may be something specific to the website you are automating.

[Turbolento]

You can try to give a simple replication example of TagUI script why hover does not work, then @ruthtxh can have a look. So far, users have not feed back that hover step does not work. It may be something specific to the website you are automating.

Thank you for your very nice advice.
This is a example rpa_for_python script:

  import rpa as r
  r.init(visual_automation=True)
  r.url("https://www.google.com/doodles")
  r.hover("//a[@id='logo']")   # not work
  r.hover("1.png")      # work well   
  r.hover(100,50)       # work well

The same thing happens with the following Tagui script:

  tagui live
  https://www.google.com/doodles
  hover //a[@id='logo']
  hover 1.png
  hover (100,200)

[ruthtxh]

Hi @Turbolento

For hover step with XPath, chrome controls the element event from the backend and upon hover the element is highlighted without the mouse moving to it visibly. For this logo link, upon hover there is no visible change. You can try this on other elements that might have visible change on hover to try this out. On the other hand, for hovering with coordinates and visual automation, the mouse event is directly controlled to move to the location.

[Turbolento]

So I can't get the element's coordinates exactly by hover xpath & echo 'mouse_xy()', right?
For code ported to a different environment, I think it is possible to get the exact coordinates of the elements via xpath.

[kensoh]

There are some internal TagUI functions that could be used to achieve this. But it is not the most commonly used, so there are not dedicated functions to do this. See below working example. chrome.mouse.getXY() is the internal function to get XY coordinates of an element on a webpage, xps666() is to convert a XPath string into a form used by TagUI underlying engine. If it is a CSS selector, xps666() is not required. Fyi @ruthtxh

https://google.com
position = chrome.mouse.getXY(xps666('//*[@name="q"]'))
echo `position.x`,`position.y`

[Sunywdev]

Hey kensoh, I have the same problem, how is this function chrome.mouse.getXY used in Rpa For Python, like this? r.send("chrome.mouse.getXY(xps666('//*[@name ='232'))")

[kensoh]

See example below. It is not straightforward, because this is not a common use case, but doable from RPA for Python -

r.url('https://google.com')
r.send('xy = chrome.mouse.getXY(xps666(\'//*[@name="q"]\'))')
r.send('dump xy.x to x.txt')
r.send('dump xy.y to y.txt')
x = int(r.load('x.txt'))
y = int(r.load('y.txt'))
print(x,y)

PS - make sure to escape double and single quotes as needed in send() so that it is valid Python string and not error

[Turbolento]

There are some internal TagUI functions that could be used to achieve this. But it is not the most commonly used, so there are not dedicated functions to do this. See below working example. chrome.mouse.getXY() is the internal function to get XY coordinates of an element on a webpage, xps666() is to convert a XPath string into a form used by TagUI underlying engine. If it is a CSS selector, xps666() is not required. Fyi @ruthtxh

https://google.com
position = chrome.mouse.getXY(xps666('//*[@name="q"]'))
echo `position.x`,`position.y`

This response from you is really great. Thank you very much.