Possible for Chrome extension to record actions in Gmail - no, and then how

[lookang]

Hi ken,
is it possible for the chrome-extension://egdllmehgfgjebhlkjmcnhiocfcidnjk/tagui.html to record in gmail actions?
i tried but the export only capture only one move
https://mail.google.com/mail/u/0/#inbox

the other actions are not recorded in text by the chrome-extension://egdllmehgfgjebhlkjmcnhiocfcidnjk/tagui.html
like
click email with title "notifications"
copy number 123456 depending on the latest email

the reason is i need to automate the 2 factor authentication sent to my gmail

[kensoh]

Hi @lookang, the Chrome extension is built on another open-source project for recorder which cannot record frames. Web frames is a way of doing websites which will segment a webpage into different parts and then load and render them separately. TagUI supports the use of frames - https://tagui.readthedocs.io/en/latest/reference.html#frame

However, to do it by inspecting which frame to use and which web element to refer to may take up too much effort, as compared to using visual automation method and keyboard shortcuts. For eg, you could map your mental process of opening the email and copying out the 2FA code, then do it directly using TagUI steps.

You can make use of image snapshots as identifiers (including editing to remove areas of dynamic content and save as .png), (x,y) co-ordinates and keyboard shortcuts to navigate Gmail to copy out the 2FA code eg keyboard [ctrl]c. After the code is copied out to clipboard, you can access it using clipboard() function, for eg below, where 2fa_textbox is the identifier where to type the code before going to the next step -

2fa = clipboard()
type 2fa_textbox as `2fa`

Ping us when you run into any questions or issues! You could also join us at this week's Live Q&A and we could look at it together - #914

[kensoh]

Video of more context on above recommendations - https://youtu.be/fLwOqK1y0uA?t=1254

[lookang]

@kensoh is there an example *.tag for visual " image snapshots as identifiers (including editing to remove areas of dynamic content and save as .png),"

anyway, i manage to automate until the gmail identifying the XXXXXX 6 digits code for copying and pasting into the other website.

below is the comments
// preconditions by lookang
// use the chrome that tagUI open and key in website username and website password
// chrome will offer to save password, click yes
//similarly when visiting gmail again enter your gmail and gmail password and save them on the browser that tagUI Opens
// when these are all done, there is not need to tagUI username password etc , just pure action code

[kensoh]

That's great! Using the Gmail example, consider the following image. Telling TagUI to use this to click on the first message on top will not work, because the content will be different every time and not result in a match.

In the following image, the content is deleted and cleared away, before saving as .png file (which stores transparency info). This preserves the part of the image which TagUI will use to perform the match, while ignoring the content area. Using this will let TagUI click on the first match found from the top left of the screen and work.

Note above image the content portion is not white, it is transparent. It is showing white because the GitHub page is white.

Copying my team-mate @ruthtxh for info.

[lookang]

@kensoh we used Xpath instead of OCR.
Thank you for the awesome tool.
I have started https://github.com/lookang/TagUI/blob/main/login/loginSLSCGadmin.tag
for the full tag code.
Below is a sample of how we did tagui on chrome gmail

//INSTALLATION
// https://tagui.readthedocs.io/en/latest/setup.html

// HOW TO USE
// copy the text here and create a text file and save it into your TagUI folder
// for me it was /Users/lookang/Desktop/tagui/flows/lookang/20210208
// to run the code, 
// use cmd line in Windows or terminal in MacOSX
// cd /Users/lookang/Desktop/tagui/flows/lookang/20210208
// type $  tagui loginSLSCGadmin.tag
// depending on your file name saved as loginSLSCGadmin.tag

// preconditions by lookang and shaun
// use the chrome that tagUI open and key in website username and website password
// chrome will offer to save password, click yes
//similarly when visiting gmail again enter your gmail and gmail password and save them on the browser that tagUI Opens
// when these are all done, there is not need to tagUI username password etc , just pure action code
// another condition is in the gmail, delete all previous  Verification Code for SLS Account Login, the script is not clever to scroll down to find the latest email

// WHAT THIS CODE DOES:
// 1. Visit SLS to trigger the 2FA, login is handle by Chrome password save 
// 2. Opens gmail, login is handle by Chrome password save
// 2b. Open the first email 
// 3. Get the 2FA code  

///////////////////////////////////

//Enter in to SLS
https://vle.learning.moe.edu.sg/login

//using TagUI Web Automation
//type username as XX
//type password as XX
click .button.login
wait 5

///////////////////////////////////

// PRECONDITION: allow popups on SLS site
// opens gmail in new window
dom gmail_window = window.open("https://mail.google.com/mail/u/0/#inbox")

code = ""

popup mail
	// If this is the first time logging into google from this browser, attempt to sign in
	if url() contains 'https://www.google.com/intl/en-GB/gmail/about/'
		
		// Click Sign in 
		click /html/body/div[2]/div[1]/div[4]/ul[1]/li[2]/a

		popup signin
			type //*[@id="identifierId"] as xx@gmail.com // replace xx with your own email address
			click Next
			type //*[@id="password"]/div[1]/div/div[1]/input as xx // replace xx with your own email password
			click Next
			wait 20


	wait 5
	// click the first email by notifications@sls.ufinity.com in the table
	click //*[@email="notifications@sls.ufinity.com"]/ancestor-or-self::tr

	// read the 2fa code 
	//line below is what i copied from the developer Xpath full path
	// read the 2fa code 
	read //div[7]/div[3]/div/div[2]/div[1]/div[2]/div/div/div/div/div[2]/div/div[1]/div/div[2]/div/table/tr/td[1]/div[2]/div[2]/div/div[3]/div/div/div/div/div/div[1]/div[2]/div[3]/div[3]/div/div[2]/h2 to code

	echo `code`
	wait 5 // if wrong i can manually copy from browser to continue debugging

// close gmail window
dom gmail_window.close()

type otp as `code`
click .field-set.type-button.otp-submit button

wait 500 //to load the SLS page correctly first before next step

[kensoh]

Wow, thank you for being an awesome user of the tool! Impressive stuffs you created here 😲 I have never ever come across someone using it this way for 2FA handling, brilliant haha.

Copying my team-mate @ruthtxh on your dom + popup + gmail method to handle 2FA,
and your full flow file with sophisticated logic in JavaScript.