Error while retrieving photos in wait.until(element_has_substring_or_substring((By.XPATH, "//body") #10

mahdi1234 · 2020-10-03T15:52:22Z

With b1a2a56

Google Photos : XXX
Traceback (most recent call last):              
  File "hunt.py", line 114, in <module>
    gpics(gaiaID, client, cookies, cfg)
  File "/GHunt/lib/photos.py", line 97, in gpics
    out = get_source(gaiaID, client, cookies, cfg)
  File "/GHunt/lib/photos.py", line 82, in get_source
    result = wait.until(element_has_substring_or_substring((By.XPATH, "//body"), photos_trigger, no_photos_trigger))
  File "/home/xxx/.local/lib/python3.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

The text was updated successfully, but these errors were encountered:

mxrch · 2020-10-03T15:55:51Z

I'll fix it but I want to know, how is your internet connection ? Good or not ? How much mo/s ? Because I think 30 seconds timeout is amply sufficient..

mahdi1234 · 2020-10-03T16:14:27Z

@mxrch I have 100mbit cable, no issue with internet, set timeout to 300s and still fails, could it be due to large photo gallery?

Data are incoming on 400KiB/s only (while reviews go 3.5 MiB/s).

mxrch · 2020-10-03T16:26:36Z

@mahdi1234 no it can't be that, if you have albums it will break the wait.
Either it doesn't click well on the "back" button, either it loads another language than english. Otherwise, I don't know..
Can you add these lines :

open('debug.html', 'w').write(repr(driver.page_source))
print("----START DEBUG ---")
print(driver.find_elements(By.XPATH, "//button"))
print("----END DEBUG ---")

between these two lines at line 63 in lib/photos.py :

	driver.get(f"https://get.google.com/albumarchive/{gaiaID}/albums/profile-photos?hl=en")
        [HERE]
	tmprinter.out('Fetching the Google Photos albums overview...')

And upload the debug.html file somewhere & give the link or put it in a pastebin ?

mahdi1234 · 2020-10-03T16:35:56Z

@mxrch here's the link https://upload.disroot.org/r/oz6AvU81#ZSCE6ceIJOCSkzi5Ad+LvFKj0m2uBTO/70YcSvCGglQ= will get self destruct on first download.

ItsIgnacioPortal · 2020-10-03T21:36:53Z

Same error here.
Python: 3.8.5
Chrome: Version 83.0.4103.116 (Developer Build) built on Debian bullseye/sid, running on Debian kali-rolling (64-bit)
GHunt version: Latest (ea00a24)

The only difference is mine says the error happened on line 86, but it's probably bc of the latest commits:

File "/usr/lib/python3/dist-packages/selenium/webdriver/support/wait.py", line 86, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

It only happens on target accounts on wich I get the Got the albums overview ! message.

mxrch · 2020-10-03T21:52:43Z

I don't understand since in your debug.html @mahdi1234 you have one of the two "triggers" to detect if the page finished loading, on your file there is the string "reached the end" in the body, so it should not wait.
But I have an hypothesis, maybe the EC.text_to_be_present_in_element requires the element to not have the text in the element at the moment where this code is executed.
I left this condition to be sure but I don't think it's necessary since the page is already loaded when the "Album Archive" is in the page, so I'll try to push a fix in another branch so you can test.
(sorry if I'm slow with the issues & pull requests & fixes it's the first time I need to manage a "popular" project, I learn)

mxrch · 2020-10-03T22:10:28Z

@PinkDev1 @mahdi1234 I pushed a potential fix in the i10 branch (35173d0) can you checkout on it and tell me if it fixes the issue ?

ItsIgnacioPortal · 2020-10-03T22:31:11Z

Welp, it doesn't crash anymore, so I guess it's a fix!

Google Photos : https://get.google.com/albumarchive/<REDACTED>
=> Couldn't fetch the public photos.

I've noted that you can view the albumarchive, but only when logged in with the target account. From any other account, it'll just throw a 404. I'll add a disclaimer for this, since even while having the target account's password this tool can provide plenty of automated insight.

I'd also like to note that the cookies didn't change after I logged in to my spy account on another browser, outside my VM but on the same network. There's two possibilities:

RNG magic (extremely unlikely)
They didn't change bc I logged in from the same IP

So I'll add a disclaimer for this as well. Wait for my PR

mxrch · 2020-10-03T22:40:49Z

@PinkDev1 yes it happens sometimes, it's not happening with my emails list, but Sector035 had the same problem with his account.
I don't think it's an RNG or something to do with the logged account, but with a confidentiality setting you have set, or something else obscure..
I'll add a disclaimer about it ! Unfortunately we have to deal with it while we use this bypass, but at least it works with the great majority (I think)

mxrch · 2020-10-03T22:42:54Z

Don't hesitate to contact me on Twitter with your address email you are using GHunt with, I think I can't debug it but we never know

fix issue #10

ItsIgnacioPortal · 2020-10-03T22:52:24Z

...I don't think it's an RNG or something to do with the logged account, but with a confidentiality setting you have set, or something else obscure..

That may be the case. I've tweaked a lot with the settings of my google accounts

ItsIgnacioPortal · 2020-10-03T22:55:21Z

I've noted that you can view the albumarchive, but only when logged in with the target account. From any other account, it'll just throw a 404.

Update on this: It only happens when visiting the link directly, on a browser while logged-in with the target account. But on GHunt, even when authenticated with the target's google cookies, it says Couldn't fetch the public photos. ... I'll need to do further testing

ghost · 2020-10-03T22:56:24Z

I just ran the tool and I'm getting the same crash .

ItsIgnacioPortal · 2020-10-03T22:58:31Z

I just ran the tool and I'm getting the same crash .

Well that's weird. Are you sure you downloaded the correct branch?
If unsure, delete your current GHunt and redownload with git clone -b i10 https://github.com/mxrch/GHunt.git GHunt-i10

mxrch · 2020-10-03T22:59:16Z

@PinkDev1 I already merged the fix with the master branch !

mxrch · 2020-10-03T23:00:51Z

@Adik69 can you show the output ?

mxrch · 2020-10-03T23:02:20Z

I've noted that you can view the albumarchive, but only when logged in with the target account. From any other account, it'll just throw a 404.

Update on this: It only happens when visiting the link directly, on a browser while logged-in with the target account. But on GHunt, even when authenticated with the target's google cookies, it says Couldn't fetch the public photos. ... I'll need to do further testing

@PinkDev1 Can I have the Google ID you are trying with ? Don't worry I can't get the email from this

ItsIgnacioPortal · 2020-10-03T23:47:25Z

UPDATE: I think I found the culprit:

On account A I have tweaked my settings to harden my google account as much as possible. This account has almost all personalization disabled, and uses very few google services.
When I visited account A's albumarchive on the browser, while signed in as account A, I was able to see it's photo album, but not any option to share it with anyone (more on this later).
GHunt said:

Unable to fetch connected Google services.
...
Google Photos : https://get.google.com/albumarchive/<REDACTED>
=> Couldn't fetch the public photos.

However, on account B the settings are pretty much default, and it's prob 8~ years old. This account uses many google services and has linked many accounts.
When I visited account B's albumarchive on the browser, while signed in as account B, I was able to see it's photo album, and an option to share it with other users:

Clicking on more information ->

"More Information" link

GHunt said:

Activated Google services :
- Photos

Google Photos : https://get.google.com/albumarchive/<REDACTED>
=> Couldn't fetch the public photos.

It seems GHunt needs this setting to be turned on. I assume this setting has been turned on on accounts that have used Picasa, Picasa Web Albums, or the Picasa Web Albums API. My theory is that the images stored on Picasa were archived on the user's Google Photos, and then his account was automatically linked to Picasa in order to keep the service working. Therefor, GHunt will succeed on scrapping these account's albumarchive.

It's probably extremely uncommon for a user to manually go (and somehow find) their albumarchive, and turn on that setting.

See: https://support.google.com/picasa/answer/6383491

I haven't tested this myself bc AFAIK, there's no option to turn it off, and I don't want to expose my personal accounts.
Can you verify if my theory is true?

PS: No, I won't give you any google ID, I'm way too paranoid 😂

mahdi1234 · 2020-10-04T08:13:10Z

@mxrch I confirm it fixed the issue on two accounts I had it.

ItsIgnacioPortal · 2020-10-06T19:56:54Z

Well, this issue was fixed on 35173d0, and I've already opened a PR to add a Disclaimer to the README.
@mxrch this issue can finally be closed :)

Added back the Google Maps advisory Re-worded some stuff to make it more neutral Made hyper-link to issue mxrch#10

mxrch added the bug Something isn't working label Oct 3, 2020

mxrch mentioned this issue Oct 3, 2020

Stuck on "Got the albums overview !" #13

Closed

mxrch added a commit that referenced this issue Oct 3, 2020

fix issue #10

35173d0

mxrch added a commit that referenced this issue Oct 3, 2020

Merge pull request #17 from mxrch/i10

81805f7

fix issue #10

ItsIgnacioPortal mentioned this issue Oct 3, 2020

Added disclaimers to README #18

Closed

ItsIgnacioPortal mentioned this issue Oct 8, 2020

Added disclaimer/explanation for some features on README.md & Code analisys with CodeQL #55

Merged

mxrch closed this as completed Oct 9, 2020

ItsIgnacioPortal added a commit to ItsIgnacioPortal/GHunt that referenced this issue Oct 9, 2020

Corrections and aditions

81e6405

Added back the Google Maps advisory Re-worded some stuff to make it more neutral Made hyper-link to issue mxrch#10

ralghamdi mentioned this issue Mar 23, 2022

unknown error: DevToolsActivePort file doesn't exist #317

Closed

juggel90 mentioned this issue Apr 22, 2022

TimeoutException scraping Google Maps reviews #329

Closed

fraspace5 mentioned this issue May 4, 2022

Error #337

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error while retrieving photos in wait.until(element_has_substring_or_substring((By.XPATH, "//body") #10

Error while retrieving photos in wait.until(element_has_substring_or_substring((By.XPATH, "//body") #10

mahdi1234 commented Oct 3, 2020

mxrch commented Oct 3, 2020

mahdi1234 commented Oct 3, 2020 •

edited

Loading

mxrch commented Oct 3, 2020

mahdi1234 commented Oct 3, 2020

ItsIgnacioPortal commented Oct 3, 2020

mxrch commented Oct 3, 2020

mxrch commented Oct 3, 2020

ItsIgnacioPortal commented Oct 3, 2020 •

edited

Loading

mxrch commented Oct 3, 2020

mxrch commented Oct 3, 2020

ItsIgnacioPortal commented Oct 3, 2020

ItsIgnacioPortal commented Oct 3, 2020

ghost commented Oct 3, 2020

ItsIgnacioPortal commented Oct 3, 2020

mxrch commented Oct 3, 2020

mxrch commented Oct 3, 2020 •

edited

Loading

mxrch commented Oct 3, 2020 •

edited

Loading

ItsIgnacioPortal commented Oct 3, 2020 •

edited

Loading

mahdi1234 commented Oct 4, 2020

ItsIgnacioPortal commented Oct 6, 2020

Error while retrieving photos in wait.until(element_has_substring_or_substring((By.XPATH, "//body") #10

Error while retrieving photos in wait.until(element_has_substring_or_substring((By.XPATH, "//body") #10

Comments

mahdi1234 commented Oct 3, 2020

mxrch commented Oct 3, 2020

mahdi1234 commented Oct 3, 2020 • edited Loading

mxrch commented Oct 3, 2020

mahdi1234 commented Oct 3, 2020

ItsIgnacioPortal commented Oct 3, 2020

mxrch commented Oct 3, 2020

mxrch commented Oct 3, 2020

ItsIgnacioPortal commented Oct 3, 2020 • edited Loading

mxrch commented Oct 3, 2020

mxrch commented Oct 3, 2020

ItsIgnacioPortal commented Oct 3, 2020

ItsIgnacioPortal commented Oct 3, 2020

ghost commented Oct 3, 2020

ItsIgnacioPortal commented Oct 3, 2020

mxrch commented Oct 3, 2020

mxrch commented Oct 3, 2020 • edited Loading

mxrch commented Oct 3, 2020 • edited Loading

ItsIgnacioPortal commented Oct 3, 2020 • edited Loading

mahdi1234 commented Oct 4, 2020

ItsIgnacioPortal commented Oct 6, 2020

mahdi1234 commented Oct 3, 2020 •

edited

Loading

ItsIgnacioPortal commented Oct 3, 2020 •

edited

Loading

mxrch commented Oct 3, 2020 •

edited

Loading

mxrch commented Oct 3, 2020 •

edited

Loading

ItsIgnacioPortal commented Oct 3, 2020 •

edited

Loading