Cannot start emailproxy.py

[0x7464]

Hi,

Following your readme - everything went fine until I tried to start emailproxy.py and got the following error below, please forgive me if this is something trivial - I'm not a programmer :)

Error:

Traceback (most recent call last):
File "emailproxy.py", line 26, in
import pystray
File "/usr/local/lib/python3.6/dist-packages/pystray/init.py", line 48, in
Icon = backend().Icon
File "/usr/local/lib/python3.6/dist-packages/pystray/init.py", line 40, in backend
return importlib.import_module(package + '._' + module)
File "/usr/lib/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/usr/local/lib/python3.6/dist-packages/pystray/_appindicator.py", line 19, in
gi.require_version('Gtk', '3.0')
File "/usr/lib/python3/dist-packages/gi/init.py", line 130, in require_version
raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace Gtk not available

OS: Ubuntu 18.04 LTS
Python: 3.6.9

Things I tried:
sudo apt-get install libgtk-3-dev
sudo apt install python3-gi gobject-introspection gir1.2-gtk-3.0

Thanks for any help you can give - your app sounds really great and could save me a lot of headaches.

[simonrob]

Thanks for reporting this. As noted in the readme I haven't actually properly tested this script on anything other than macOS, but I'd like to make it work as widely as possible if others find it useful.

I tried Ubuntu 18.04.3 in a virtual machine, and ran into a similar error to the one you have reported. The problem is caused by pystray, which is used to display the menu bar icon. I was able to resolve this with sudo apt install gir1.2-ayatanaappindicator3-0.1, and the proxy works fine afterwards. Please could you try this and let me know whether it works for you?

While investigating this problem I took the opportunity to make a few Linux-targeted improvements, so it is also worth you updating from this repository to get the latest version. There does still seem to be a lingering issue with reloading the configuration file after changes, so for the time being it is best to just restart the script each time until I've had chance to investigate this.

[0x7464]

Hi Simon,

Thanks a lot for your reply. I was able to install the "gir1.2-ayatanaappindicator3-0.1" package, however now when I run emailproxy.py I get the following error:

Traceback (most recent call last):
File "emailproxy.py", line 26, in
import pystray
File "/usr/local/lib/python3.6/dist-packages/pystray/init.py", line 48, in
Icon = backend().Icon
File "/usr/local/lib/python3.6/dist-packages/pystray/init.py", line 40, in backend
return importlib.import_module(package + '._' + module)
File "/usr/lib/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/usr/local/lib/python3.6/dist-packages/pystray/_appindicator.py", line 22, in
gi.require_version('AppIndicator3', '0.1')
File "/usr/lib/python3/dist-packages/gi/init.py", line 130, in require_version
raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace AppIndicator3 not available

I then did a bit of googling and tried installing gir1.2-appindicator3-0.1. Now when I run emailproxy.py it starts and looks like its working with the following output:

python3 emailproxy.py
2021-04-29 21:44:49: Starting IMAP server at localhost:1995 proxying imap.gmail.com:993
2021-04-29 21:44:49: Initialised Email OAuth 2.0 Proxy - listening for authentication requests

(emailproxy.py:24462): Gdk-CRITICAL **: 21:44:49.792: gdk_window_thaw_toplevel_updates: assertion 'window->update_and_descendants_freeze_count > 0' failed

I can also see an "OA" mail icon at the top righthand side of my screen also, but I cannot interact with it in any way. I've setup all the other parts but when it comes to the authorizing - I get a popup window basically saying "Please authorize your account from the menu" and I can click it but it doesn't seem to do anything - maybe this is because I cannot interact with the button ? which may open menus allowing me to authorize ?

A thought Simon, would it be possible to allow these interactions like allowing authorization etc to be accepted via command-line - therefore negating any problems accessing the menu system ?

As always - thanks for your help.

[simonrob]

Thanks for following up. At the moment the icon is quite a core part of the script because it handles launching the popup browser window that is required for authentication. I could certainly extract it, but may not have time in the near future to do this.

Thanks also for the pointer to gir1.2-appindicator. The problem seems to still be linked to the use of pystray, and may actually be a bug I introduced in a recent commit that is related to the different behaviour of macOS vs. Linux platforms. I've looked again, and actually had to remove and re-install gir1.2-appindicator to get rid of that error, but after doing that I can't replicate either this or the lack of a working menu. Could you try again with the latest commit?

[0x7464]

Hi Simon,

Seems your latest commit has changed the startup, which is now failing - giving the following error:

Traceback (most recent call last):
File "emailproxy.py", line 27, in
import timeago as timeago
ModuleNotFoundError: No module named 'timeago'

I did a 'pip3 install timeago' to get rid of this error as it looks like I just didn't have the module.

I then reloaded emailproxy.py and I can now interact with and click on the OA button bringing up menus etc.
However when it comes to authorization - when I click the authorization popup it seems to do nothing and when I go to the menus and click Authorize account and click on my account it brings back the following error:

[pywebview] GTK cannot be loaded
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/webview/guilib.py", line 16, in import_gtk
import webview.platforms.gtk as guilib
File "/usr/local/lib/python3.6/dist-packages/webview/platforms/gtk.py", line 28, in
gi.require_version('WebKit2', '4.0')
File "/usr/lib/python3/dist-packages/gi/init.py", line 130, in require_version
raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace WebKit2 not available
[pywebview] QT cannot be loaded
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/webview/guilib.py", line 29, in import_qt
import webview.platforms.qt as guilib
File "/usr/local/lib/python3.6/dist-packages/webview/platforms/qt.py", line 29, in
from PyQt5 import QtCore
ModuleNotFoundError: No module named 'PyQt5'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pystray/_base.py", line 267, in inner
callback(self)
File "/usr/local/lib/python3.6/dist-packages/pystray/_base.py", line 368, in call
return self._action(icon, self)
File "emailproxy.py", line 1034, in authorise_account
webview.start(self.handle_authorisation_windows)
File "/usr/local/lib/python3.6/dist-packages/webview/init.py", line 96, in start
guilib = initialize(gui)
File "/usr/local/lib/python3.6/dist-packages/webview/guilib.py", line 93, in initialize
raise WebViewException('You must have either QT or GTK with Python extensions installed in order to use pywebview.')
webview.util.WebViewException: You must have either QT or GTK with Python extensions installed in order to use pywebview.

I then ran 'pip3 install --upgrade pip' and after that 'pip3 install pyqt5'. Then the error changed to this:

[pywebview] GTK cannot be loaded
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/webview/guilib.py", line 16, in import_gtk
import webview.platforms.gtk as guilib
File "/usr/local/lib/python3.6/dist-packages/webview/platforms/gtk.py", line 28, in
gi.require_version('WebKit2', '4.0')
File "/usr/lib/python3/dist-packages/gi/init.py", line 130, in require_version
raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace WebKit2 not available
[pywebview] QT cannot be loaded
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/webview/platforms/qt.py", line 38, in
from PyQt5.QtWebEngineWidgets import QWebEngineView as QWebView, QWebEnginePage as QWebPage
ModuleNotFoundError: No module named 'PyQt5.QtWebEngineWidgets'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/webview/guilib.py", line 29, in import_qt
import webview.platforms.qt as guilib
File "/usr/local/lib/python3.6/dist-packages/webview/platforms/qt.py", line 43, in
from PyQt5 import QtWebKitWidgets
ImportError: cannot import name 'QtWebKitWidgets'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pystray/_base.py", line 267, in inner
callback(self)
File "/usr/local/lib/python3.6/dist-packages/pystray/_base.py", line 368, in call
return self._action(icon, self)
File "emailproxy.py", line 1034, in authorise_account
webview.start(self.handle_authorisation_windows)
File "/usr/local/lib/python3.6/dist-packages/webview/init.py", line 96, in start
guilib = initialize(gui)
File "/usr/local/lib/python3.6/dist-packages/webview/guilib.py", line 93, in initialize
raise WebViewException('You must have either QT or GTK with Python extensions installed in order to use pywebview.')
webview.util.WebViewException: You must have either QT or GTK with Python extensions installed in order to use pywebview.

For which I ran a 'pip3 install PyQtWebEngine' - and now it seems I can interact with the menus fully ! - I've just authorized the mail account (however I did get a Authorization Error 400: redirect_uri_mismatch - but I think that's just a misconfiguration in my configuration file).

I'll test it end to end over the coming days and let you know how it goes (as I guess it's your Sunday now), have a good weekend and thanks for your help.

[0x7464]

Hi Simon,

I did some more testing - and it seems no matter what I do I cant get the authorization to work with Gmail. It comes back with the "Couldn't sign you in - this browser or app may not be secure. Try using a different browser. If you're already using a supported browser, you can refresh your screen and try again to sign in".
Is it possible for you to test this still works in your environment - I have read a few different pages about google removing functionality for security reasons (but it may also be a misconfiguration on my part also).

Another question Simon - when authorizing an account window popups - what app is that - is it just a python window or does it pipe to some browser? As this could be a problem if its not considered supported by security functionality by google. If it is a python app - are you able to make it pop an actual chrome/browser window ? As when I hit the OAUTH 2.0 gmail login page with the browser, everything seems to work fine.

Cheers

[simonrob]

Thanks for the update - great to hear you got the menus and webview popup working. I will add a link to this issue from the readme to help others who might encounter a similar thing. Sorry about the timeago issue - I should have mentioned updating from the requirements file after getting the new version of the script.

The Authorization Error 400: redirect_uri_mismatch is the key part here - while it may seem like you've authorised successfully, this is a critical error as it is how the script receives the authorisation tokens and is allowed to use them to transparently log in for you. I haven't actually tested with a redirect_uri (in emailproxy.config) value of anything other than http://localhost, which is the default for client applications like this one, so this could be a related issue. What is your redirect_uri?

You can check whether your account has been successfully authorised by looking in emailproxy.config. If it was successful, your account details will have been updated with values like token_salt, access_token, access_token_expiry, refresh_token and last_activity. If these values are not there then authentication was not successful, even if you did manage to log in properly via the popup window.

The account popup is required because after you log in in the script needs to get the values contained in the redirect_uri mentioned above. Another option (which is used by, e.g., DavMail) is to open the link in the default browser and then pop up a text box for you to manually past the resulting values, but I found this to be confusing, so chose the web view method. I agree that it is not ideal for the notification to not actually do anything when clicked, but this is itself related to pystray, python and threading, and its inability to trigger an action on click. Luckily authentication is mostly a one-time thing, so for the time being this is not a major issue.

Would you be willing to share your configuration file with me privately so that I can investigate further?

[0x7464]

Hi Simon,

I got past the Authorization Error 400: redirect_uri_mismatch - it was a simple mistake - as I'm only using localhost as a test also.
Sure I can give you my configuration file. How would you like it to be given to you ?

[simonrob]

Great - email is fine; my GitHub username at gmail.

[0x7464]

Thanks Simon - sent you an email.

[simonrob]

Thanks for sending your configuration over. I wasn't able to log in using the details in your file, but this was because your Google API client is in testing mode, and restricted to a limited selection of email addresses. To verify that this is not a limitation of testing mode itself, I created my own new API client, and it works as expected.

I wonder whether you have an API client that is somehow set up incorrectly? Google's Cloud Console is incredibly confusing at times, and it's very easy to do the wrong thing. I will also clarify the readme to explain the type of API client needed. Here's a summary of what I did to get a working client ID and secret:

• Go to the credentials page and click Create Credentials at the top
• Select OAuth client ID, and on the next page select Desktop app. Enter a name in the box that appears (this can be anything you like). Once you click Create you are given the client ID and secret to use with the proxy script

You may find that Google forces you to create your OAuth consent screen before you can finish the steps above. This is also an important part of the process because it is where you define what your client ID and secret can actually access. If you had already done this before, it's worth going back through it to check. There are three key screens on the consent page:

• The first screen collects details about your API client app. This can mostly be whatever you like, and only the name and contact details are mandatory. If your usage is anything like mine, this client will only be used for your own email, so there's no need to give things like icons or the extra optional detail. You don't need to add an app domain or authorised domains.
• The second screen is the most important - here you set the scopes your client will be able to access. Click the Add or Remove Scopes button at the top, and another menu will appear. There is a box at the bottom of the popout menu to manually add scopes - here you should enter https://mail.google.com/ (as shown in the sample configuration file). Save this and then continue to the next screen.
• The third screen asks for test user email addresses. Enter any email addresses that you want to use with the proxy. After this, the final screen is just a summary - click Back to Dashboard at the bottom.

I noticed in your sample config file that you only had SMTP enabled, and also that you had a duplicate entry for the same email address. This was commented out, but just to be clear, email address configuration sections must be unique. You don't need to duplicate them for IMAP/SMTP.

In case it helps, I find it is often easier to test things without an email client, and just use telnet for a quick check. Using the sample configuration file, you could test your Gmail login configuration from a terminal window by starting a connection using telnet localhost 1995, and then entering a01 login e@mail.com password. Replace e@mail.com with your email address, and password with any value you like during testing (see this repository's readme for an explanation of why the password is irrelevant). Once this command is sent, you should get an authentication notification from the proxy, and once authenticated you will see a response in the terminal similar to a01 OK e@mail.com authenticated (Success).

[0x7464]

Hi Simon,

Thanks a lot for your reply - I have re-setup everything as per your instructions but still come to the same problem where it brings up a login screen after clicking the authorization button and then if I type in my mail address in the field and click next it comes back saying 'this browser or app may not be secure'.

I have also tried to do the telnet test way. However I do not understand how you are getting it to work with the commands you have mentioned.
"login" is not a valid command when I try it.

220 smtp.gmail.com ESMTP x20sm10371689pjp.12 - gsmtp
ehlo smtp.gmail.com
250-smtp.gmail.com at your service, [IP.Address.Mine.Redacted]
250-SIZE 35882577
250-8BITMIME
250-AUTH PLAIN LOGIN-CLIENTTOKEN OAUTHBEARER XOAUTH
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-CHUNKING
250 SMTPUTF8

I have now tried it with login but using IMAP as the protocol and your telnet commands do work - however they end up with the same issue - I get the popup asking me to authorize from the app icon - I do so through the menus - then I get the popup that is a gmail login window and when I put my username in and click Next it says the same 'this browser or app may not be secure' error.

I think I may of figured it out - in the logs it shows the below error:

2021-05-04 08:49:40: Authorisation request received for @gmail.com
[pywebview] GTK cannot be loaded
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/webview/guilib.py", line 16, in import_gtk
import webview.platforms.gtk as guilib
File "/usr/local/lib/python3.6/dist-packages/webview/platforms/gtk.py", line 28, in
gi.require_version('WebKit2', '4.0')
File "/usr/lib/python3/dist-packages/gi/init.py", line 130, in require_version
raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace WebKit2 not available

I then read up on what WebKit2 does - and ran a 'apt install gir1.2-webkit2-4.0'. As WebKit was essentially the browser engine passing the information on, this is where it was failing to get to the next stage. I can now successfully authorize a request !

*N.B - Also sorry I wasn't paying attention but in the emailproxy.config file there is a typo in the token_url line as it reads like the following (this is only for the gmail configuration):
token_url = token_url = https://oauth2.googleapis.com/token

Thanks a lot for your help Simon.

[simonrob]

Excellent!

So, just to put it here for the benefit of others, the key pystray fix was sudo apt install gir1.2-appindicator. For pywebview it's probably best to read the dependencies page of that project, but the discusion above also has other pointers to potential errors you might see, and the related packages that can be installed to fix them.

Thanks for noting the typo in the example file; now fixed. And yes, my telnet instructions above are for an IMAP server. SMTP login is not quite as simple to test using telnet as you need to Base64 encode the username and password that you send. However, it is a fairly safe assumption that if IMAP authentication works, SMTP will too. The only additional complexity with SMTP is that there is a starttls configuration option, but if you don't know the correct setting here just try one option (i.e., True or False), and use telnet to try to connect – the connection will fail if this setting is wrong, so just switch to the other option.

It's great to hear this project is useful and now working on other platforms. Let me know how you get on.

[BahodurSaidov]

While using python docker, there is error with pystray and xlib.
So the workaround was to comment "import pystray.py" in emailproxy.py

[simonrob]

@BahodurSaidov: please take a look at the new branch discussed in #54. Let me know if this simplifies dependency setup for you?

[rickgitdone]

running on Ubuntu 22.04 and so far fully passed thru using the telnet method to authorize my gmail oauth creds :

using IMAP initially I had a bad video setup but passed by that by loading another driver. I successfully authorized my client_id to allow the API to work. now I should be able to use the oauth creds?? Now when using the smtp connection the proxy isn't showing any progress as it did for IMAP which allowed me to authorize,

not sure where to go from here

smtp connection debug:
(unsecured) proxying smtp.gmail.com:465 (STARTTLS)
--> [ Client connected ]
<-- [ Server disconnected ]
--> [ Client disconnected ]

from my exim4 smtp side:

SMTP error from remote mail server after pipelined end of data: 530-5.7.0 Authentication Required. For more information, go to\n530 5.7.0 https://support.google.com/mail/?p=WantAuthError gf1-20020a0561224e8100b004d89243da66sm60905vkb.53 - gsmtp

do I need another authorization?

[simonrob]

It looks like you're not actually logging in with the required SMTP commands. Please post the full debug log as a new issue if this still happens once you configure your client to log in to the server.