Gmail OAuth and Standard PHP IMAP

[freescout-helpdesk]

• Starting February 15, 2021, G Suite accounts will only allow access to apps using OAuth
• June 15, 2020 - Users who try to connect to an LSA (non-oauth) for the first time will no longer be able to do so.

(freescout-help-desk/freescout#390)

1. There is no way to use standard PHP IMAP extension https://www.php.net/manual/en/ref.imap.php to connect to Gmail using OAuth?

2. It will be impossible to access Gmail also via POP3 protocol without OAuth?

[freescout-helpdesk]

If both are true, it means all PHP projects using standrard PHP IMAP functions will need to redesign their projects to use Zend Mail.

Just on the GitHub currently there are 121,653 scripts using standard PHP IMAP functions.

Most of them will not bother to redesign their projects to use Zend Mail, they will just ask G Suite users to stay away from their apps.

PHP IMAP is standard extension written in C, it is fast and reliable. It is just weird to switch to third-party library.

So if since February 15, 2021 all IMAP requests to G Suite will be made via OAuth why don't Google just allow to pass OAuth token as an IMAP password (https://developers.google.com/gmail/imap/xoauth2-protocol)? In this case all existing IMAP libraries will continue to function.

Why to reinvent the bicycle and force the whole world to use SASL XOAUTH2 protocol when the goal can be achieved within the standard IMAP authentication?

All Google need to do is to receive OAuth token in IMAP password for G Suite accounts. This is very simple, obvious, logical and elegant solution causing minumum headache to other developers. Google even can keep using SASL XOAUTH2.

[TomasVotruba]

Most of them will not bother to redesign their projects to use Zend Mail, they will just ask G Suite users to stay away from their apps.

That can be solved by automated upgrade tools.
Number of projects using old technology is not a valid reason to not use better one. Once there were more horses than cars. But is that a reason to walk to another state?

What I'm more interested are the reasons behind it. Could you share them?

[freescout-helpdesk]

That can be solved by automated upgrade tools.

We are afraid, not in this case. Zend Mail and PHP IMAP are not compatible. For some projects it will lead to months of development, debugging and bugfixing (like in case of our project).

What I'm more interested are the reasons behind it. Could you share them?

We don't want to waste time switching to the third-party library when the native PHP extension can be used. This is ridiculous... When starting the project we've intentionally chosen PHP IMAP to make the app as fast as possible and now Google makes the whole PHP world to stop using PHP IMAP extension. Should we sacrifce performance and common sense and abandon PHP IMAP extension?

Number of projects using old technology is not a valid reason to not use better one.

This is an example when the new authorization technology (OAuth) can be used within the standard IMAP authorization without causing a big headache to thouthands of developers.

[TomasVotruba]

We are afraid, not in this case. Zend Mail and PHP IMAP are not compatible.

Refactoring/migration isn't about compatibility. It's about transition from A to B.
They're both mailing concepts, which allows transition. It's not perfect, it will require some work, but it's possible to apply over whole Github code bases for every single use.

[freescout-helpdesk]

Check out https://packagist.org/packages/php-imap/php-imap package with over 1M installs. It is using standard PHP IMAP. More than 1M projects utilizing this package will be unable to connect to G Suite Gmail accounts via IMAP, because G Suite OAuth is not compatible with PHP IMAP.

The idea to make IMAP interaction more secure is good, but it can be implemented in a more considerate way.

@TomasVotruba are you in contact with Google developers? It would be good to know what is the reason not to pass OAuth token as IMAP password when OAuth token does serve as a password during IMAP authentication.

[TomasVotruba]

Again, the number of installs and spread is irelevant for innovation.
I know it sucks, but that how innovation works. People don't like change, that's how our brains work.

If you're able to show me migration path, I can help with automation for all Github repositories that use it.

[tedivm]

This isn't "innovation", this is Google breaking standards. Innovating would be evolving the standards- this is just a power grab by Google.

[tedivm]

@freescout-helpdesk - my imap package only has half a million installs, but I've already been getting requests to deal with this situation.

It's not just a programming issue either. My company is currently using Google Suites, and we have to turn off this OAuth requirement for our users to be able to use their existing calendar and email clients.

[molbal]

This is not evolving and not innovation. This is forcing one standard and dropping support of another tool (The IMAP library) If using IMAP would be a real security risk or it had known vulnerabilities I would understand this move from Google. Developers will gain nothing from switching to another library, because most (or all?) functions can be done with IMAP too.

But if this rule gets enforced then thousands of developers will have to work hours just to RETAIN current functionality. It might be automatized, or it might be manual work or somewhere in between, but even if it's just 20 minutes of work, that 20 minutes could go towards improving the project and not refacing part of a project forced by a 3rd party.

[tedivm]

Yeah, if google cared about innovation here they'd work to evolve the standards. If they cared about making this easy on developers they'd use some of their resources to start upgrading projects directly.

Instead what they're doing is enforcing a monopoly while hoping a bunch of developers will give them free labor.

[TomasVotruba]

I see there is no need for change, just complaining about it. I can't help with that.
If you're ever open to a change, let me know. I help huge migrations on daily bases.

[hopeseekr]

Can someone please tell me how the NodeJS, C#, Java, Python, and (especially since it's Google) Go Languages are handling this?

EDIT: @TomasVotruba, you are exceedingly tone deaf on this issue. I recommend that you let someone else speak on this.

[sombatos]

No way I'm gonna abandon PHP IMAP extension! To abandon official PHP extension is insanity!

[freescout-helpdesk]

Google's intention to make the world of IMAP more secure may be good, but it is done in an upside down way.

The proper procedure for this would be:

1. Make sure PHP IMAP extension supports OAuth.
2. Publish instructions for developers on how to authenticate via OAuth in PHP IMAP.
3. Developers make minor changes in imap_open function calls.
4. End users are happy.
5. Everybody happy.

But in reality we've got an upside down workflow:

1. Google did not make sure that PHP IMAP extension supports OAuth.
2. Google suggested developers to switch from the official PHP IMAP extension to the third-party library.
3. Developers suffer because of tons of work they have to do to migrate their projects from the official PHP IMAP extension to the third-party library (this means to change not just one function call, but to refactor the whole code).
4. End users suffer because they get a performance drop due to a switch from the native PHP extension written in C to the third-party library.
5. Nobody is happy.

Google had many years to make sure transition to OAuth in PHP IMAP will go smoothly, but failed by some reason. So now it is developers and end users who have to suffer.

So maybe best for PHP developers would be not to make any changes in their projects using PHP IMAP and let Google perform their part of the work first - make sure that Gmail IMAP OAuth and PHP are compatible. Don't worry, just be patient, now G Suite is interested in smooth transition more than anyone else.

[freescout-helpdesk]

Here is the discussion on PHP.net:

• https://wiki.php.net/todo/ext/imap?s[]=imap
• https://bugs.php.net/bug.php?id=78572

They are discussing a further plan of maintaining the PHP IMAP extension. One of the alternatives discussed is Horde/Imap_Client which probably supports OAuth2. The library can be installed via the PEAR installer or Compoer. But documentation looks quite poor.

[hopeseekr]

Well, I did my own research:

1. NodeJS + GMail + IMAP + OAuth2: https://www.example-code.com/nodejs/gmail_imap_login_oauth2.asp
2. Python + GMail + IMAP + OAuth2: https://developers.google.com/gmail/imap/xoauth2-libraries
3. C# + GMail + IMAP + OAuth2: https://www.limilabs.com/blog/oauth2-gmail-imap-installed-applications
4. Java + GMail + IMAP + OAuth2: https://javaee.github.io/javamail/OAuth2 and https://developers.google.com/gmail/imap/xoauth2-libraries
5. Go + GMail + IMAP + OAuth2:

Maybe the PHP devs should implement OAuth2 for the IMAP extension? But then, I did more research and apparently this new OAuth2 route will require businesses to pluck down $15,000, minimum, for penetration testing? And then I realize that this is probably the biggest antitrust thing I've witnessed in the last 10 years.

[molbal]

I agree that building a drop in replacement library might be our best choice. If Microsoft does the same then all the reasons we listed here also applies to their decision of course.

[freescout-helpdesk]

The IMAP extension is based on a C library that has not been maintained for the last decade. As such, it is highly unlikely that the IMAP extension will ever see new features.

In fact, it is quite likely that the extension will be removed from the PHP distribution for PHP 8, though no decision has been made on this yet. I would encourage anyone still using the IMAP extension to instead switch to one of the pure-PHP mailing libraries. I'm not particularly familiar with mailing libraries, so don't know which of them already support XOAUTH2...

https://news-web.php.net/php.internals/107950

[freescout-helpdesk]

Ideal solution in this situation would be to create some kind of wrapper around Zend Mail and provide same set of functions as PHP IMAP but with x-prefix for example (ximap_open, ximap_fetchbody, ximap_list, etc.) for easier migration from PHP IMAP extension.

In this case PHP developers to allow OAuth-authentication will just need to rename IMAP function: imap_open to ximap_open, etc. If anybody feel like joining the efforts you are welcome: https://github.com/freescout-helpdesk/ximap

[bishopb]

PHP IMAP extension maintainer here.

Maybe the PHP devs should implement OAuth2 for the IMAP extension?

We could, @hopeseekr , that's an option, but doing so would effectively require rewriting the extension from the ground up. Here's why: the current implementation is a thin wrapper around the c-client library from the University of Washington. This library was (AFAIK) the original IMAP protocol implementation: it began in 1988 and was last updated in 2007. All of this long before OAuth became a thing. There is no concept of token exchange or flows in the library, and we'd have to work that in, without breaking existing behavior.

That is not a particularly appealing option, because there are excellent user-space IMAP implementations that have better performance than the extension and support OAuth already.

So, @freescout-helpdesk when you said:

... we've intentionally chosen PHP IMAP to make the app as fast as possible

that may have been true years ago, but today Horde/Imap-client is one candidate that is a far better choice. Horde specifically tailored and wrote their own to bypass the performance, feature, and scalability problems of the extension. Zend/Zend_Mail is a good choice as well, and it may be equally as performant. I'm sure there are other IMAP implementations out there as well.

Ideal solution in this situation would be to create some kind of wrapper around Zend Mail and provide same set of functions as PHP IMAP but with x-prefix for example (ximap_open, ximap_fetchbody, ximap_list, etc.) for easier migration from PHP IMAP extension.

That's where we (the people maintaining the extension) think the future lay, but note that there are two problems to solve:

1. The existing API was designed for credential authentication only, not auth flows. No part of the existing API naturally aligns with the concept of OAuth. So it's not as simple as "rename imap_open to ximap_open and you're done". The consuming software needs to handle auth flows as their own distinct thing.

2. If the PHP maintainers provide a user-land shim, it has to be agnostic with respect to the driving libraries and the installation mechanism. We (the people maintaining the extension) can't seem like we're supporting one project over another simply by choosing to include support for it. So that means we provide an interface and a pear package, and that's it. That's not a drop-in solution, though, it puts work on the consumer to get it working, which is unlike how the extension ecosystem operates. That's not ideal.

When JIT become available in PHP 8, we have finally the opportunity to discuss writing native extensions in PHP itself (item #1, bullet #2). This means we could write a modern IMAP implementation in PHP, bundle it as a JIT-extension with the performance of C code, and everyone is happy. The timeline on that, though, is beyond the 2021 Google deadline, so we need a real solution before then.

Please feel free to comment on the direction you'd like PHP to go via Issue 78572 in the PHP tracker.

[nand2]

Ok I went through the migration for one of our projects, where we scan mailboxes of a GSuite account.
Here are the steps I did :

• Added laminas/laminas-mail to composer.json (laminas/laminas-mail is zend-mail renamed end of 2019)

• Added php-mime-mail-parser/php-mime-mail-parser to composer.json

• Added the PHP mailparse extension (requirement of php-mime-mail-parser)

• Added google/apiclient to composer.json

• Created a service account key using the following guide , which is slightly not up to date, and with the following differences : We used the key type JSON (and not P12), we didn't get a private key password. In the end, we have a JSON file that we received from there.

• Used the following code for getting an access token to be used for OAuth authentification

            // Google client : Fetch an OAuth access token to access the mailbox
            $client = new \Google_Client();
            $client->useApplicationDefaultCredentials();
            //$client->setClientId(''); // Seems to be not necessary
            $client->setConfig('client_email', /** COPY here the OAuthClientEmail field of the JSON file  */);
            $client->setConfig('signing_key', /** COPY here the private_key field of the JSON file  */);
            //$client->setConfig('signing_algorithm', 'HS256'); // Seems to be not necessary
            // Impersonate user we want to process
            $client->setSubject(/** Put here the email of the mailbox you want to load */);
            // Indicate the permissions we are asking for
            $client->addScope(\Google_Service_Gmail::MAIL_GOOGLE_COM);
            // Fetch an access token. Careful, the token itself is inside the returned array
            $accessTokenData = $client->fetchAccessTokenWithAssertion();
            $accessToken = $accessTokenData['access_token'];

• Used the following code for authentificating with OAuth :

            // Laminas (ex-Zend) Imap connection
            $imap = new \Laminas\Mail\Protocol\Imap('imap.gmail.com', 993, 'ssl');

            // Sending the OAuth authentification
            $authString = base64_encode("user=" . /** Put here the email of the mailbox you want to load */ . "\1auth=Bearer " . $accessToken . "\1\1");
            $authenticateParams = array('XOAUTH2', $authString);
            $imap->sendRequest('AUTHENTICATE', $authenticateParams);

            // If everything went well, expecting the "CAPABILITY" message
            $imap->readLine($tokens);
            if(count($tokens) == 0 || $tokens[0] != "CAPABILITY") {
                echo "ERROR while connecting, got : " . implode(" ", $tokens);
                exit;
            }
  
            // Get the storage object
            $storage = new \Laminas\Mail\Storage\Imap($imap);
            // This should work if you authentified successfully : 
            echo 'Total messages: ' . $storage->countMessages();

• Then, because laminas-mail/zend-mail is too much low level and not practical to read individual emails, I transfer the work to php-mime-mail-parser :

            // Get messages of the last 1 day
            $scanStartDate = new \Datetime("-1 days");
            $messageNumbers = $imap->search(['SINCE ' . $scanStartDate->format('d-M-Y')]);
            $messages = [];
            foreach($messageNumbers as $messageNumber) {
                // Get the raw message text
                $message = $storage->getMessage($messageNumber);
                $rawHeaders = $storage->getRawHeader($messageNumber);
                $rawContent = $storage->getRawContent($messageNumber);
                $rawMessage = $rawHeaders . "\n" . $rawContent;

                // Parse the message with PhpMimeMailParser
                $messageParser = new \PhpMimeMailParser\Parser();
                $messages[$messageNumber] = $messageParser->setText($rawMessage);
            }

            // Processing messages
            foreach ($messages as $messageNumber => $message) {
              // We then can use the much better php-mime-mail-parser API
              // to deal with messages
            }

Hope it helps, and I am looking at simpler alternatives :-)

[haiderpro]

Hi Guys,

I know Google, Yahoo and Microsoft are the big players and all of them have the resources to play heartlessly with our feelings but they are forgetting one thing that we have the talent so our talent and hard / smart work can beat them :-)

I work with some highly reputed clients and they were also facing the same issues. I have already solved the first few ones required by them including allowing them to send email via Gmail's Oauth2 Token System and also marking emails with stars (star / not starred option of Google), move messages, delete messages etc. I also designed it to be backward compatible so they can use any normal IMAP or Oauth2 based complicated email system. Actually I am doing programming for years and I am mostly required to do the tasks that others think is impossible. They give me tasks which other could not complete and I am also required to complete that task in easy to read simple coding.

When I first found out about this problem I also thought the same as following but did not attempt it because I thought it would be too much extra work to complete the given tasks:

Ideal solution in this situation would be to create some kind of wrapper around Zend Mail and provide same set of functions as PHP IMAP but with x-prefix for example (ximap_open, ximap_fetchbody, ximap_list, etc.) for easier migration from PHP IMAP extension.

In this case PHP developers to allow OAuth-authentication will just need to rename IMAP function: imap_open to ximap_open, etc. If anybody feel like joining the efforts you are welcome: https://github.com/freescout-helpdesk/ximap

But when I saw this thread I started to think again because it looks like a lot of people are suffering because of this problem. In the start I also tried to search for any ready-made solutions but no use and unfortunately no one either dared to respond to my same problem on forums that are ruled by very good programmers. So I thought I could offer my services to help the digital family get trough this painful problem. So I want to know what are the php IMAP extension's functions that are mostly used by this community which they think will force them to modify a lot of code so that we can start to replicate those first.

I am confident that I can replicate any of the functions of PHP 's IMAP system that can work with both Oauth2 as well as normal Imap. In fact I think I can even add more features to it or make it more easy by offering more features that were not possible earlier or were too much complicated with PHP IMAP system. So please let me know which functions you think are most popular ones and should be provided at first. Zend-mail can also be fully supported with it if that makes the majority happy :-). Hurry, respond please... before Google HR find and hire me and stop me to make this code public ;-)

My only concern is that it will take my good amount of time and effort (I think 15 days or so for the basic functions) if I work with full dedication to expedite the process but that is going to cost me a lot so any ideas how can we cope with it ?

[freescout-helpdesk]

@haiderpro this would be great. In our case we need imap_ functions used by this library: https://github.com/Webklex/laravel-imap

[freescout-helpdesk]

@haiderpro any progress?

[freescout-helpdesk]

PHP devs are trying to add xoauth2 support to imap_open:
https://wiki.php.net/todo/ext/imap/xoauth2

Request #64039 | suppport XOAUTH2 in imap_open
https://bugs.php.net/bug.php?id=64039

[freescout-helpdesk]

Small library to support OAuth for IMAP: https://github.com/vmuthal/VivOAuthIMAP

[freescout-helpdesk]

@bishopb Thanks for your work. Is it possible to estimate approximately when XOAUTH2 can be added to PHP IMAP extension? Can it happen this year or chances are very small? It would help developers to figure out what to do: to wait or to start redesigning their application to use some OAuth-compatible IMAP library.

[freescout-helpdesk]

In G Suite according to this, App Passwords will continue to work after oAuth 2.0 will be enforced (information is confirmed by G Suite support).

Situation with App Passwords and enforcing oAuth in Microsoft Office 365 Exchange is not so clear yet. So if you are using Microsoft Office 365 Exchange please try to clarify with their support and share here the info on if it will be possible to use App Passwords for IMAP authentication after October 13th, 2020 when Microsoft 365 will enforce oAuth 2.0 authentication (or Modern Authentication as they call it).

[freescout-helpdesk]

oAuth 2.0, G Suite, Microsoft 365 and PHP
https://medium.com/@freescout/oauth-2-0-g-suite-microsoft-365-and-php-7da16ca74314

[fulldecent]

I use the Horde IMAP extension to login to all my employee's accounts and check for old messages in their inbox. This goes on a dashboard. We are a small business / here is what a real-life small business solution looks like.

I have everyone's password in a file. We forbid 2FA because it requires waking people up in different time zone each time you login.

I know Google/SJW hates this. I know there is probably some $1-per-user-per-minute service that might provide a similar experience.

I hope the new version will still allow my workflow.

---

@rcknr, @jeffreyalles, haters can downvote, but you're better to share your enlightenment. We can track our support team's email performance, can you?

[EthraZa]

@bishopb is the imap_open propose still alive?
Here I am been pressed to decide if wait for this or start to work with a zend-something solution.

[freescout-helpdesk]

Webklex did it: https://github.com/Webklex/php-imap

[Ahmed-Aboud]

IMAP can still work if you use 2 step verification + generate an app password

https://www.lifewire.com/get-a-password-to-access-gmail-by-pop-imap-2-1171882

[francescobianco]

I hope this help someone ddeboer/imap#443 (comment)

cc @tedivm @fulldecent @TomasVotruba @hopeseekr @EthraZa

[lwcorp]

Just adding on May 30th, 2022 even free Gmail blocked support for non OAuth access - see Less secure apps & your Google Account.

2 step verification + generate an app password is still an option, but not too comfortable for scripts and Google officially asks not to use it: "App Passwords aren’t recommended and are unnecessary in most cases".

[mbesta-unidays]

Can somebody suggest me how do I refresh the token using service account, please find the below code.
var certificate = new X509Certificate2(
serviceAccountCertPath,
serviceAccountCertPassword,
X509KeyStorageFlags.Exportable);

            var credential = new ServiceAccountCredential(
                new ServiceAccountCredential.Initializer(serviceAccountEmail)
                {
                    Scopes = new[] { GoogleScope.ImapAndSmtp.Name },
                    User = userEmail
                    ,
                }.FromCertificate(certificate));
         
            var success =  credential.RequestAccessTokenAsync(CancellationToken.None);

            Task.WaitAll(success);

[junyer]

Sorry, this project provides Gmail OAuth2 tools – it's in the name!

[haiderpro]

Those who are still having issues due to Gmail's forced decision to move to Oauth system, can either use following:
https://github.com/javanile/php-imap2
(There are still some flaws, but it works after little tweaking)

Or I can also confirm that Gmail password for scripts / apps after two step verification also works like normal gmail imap password.

So one of the above should solve your issues.