503 Service Unavailable error when requesting 'admin/fastlyCdn_Edge_Dictionary/ListAll' #163

hostep · 2018-04-05T08:53:20Z

Hi guys

Yesterday we enabled Fastly on a staging environment of the Magento Cloud.

But we sometimes see errors popping up in random adminhtml pages under Stores > Configuration (not only in Advanced > System), this error is displayed in a red box: A technical problem with the server created an error. Try again to continue what you were doing. If the problem persists, try again later.

When looking through the Network tab of my inspector, I can see that some XHR requests to the Fastly module throws a 503 error which causes this problem.

I can see 3 similar XHR requests, two of them work and one fails.
The ones which work:

The one which fails with a 503 error:

https://example.com/my-super-secret/admin/fastlyCdn_Edge_Dictionary/ListAll/key/aabbccddeeff/?active_version=8

And I also sometimes see this one fail with a 503 error:

https://example.com/my-super-secret/admin/fastlyCdn_Vcl/checkAuthDictionary/key/aabbccddeefff/?isAjax=true

It's a bit random these errors, sometimes it's the first one, sometimes the other one, sometimes even both, and then sometimes no errors, really strange behavior.

Looking at the actual contents of the response, we see something like this (it feels like this is a server error somehow, but no idea how to debug this):

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
  <head>
    <title>503 Service Unavailable</title>
  </head>
  <body>
    <h1>Error 503 Service Unavailable</h1>
    <p>Service Unavailable</p>
    <h3>Guru Mediation:</h3>
    <p>Details: cache-ams4432-AMS some-number some-other-number</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>

Extra info which might be relevant:

we have no Edge Dictionaries setup at the moment.
Magento 2.2.2
fastly/magento2 version 1.2.48

Any ideas what can cause this or how we can solve this?

If there are no errors, those request return valid looking json strings, so I don't think the problem is in the Fastly module itself, but actually on the server side. If we have to report this to the Magento Cloud support instead, please let us know.

Thanks!

The text was updated successfully, but these errors were encountered:

vvuksan · 2018-04-05T13:25:57Z

We are looking into it.

tnikcevs · 2018-04-06T11:13:09Z

Hello

I've attempted to reproduce the issue on Magento versions (2.2.2 and 2.2.3) and both Fastly versions 1.2.48 and 1.2.49 (last) and was unable to reproduce the issue. I've used clean version with only domain set (without any config) and our working version (with old config) in testing.

If possible please check logs for server errors and let us know if this is something related to part of the fastly code so we will look into it.

also if you used active_version=8, can you let us know what config you have adjusted in version 8 (have you used magento admin to update config or did you make custom adjustments?)

Regards,
Tomislav

jordanvector · 2018-04-09T17:30:18Z

I can confirm that this is an issue happening in a magneto-cloud pro environment. I am seeing this error on this call on every single page load in the admin:

/admin/admin/fastlyCdn_Vcl/getBackends/key/6a6c898626bdabaec07b2e044c54e7c2e9b309e1b44df8151a81dceff3b19ba8/

Exception: Noticed exception 'Exception' with message 'Warning: session_start(): Failed to read session data: user (path: /var/lib/php/sessions) in /app/7alebf6ygqtlm/vendor/magento/framework/Session/SessionManager.php on line 189' in /app/7alebf6ygqtlm/vendor/magento/framework/App/ErrorHandler.php:61
in Magento\Framework\App\ErrorHandler::handler called at ? (?)
in session_start called at /app/7alebf6ygqtlm/vendor/magento/framework/Session/SessionManager.php (189)
in Magento\Framework\Session\SessionManager::start called at /app/7alebf6ygqtlm/generated/code/Magento/Backend/Model/Session/Interceptor.php (63)
in Magento\Backend\Model\Session\Interceptor::start called at /app/7alebf6ygqtlm/vendor/magento/framework/Session/SessionManager.php (130)
in Magento\Framework\Session\SessionManager::__construct called at /app/7alebf6ygqtlm/generated/code/Magento/Backend/Model/Session/Interceptor.php (14)
in Magento\Backend\Model\Session\Interceptor::__construct called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php (111)
in Magento\Framework\ObjectManager\Factory\AbstractFactory::createObject called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php (66)
in Magento\Framework\ObjectManager\Factory\Dynamic\Developer::create called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/ObjectManager.php (70)
in Magento\Framework\ObjectManager\ObjectManager::get called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php (144)
in Magento\Framework\ObjectManager\Factory\AbstractFactory::resolveArgument called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php (230)
in Magento\Framework\ObjectManager\Factory\AbstractFactory::resolveArgumentsInRuntime called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php (34)
in Magento\Framework\ObjectManager\Factory\Dynamic\Developer::_resolveArguments called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php (59)
in Magento\Framework\ObjectManager\Factory\Dynamic\Developer::create called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/ObjectManager.php (70)
in Magento\Framework\ObjectManager\ObjectManager::get called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php (144)
in Magento\Framework\ObjectManager\Factory\AbstractFactory::resolveArgument called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php (230)
in Magento\Framework\ObjectManager\Factory\AbstractFactory::resolveArgumentsInRuntime called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php (34)
in Magento\Framework\ObjectManager\Factory\Dynamic\Developer::_resolveArguments called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php (59)
in Magento\Framework\ObjectManager\Factory\Dynamic\Developer::create called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/ObjectManager.php (70)
in Magento\Framework\ObjectManager\ObjectManager::get called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php (144)
in Magento\Framework\ObjectManager\Factory\AbstractFactory::resolveArgument called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php (230)
in Magento\Framework\ObjectManager\Factory\AbstractFactory::resolveArgumentsInRuntime called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php (34)
in Magento\Framework\ObjectManager\Factory\Dynamic\Developer::_resolveArguments called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php (59)
in Magento\Framework\ObjectManager\Factory\Dynamic\Developer::create called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/ObjectManager.php (70)
in Magento\Framework\ObjectManager\ObjectManager::get called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php (144)
in Magento\Framework\ObjectManager\Factory\AbstractFactory::resolveArgument called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php (230)
in Magento\Framework\ObjectManager\Factory\AbstractFactory::resolveArgumentsInRuntime called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php (34)
in Magento\Framework\ObjectManager\Factory\Dynamic\Developer::_resolveArguments called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php (59)
in Magento\Framework\ObjectManager\Factory\Dynamic\Developer::create called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/ObjectManager.php (70)
in Magento\Framework\ObjectManager\ObjectManager::get called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php (144)
in Magento\Framework\ObjectManager\Factory\AbstractFactory::resolveArgument called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php (230)
in Magento\Framework\ObjectManager\Factory\AbstractFactory::resolveArgumentsInRuntime called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php (34)
in Magento\Framework\ObjectManager\Factory\Dynamic\Developer::_resolveArguments called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php (59)
in Magento\Framework\ObjectManager\Factory\Dynamic\Developer::create called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/ObjectManager.php (70)
in Magento\Framework\ObjectManager\ObjectManager::get called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php (144)
in Magento\Framework\ObjectManager\Factory\AbstractFactory::resolveArgument called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php (230)
in Magento\Framework\ObjectManager\Factory\AbstractFactory::resolveArgumentsInRuntime called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php (34)
in Magento\Framework\ObjectManager\Factory\Dynamic\Developer::_resolveArguments called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php (59)
in Magento\Framework\ObjectManager\Factory\Dynamic\Developer::create called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/ObjectManager.php (70)
in Magento\Framework\ObjectManager\ObjectManager::get called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php (144)
in Magento\Framework\ObjectManager\Factory\AbstractFactory::resolveArgument called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php (230)
in Magento\Framework\ObjectManager\Factory\AbstractFactory::resolveArgumentsInRuntime called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php (34)
in Magento\Framework\ObjectManager\Factory\Dynamic\Developer::_resolveArguments called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php (59)
in Magento\Framework\ObjectManager\Factory\Dynamic\Developer::create called at /app/7alebf6ygqtlm/vendor/magento/framework/ObjectManager/ObjectManager.php (70)
in Magento\Framework\ObjectManager\ObjectManager::get called at /app/7alebf6ygqtlm/vendor/magento/framework/Interception/PluginList/PluginList.php (250)
in Magento\Framework\Interception\PluginList\PluginList::getPlugin called at /app/7alebf6ygqtlm/vendor/magento/framework/Interception/Interceptor.php (119)
in Magento\Framework\App\FrontController\Interceptor::Magento\Framework\Interception{closure} called at /app/7alebf6ygqtlm/vendor/magento/framework/Interception/Interceptor.php (153)
in Magento\Framework\App\FrontController\Interceptor::___callPlugins called at /app/7alebf6ygqtlm/generated/code/Magento/Framework/App/FrontController/Interceptor.php (26)
in Magento\Framework\App\FrontController\Interceptor::dispatch called at /app/7alebf6ygqtlm/vendor/magento/framework/App/Http.php (135)
in Magento\Framework\App\Http::launch called at /app/7alebf6ygqtlm/vendor/magento/framework/Interception/Interceptor.php (58)
in Magento\Framework\App\Http\Interceptor::___callParent called at /app/7alebf6ygqtlm/vendor/magento/framework/Interception/Interceptor.php (138)
in Magento\Framework\App\Http\Interceptor::Magento\Framework\Interception{closure} called at /app/7alebf6ygqtlm/vendor/msp/shield/Plugin/AppInterfacePlugin.php (111)
in MSP\Shield\Plugin\AppInterfacePlugin::aroundLaunch called at /app/7alebf6ygqtlm/vendor/magento/framework/Interception/Interceptor.php (135)
in Magento\Framework\App\Http\Interceptor::Magento\Framework\Interception{closure} called at /app/7alebf6ygqtlm/vendor/magento/framework/Interception/Interceptor.php (153)
in Magento\Framework\App\Http\Interceptor::___callPlugins called at /app/7alebf6ygqtlm/generated/code/Magento/Framework/App/Http/Interceptor.php (26)
in Magento\Framework\App\Http\Interceptor::launch called at /app/7alebf6ygqtlm/vendor/magento/framework/App/Bootstrap.php (256)
in Magento\Framework\App\Bootstrap::run called at /app/7alebf6ygqtlm/pub/index.php (37)

I am also not sure why its looking for sessions in var/lib/php/sessions. Also I am on the latest fastly extension 1.2.49. Thoughts?

vvuksan · 2018-04-09T18:29:16Z

@jordanvector what happens if you revert to an older version ?

jordanvector · 2018-04-09T18:41:05Z

@vvuksan good question, I can't remember where started but I am pretty sure it was with the last 2 versions. I can revert 2 versions and report back

vvuksan · 2018-04-09T18:50:38Z

That would be great @jordanvector as we are not able to reproduce this.

jordanvector · 2018-04-09T18:53:05Z

@vvuksan I have noticed this typically after a deployment on magento cloud just fyi

vvuksan · 2018-04-09T19:02:22Z

Does the issue clear itself or does it persist ?

jordanvector · 2018-04-09T19:13:16Z

the issue persists but I was able to get rid of it for a period of time by adding a dictionary in the backend options

vvuksan · 2018-04-09T19:14:48Z

That's helpful. Just to confirm to workaround around the issue all you had to do was Edge Dictionary ?

jordanvector · 2018-04-09T20:07:52Z

@vvuksan correct the workaround for me was to add an edge dictionary

udovicic · 2018-04-10T05:19:27Z

Hi @jordanvector ,

After adding edge dictionaries and browsing through admin, issue no longer persists?

Trace you have pasted has nothing to do with the Fastly, as it is not mentioned. Application broke because it couldn't start the PHP session as indicate. Reason behind that seems to be because session path, in this case /var/lib/php/sessions, was not accessible. I am not 100% sure, but I do not think Magento cloud is supposed to store sessions on file system. If there are multiple web servers present, it would either be too slow, or it wouldn't work at all, which would explain the randomness you are talking about. Please confirm that sessions are stored somewhere else(preferably Redis), unless you have only one web server and both /var/lib/php/sessions and var/ within Magento root are accessible and writable.

jordanvector · 2018-04-10T15:44:05Z

@udovicic so we are definitely using redis for session storage which is very odd that it is looking at /var/lib/php/sessions. Here is the quick line from env.php (minus connection values)

'session' =>
array (
'save' => 'redis',
'redis' =>

jordanvector · 2018-04-10T16:02:23Z

@udovicic upon further investigation sessions are be written to redis (so not quite sure whats going on there). I have a ticket open with magento as well, will update if I hear anything from them

jordanvector · 2018-04-10T22:05:37Z

@udovicic after digging into this it seems like my issue specifically was not linked to fastly. the magento cloud php.ini by default has sessions configured to save to the filesystem by default which was causing the error that I was seeing. By default it looks like this in php.ini:

session.save_handler => files => files
session.save_path => /var/lib/php/sessions => /var/lib/php/sessions

When I updated the php.ini to save to redis like this:

session.save_handler = redis
session.save_path=localhost:6370 (your port probably will vary based on configuration)

It seemed to resolve the issue. I still have a ticket that magento has not responded to so if they come back with any other suggestions I will report them here.

yyevgenii · 2018-05-02T15:46:01Z

The issue is caused by Redis session blocking mechanism. The Cm/RedisSession library has a variable which limits the maximum number of processes that can wait for lock on one session. By default it's value is 6. The default value leads to errors on Magento frontend/backend when, for example, a web page makes several AJAX requests to Magento.

That's what you see when navigate to Stores > Configuration > System: Fastly extension starts sending several AJAX requests to Magento and you reach the maximum concurrency limit. When it happens, Magento responds with 503 error and you can find "Area code is not set" errors in the var/report/ directory.

To avoid the issue, you have to increase the max_concurrency value in app/etc/env.php:

'session' =>
        array (
            'save' => 'redis',
            'redis' =>
                array (
                    'host' => '127.0.0.1',
                    'port' => '6379',
                    'database' => 0,
                    'max_concurrency' => 20,
                ),
        ),

If you're a Magento Cloud user, you mustn't edit env.php manually, but add the SESSION_CONFIGURATION variable with Redis configuration in JSON format. You can get more details by submitting a support request.

yyevgenii · 2018-05-02T15:47:09Z

@vvuksan it's not a fastly/magtento2 bug, you can close the issue.

vvuksan · 2018-05-02T15:52:01Z

wow thanks :-)

hostep · 2018-05-02T17:11:26Z

Thanks for the info @yyevgenii!

I'll test this when I find some time to work on this project again (probably next week).

agorbulin · 2018-05-03T07:44:57Z

I confirmed, that after changing session settings issue was fixed.
For Magento cloud you can just make env variable SESSION_CONFIGURATION (type is json) with content:

{
"save": "redis",
"redis": {
"host": "localhost",
"port": "6372",
"database": 0,
"max_concurrency": 20
}
}

Note: your environment settings can be differ

hostep · 2018-05-09T13:54:01Z

Can confirm the fix from @agorbulin.

I first tried to change the concurrency value using the .magento.env.yaml file for the Magento Cloud hosting, but it's like @agorbulin is saying: some settings (host, port) are different over different environments, so we had to also define that environment variable using the Cloud UI per environment specific.

And it solves the problem indeed!

dborishansky · 2018-05-10T00:12:55Z

Increasing max_concurrency did address the errors in the admin / logs, but most unfortunately introduced performance issues on the front end. Still hunting for a viable solution to this issue.

The burden of responsibility here seems to fall to the fastly module, if reducing the number of AJAX requests is feasible?

vvuksan · 2018-05-10T07:18:09Z

Can you elaborate on performance issues on the front end ? I'm a bit surprised that one cannot make more than 6 AJAX requests.

dborishansky · 2018-05-10T12:42:59Z

We were seeing things like this with max_concurrency set to 20:

\Magento\Framework\Session\SaveHandler\Redis::read() seems to become a bottleneck.

The impact was seen across many transactions, i.e. price rendering was about 3x slower than prior to the redis configuration change.

vvuksan · 2018-05-10T16:51:39Z

Sounds like a lock contention.

vvuksan · 2018-05-11T09:11:00Z

We'll look into doing this differently.

alanablett · 2018-05-15T08:36:00Z

Getting the same issue here on Cloud. Using those redis settings caused users to be logged out intermittently so that's a no go for us unfortunately.

vvuksan · 2018-05-17T13:59:31Z

We have addressed this in #177. Fixes were released as part of 1.2.54.

hostep · 2018-05-23T12:20:10Z

Just tested by removing the SESSION_CONFIGURATION variable again and upgrading to version 1.2.54 and all looks fixed indeed. Thanks @vvuksan!

vvuksan closed this as completed May 2, 2018

vvuksan reopened this May 11, 2018

vvuksan closed this as completed May 17, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

503 Service Unavailable error when requesting 'admin/fastlyCdn_Edge_Dictionary/ListAll' #163

503 Service Unavailable error when requesting 'admin/fastlyCdn_Edge_Dictionary/ListAll' #163

hostep commented Apr 5, 2018

vvuksan commented Apr 5, 2018

tnikcevs commented Apr 6, 2018

jordanvector commented Apr 9, 2018

vvuksan commented Apr 9, 2018

jordanvector commented Apr 9, 2018

vvuksan commented Apr 9, 2018

jordanvector commented Apr 9, 2018

vvuksan commented Apr 9, 2018

jordanvector commented Apr 9, 2018

vvuksan commented Apr 9, 2018 •

edited

Loading

jordanvector commented Apr 9, 2018

udovicic commented Apr 10, 2018

jordanvector commented Apr 10, 2018

jordanvector commented Apr 10, 2018

jordanvector commented Apr 10, 2018 •

edited

Loading

yyevgenii commented May 2, 2018

yyevgenii commented May 2, 2018

vvuksan commented May 2, 2018

hostep commented May 2, 2018

agorbulin commented May 3, 2018 •

edited

Loading

hostep commented May 9, 2018

dborishansky commented May 10, 2018

vvuksan commented May 10, 2018

dborishansky commented May 10, 2018

vvuksan commented May 10, 2018

vvuksan commented May 11, 2018

alanablett commented May 15, 2018

vvuksan commented May 17, 2018

hostep commented May 23, 2018

503 Service Unavailable error when requesting 'admin/fastlyCdn_Edge_Dictionary/ListAll' #163

503 Service Unavailable error when requesting 'admin/fastlyCdn_Edge_Dictionary/ListAll' #163

Comments

hostep commented Apr 5, 2018

vvuksan commented Apr 5, 2018

tnikcevs commented Apr 6, 2018

jordanvector commented Apr 9, 2018

vvuksan commented Apr 9, 2018

jordanvector commented Apr 9, 2018

vvuksan commented Apr 9, 2018

jordanvector commented Apr 9, 2018

vvuksan commented Apr 9, 2018

jordanvector commented Apr 9, 2018

vvuksan commented Apr 9, 2018 • edited Loading

jordanvector commented Apr 9, 2018

udovicic commented Apr 10, 2018

jordanvector commented Apr 10, 2018

jordanvector commented Apr 10, 2018

jordanvector commented Apr 10, 2018 • edited Loading

yyevgenii commented May 2, 2018

yyevgenii commented May 2, 2018

vvuksan commented May 2, 2018

hostep commented May 2, 2018

agorbulin commented May 3, 2018 • edited Loading

hostep commented May 9, 2018

dborishansky commented May 10, 2018

vvuksan commented May 10, 2018

dborishansky commented May 10, 2018

vvuksan commented May 10, 2018

vvuksan commented May 11, 2018

alanablett commented May 15, 2018

vvuksan commented May 17, 2018

hostep commented May 23, 2018

vvuksan commented Apr 9, 2018 •

edited

Loading

jordanvector commented Apr 10, 2018 •

edited

Loading

agorbulin commented May 3, 2018 •

edited

Loading