Varnish ESI blocks don't load when HTTPS is used site-wide

[erikhansen]

This issue is related to #2657 that the OP closed.

Steps to reproduce

1. Install Magento 2.0.2 with sample data
2. Configure all urls to load from HTTPS (e.g., set "Base URL" and "Secure Base URL" to https://example.dev)
3. Load frontend and you'll see the navigation present.
4. Configure and enable Varnish (see documentation)
5. Load the frontend and you'll see the main navigation is missing:
   

Potential Solutions

1. Modify Varnish ESI to ignore HTTPS

Per this Stack Exchange answer, if you modify your /etc/sysconfig/varnish file and add -p feature=+esi_ignore_https to the arguments passed to Varnish, the problem will be solved. For example:

DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
             -f ${VARNISH_VCL_CONF} \
             -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
             -p feature=+esi_ignore_https \
             -p thread_pool_min=${VARNISH_MIN_THREADS} \
             -p thread_pool_max=${VARNISH_MAX_THREADS} \
             -S ${VARNISH_SECRET_FILE} \
             -s ${VARNISH_STORAGE}"

This solution is merely a workaround for the fact that Magento 2 returns HTTPS urls as ESI urls. Also, this option is only available in Varnish 4.

2. Modify Magento's \Magento\PageCache\Observer\ProcessLayoutRenderElement::_wrapEsi method to never return HTTP urls

Per the #2657 issue raised by @boldhedgehog, the proper solution is to modify Magento to only return HTTP urls to Varnish. How exactly this is handled is up for discussion, as I don't think it would be proper to simply replace "https" with "https".

[mazhalai]

@erikhansen Thank you for reporting, we have created MAGETWO-51149 to investigate and fix.

[rip057]

why on earth is varnish the "go to" app when there is no ssl support? and +1 for my navigation bar not showing up, with or without the work around. also someone should fix the "never return HTTP urls"
and the [simply replace "https" with "https"] it is a little confusing.

[erikhansen]

@rip057 You can use a server like Nginx (example config) or Pound as an SSL terminator to pass through requests to Varnish so that HTTPS requests can still take advantage of Varnish's memory-based cache system. So while it doesn't natively support SSL, it can still work with SSL by having a system sitting in front of it handling the SSL connection.

[rip057]

I've decided to dive in to apache mod_mem_cache, and mod_file_cache, and
customize it. it just sucks that magneto is kind of pointing you, me and
anyone who doesn't want to look around, towards something that is
inherently lacking right now. I don't know a single website that doesn't
want a full green bar on their web link name.
On Apr 18, 2016 11:40 AM, "Erik Hansen" notifications@github.com wrote:

@rip057 https://github.com/rip057 You can use an SSL terminator like
Nginx or Pound so that HTTPS requests can still take advantage of Varnish's
memory-based cache system.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#3897 (comment)

[rip057]

Also sorry for being harsh, I just spent quite a lot of time getting
magneto 2.0 in, then scratching and starting over, then again, only to
find out about varnish at the end, and the only reason I found out about
varnish is my effing navigational bar took a vacation, and I spent another
several hours trying to figure out if I caused it by disabling something.
Anyhow has anyone tried mod_mem_cache? Have any xp with it?
On Apr 18, 2016 12:05 PM, "Chris Clark" rip057@gmail.com wrote:

I've decided to dive in to apache mod_mem_cache, and mod_file_cache, and
customize it. it just sucks that magneto is kind of pointing you, me and
anyone who doesn't want to look around, towards something that is
inherently lacking right now. I don't know a single website that doesn't
want a full green bar on their web link name.
On Apr 18, 2016 11:40 AM, "Erik Hansen" notifications@github.com wrote:

@rip057 https://github.com/rip057 You can use an SSL terminator like
Nginx or Pound so that HTTPS requests can still take advantage of Varnish's
memory-based cache system.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#3897 (comment)

[erikhansen]

@rip057 There is certainly a learning curve with Magento 2, but after having worked with it for ~9 months (after 7 years on M1), I think it is a great platform that will server merchants well for many years to come.

Magento 2 decided to ship with Varnish support natively as many merchants and solutions partners implemented Varnish on M1 to build highly scalable and performant sites. However using Varnish as a cache system, while "recommended", is optional. For merchants that don't have large amounts of traffic (especially burst traffic), they can just use the native Magento caching system and forego Varnish altogether.

[rip057]

But still you got no opinion on mod_cache? It is also a memory cache, I
mean basically the same thing as varnish
On Apr 18, 2016 12:15 PM, "Erik Hansen" notifications@github.com wrote:

@rip057 https://github.com/rip057 There is certainly a learning curve
with Magento 2, but after having worked with it for ~9 months (after 7
years on M1), I think it is a great platform that will server merchants
well for many years to come.

Magento 2 decided to ship with Varnish support natively as many merchants
and solutions partners implemented Varnish on M1 to build highly scalable
and performant sites. However using Varnish as a cache system, while
"recommended", is optional. For merchants that don't have large amounts of
traffic (especially burst traffic), they can just use the native Magento
caching system and forego Varnish altogether.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#3897 (comment)

[erikhansen]

@rip057 Many people have discontinued using Apache in favor of Nginx, so if Apache isn't already a part of your technology stack, it would make more sense to use an application designed for the specific purpose of being a memory-based cache, rather than a module for a more general-purpose server.

Since Magento natively integrates with Varnish, I wouldn't spend my time investigating an alternative cache system like Apache+mod_cache.

[rip057]

Maybe, it probably like what flavor do you run kde or gnome, personal
choice I guess, I just have an affinity for apache. They have done so much
for the world in general
On Apr 18, 2016 4:20 PM, "Erik Hansen" notifications@github.com wrote:

@rip057 https://github.com/rip057 Many people have discontinued using
Apache in favor of Nginx, so if Apache isn't already a part of your
technology stack, it would make more sense to use an application designed
for the specific purpose of being a memory-based cache, rather than a
module for a more general-purpose server.

Since Magento natively integrates with Varnish, I wouldn't spend my time
investigating an alternative cache system like Apache+mod_cache.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#3897 (comment)

[davidalger]

@rip057 One thing to keep in mind here is also that Magento 2 is designed to scale, and the web stack that's officially supported must keep that in mind. Varnish is a scaling tool. In a clustered deployment scenario, it's very common to terminate SSL in the load-balancer. In that situation, varnish lacking SSL termination support doesn't really matter since it's not needed in the reverse proxy layer. Where SSL termination is not in the load-balancer, we are setting up an nginx -> varnish -> nginx sandwich, using nginx for SSL termination, then passing on to varnish for the app acceleration, and then on to nginx + php-fpm to actually run the application. This results in a still very lightweight setup. Similar could be accomplished with apache were one to take the time to build appropriate configuration files.

[rip057]

So does anyone have any experience with both? Anyone have experience at all
with mod_cache or mod_socache_shmcb. I get they are relatively new to the
release section of apache. The whole navi bar, is the only reason for all
ofthis, although I'm sure there are other broken parts

[davidalger]

@rip057 That is really a question better suited for the forums, as we need to keep the issue tracker on it's core focus, technical issues / bugs with the Magento 2 platform. I would recommend you take that over to the Community Forums or Magento Stack Exchange site. Thanks!

[rip057]

Word sorry, I just felt people that dont have a Wal-Mart.com traffic site
would find this page like I did cause of the navi bar
On Apr 19, 2016 1:27 PM, "David Alger" notifications@github.com wrote:

@rip057 https://github.com/rip057 That is really a question better
suited for the forums, as we need to keep the issue tracker on it's core
focus, technical issues / bugs with the Magento 2 platform. I would
recommend you take that over to the Community Forums
https://community.magento.com or Magento Stack Exchange
http://magento.stackexchange.com/ site. Thanks!

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#3897 (comment)

[choukalos]

Hey @rip057 ; we purposely chose Varnish because our community proved to us it's the winning solution. Trouble in M1 was the cost for most merchants to install/setup Varnish. We've made that much easier in M2 ( out of the box; hosting partner support ) along with making our caching much more efficient. Varnish is pretty impressive... with high cache hit ratios and M2's caching efficiency you can handle much higher multiple of traffic then in M1-EE/Redis with the same server! To make it even easier to use Varnish with M2 we've been working with our hosting partners. You should check out Nexcess, Simple Helix, etc. as they all have M2/Varnish shared host based plans that should meet your needs.

[rip057]

I appreciate all the feedback, but am having good results with apache,
thanks
On Apr 21, 2016 9:24 AM, "Chuck Choukalos" notifications@github.com wrote:

Hey @rip057 https://github.com/rip057 ; we purposely chose Varnish
because our community proved to us it's the winning solution. Trouble in M1
was the cost for most merchants to install/setup Varnish. We've made that
much easier in M2 ( out of the box; hosting partner support ) along with
making our caching much more efficient. Varnish is pretty impressive...
with high cache hit ratios and M2's caching efficiency you can handle much
higher multiple of traffic then in M1-EE/Redis with the same server! To
make it even easier to use Varnish with M2 we've been working with our
hosting partners. You should check out Nexcess, Simple Helix, etc. as they
all have M2/Varnish shared host based plans that should meet your needs.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#3897 (comment)

[peec]

After 7 hours of struggling to get HTTPS to work on Magento 2 I have given up. Categories are not showing on frontend when HTTPS is enabled..

I have currently this setup where i terminate SSL before reaching varnish:

Load blancer (HTTPS terminated to HTTP) -> Varnish -> Nginx.

• Use https on frontend is set to true in magento admin.
• Unsecure and secure url are set to https://mydomain
• "HTTPS on" env variable is passed as fastsgi param to nginx. This so that magento does not generate "http" links on frontend.. Undocumented.
• deploy:static are deployed with HTTPS=on env variable ( to avoid bugs not generatingt the /secure folder in assets).. Undocumented.

Now magento loads all resources on https, however categories are missing. Only problem I can see atm..

[gewaechshaus]

Hey,
even if this thread isn't directly related to ssl setup - we're running setups like this without problems... HaProxy terminates ssl -> varnish -> nginx -> php7-fpm/hhvm. The only thing related to categories not coming up when using varnish is to setup

       -p feature=+esi_ignore_https 

on the running varnish instance like @erikhansen wrote. You can easily verify this by temporary bypassing or disabling varnish. Does the navigation show up on routes that will hit for pass? F.e. on my account or login?

All other things related to ssl should be ok.

[piotrekkaminski]

Fix should be delivered to github shortly, already done internally and will be released with 2.1

[Jilco]

I removed the TTL attribute from /vendor/magento/module-theme/view/frontend/layout/default.xml and the menu bar was there again! But can that do any harm, if I'm correct the default TTL is enheritated. Or am I wrong?

[henk-hypershop]

Same issue here. Varnish working, removed TTL attribute. Still no navigation showing. Running on Magento 2.1.3.

[dhduc]

@Jilco it seems that removed TTL attribute is working, but have a side effect when to do that.
According to the author of @sivaschenko site, each time you updated a category, the all website page cache will be flushed. Follow his article about this problem http://www.sivaschenko.com/magento2-full-page-and-block-cache-types/

[Frits1980]

With me it doesn't work. running M 2.1.7

[bbtimx]

Is there a way to override the ttl attribute in the themes default.xml ?

[guido7171]

@bbtimx just place your customized default.xml file, in your own extension app/code/My/Module/view/frontend/layout/ folder.

[moejoee91]

Hello everyone,
im having something similar to this and it's driving me crazy, all of a sudden my nav-bar was gone, and im not even using varnish, i'm using the built in cache, all i did right before this happened is i did setup cloudflare and it went down hill from there
any ideas what might be causing the problem ?

[dev-Dhaval]

Guys, Any solution for this ?

[vetrivel-innoppl]

Not sure how many of you seen the 2.2.0 release notes. So, just posting the word from it for reference

Viewing the page with HTTPS instead of HTTP no longer causes the Category menu to disappear in installations using Varnish cache. GitHub-4540

[ghost]

Hi Guys I've managed to solve this issue doing that proposed by @Jilco but should be other solution for this?, right? I'm using 2.2.3 btw, why is this issue closed?

[gcampedelli]

I agree, this issue should be reopened. Menu is missing when using Varnish. The proposed solution using varnish settings only displays menu in checkout in Magento 2.2.3 and removing TTL is not the best solution for it.

[csdougliss]

+esi_ignore_https doesn't work for me, but removing ttl did. That is not a good solution.

Magento any ideas? @piotrekkaminski

[gcampedelli]

Hello @craigcarnell . Thanks for joining our conversation to solve this. I had the same problem using NGINX. Turns out that the problem was my NGINX settings and varnish settings that were blocking menu to be rendered. Could you please check your varnish and NGINX settings using curl "curl -I -v --location-trusted ''" using HTTP and then "curl -I -v --location-trusted ''" using HTTPS ?

[abbasalibutt]

Hello @gcampedelli. Maybe my question is a very basic question. Actually, I am talking with our hosting company and trying to convince them it is a Varnish or server configuration issue. Can you please tell me in simple words how can I explain them or ask them to change in the server or varnish cache configuration to fix this issue?

[abbasalibutt]

When I execute the command

curl -I -v --location-trusted http://URL
I get some

< HTTP/1.1 302 Found
HTTP/1.1 302 Found
< Date: Tue, 17 Jul 2018 01:00:34 GMT
Date: Tue, 17 Jul 2018 01:00:34 GMT
< Server: Apache
Server: Apache
< Pragma: no-cache
Pragma: no-cache

and with

curl -I -v --location-trusted https://URL there is no 302.

[gcampedelli]

I don't know your settings, but if you are running ssl you have to get 200 on HTTP.

[gcampedelli]

Hello @abbasalibutt. As it is hard to say because of server configuration, only thing I can tell you is my experience. In my case, it was NGINX and Varnish configuration. I had to replace default VCL and reconfigure varnish. My default VCL is like this:

`vcl 4.0;

import std;

The minimal Varnish version is 4.0

For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'

backend default {
.host = "127.0.0.1";
.port = "8080";
}

acl purge {
"127.0.0.1";
}

sub vcl_recv {
if (req.method == "PURGE") {
if (client.ip !~ purge) {
return (synth(405, "Method not allowed"));
}
if (!req.http.X-Magento-Tags-Pattern) {
return (synth(400, "X-Magento-Tags-Pattern header required"));
}
ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
return (synth(200, "Purged"));
}

if (req.method != "GET" &&
    req.method != "HEAD" &&
    req.method != "PUT" &&
    req.method != "POST" &&
    req.method != "TRACE" &&
    req.method != "OPTIONS" &&
    req.method != "DELETE") {
      /* Non-RFC2616 or CONNECT which is weird. */
      return (pipe);
}

# We only deal with GET and HEAD by default
if (req.method != "GET" && req.method != "HEAD") {
    return (pass);
}

# Bypass shopping cart, checkout and search requests
if (req.url ~ "/checkout" || req.url ~ "/catalogsearch") {
    return (pass);
}

# normalize url in case of leading HTTP scheme and domain
set req.url = regsub(req.url, "^http[s]?://", "");

# collect all cookies
std.collect(req.http.Cookie);

# Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
if (req.http.Accept-Encoding) {
    if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
        # No point in compressing these
        unset req.http.Accept-Encoding;
    } elsif (req.http.Accept-Encoding ~ "gzip") {
        set req.http.Accept-Encoding = "gzip";
    } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
        set req.http.Accept-Encoding = "deflate";
    } else {
        # unkown algorithm
        unset req.http.Accept-Encoding;
    }
}

# Remove Google gclid parameters to minimize the cache objects
set req.url = regsuball(req.url,"\?gclid=[^&]+$",""); # strips when QS = "?gclid=AAA"
set req.url = regsuball(req.url,"\?gclid=[^&]+&","?"); # strips when QS = "?gclid=AAA&foo=bar"
set req.url = regsuball(req.url,"&gclid=[^&]+",""); # strips when QS = "?foo=bar&gclid=AAA" or QS = "?foo=bar&gclid=AAA&bar=baz"

# static files are always cacheable. remove SSL flag and cookie
    if (req.url ~ "^/(pub/)?(media|static)/.*\.(ico|css|js|jpg|jpeg|png|gif|tiff|bmp|mp3|ogg|svg|swf|woff|woff2|eot|ttf|otf)$") {
    unset req.http.Https;
    unset req.http.X-Forwarded-Proto;
    unset req.http.Cookie;
}

return (hash);

}

sub vcl_hash {
if (req.http.cookie ~ "X-Magento-Vary=") {
hash_data(regsub(req.http.cookie, "^.?X-Magento-Vary=([^;]+);.*$", "\1"));
}

# For multi site configurations to not cache each other's content
if (req.http.host) {
    hash_data(req.http.host);
} else {
    hash_data(server.ip);
}

# To make sure http users don't see ssl warning
if (req.http.X-Forwarded-Proto) {
    hash_data(req.http.X-Forwarded-Proto);
}
/* {{ design_exceptions_code }} */

}

sub vcl_backend_response {
if (beresp.http.content-type ~ "text") {
set beresp.do_esi = true;
}

if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
    set beresp.do_gzip = true;
}

# cache only successfully responses and 404s
if (beresp.status != 200 && beresp.status != 404) {
    set beresp.ttl = 0s;
    set beresp.uncacheable = true;
    return (deliver);
} elsif (beresp.http.Cache-Control ~ "private") {
    set beresp.uncacheable = true;
    set beresp.ttl = 86400s;
    return (deliver);
}

if (beresp.http.X-Magento-Debug) {
    set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
}

# validate if we need to cache it and prevent from setting cookie
# images, css and js are cacheable by default so we have to remove cookie also
if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
    unset beresp.http.set-cookie;
    if (bereq.url !~ "\.(ico|css|js|jpg|jpeg|png|gif|tiff|bmp|gz|tgz|bz2|tbz|mp3|ogg|svg|swf|woff|woff2|eot|ttf|otf)(\?|$)") {
        set beresp.http.Pragma = "no-cache";
        set beresp.http.Expires = "-1";
        set beresp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
        set beresp.grace = 1m;
    }
}

If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass

if (beresp.ttl <= 0s ||
beresp.http.Surrogate-control ~ "no-store" ||
(!beresp.http.Surrogate-Control && beresp.http.Vary == "*")) {
# Mark as Hit-For-Pass for the next 2 minutes
set beresp.ttl = 120s;
set beresp.uncacheable = true;
}
return (deliver);
}

sub vcl_deliver {
if (resp.http.X-Magento-Debug) {
if (resp.http.x-varnish ~ " ") {
set resp.http.X-Magento-Cache-Debug = "HIT";
} else {
set resp.http.X-Magento-Cache-Debug = "MISS";
}
} else {
unset resp.http.Age;
}

unset resp.http.X-Magento-Debug;
unset resp.http.X-Magento-Tags;
unset resp.http.X-Powered-By;
unset resp.http.Server;
unset resp.http.X-Varnish;
unset resp.http.Via;
unset resp.http.Link;

}`

[csdougliss]

@gcampedelli

curl -I -v --location-trusted 'http://www.xxx.co.uk.local/'
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to www.xxx.co.uk.local (127.0.0.1) port 80 (#0)
> HEAD / HTTP/1.1
> Host: www.vax2.co.uk.local
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 302 Found
HTTP/1.1 302 Found

curl -I -v --location-trusted 'https://www.xxx.co.uk.local/'
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to www.xxx.co.uk.local (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.

I've just realised.. is the ESI issue because I am re-directing HTTP to HTTPS? We are HTTPS only now. Our base URL and secure base URL is https.

ESI is loaded over HTTP

<esi:include src="http://www.xxx.co.uk.local/page_cache/block/esi/blocks/%5B%22catalog.topnav%22%5D/handles/WyJkZWZhdWx0IiwiY21zX2luZGV4X2luZGV4IiwiY21zX3BhZ2VfdmlldyJd/" /></div>

[gcampedelli]

On http you have to get 200. To achieve that, you have to configure NGINX with proxy-pass.

Rebuilt URL to:myurl

• Trying 54.207.126.154...
• Connected to localhost port 80 (#0)

HEAD / HTTP/1.1
Host: localhost
User-Agent: curl/7.47.0
Accept: /

< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Tue, 17 Jul 2018 07:53:40 GMT
Date: Tue, 17 Jul 2018 07:53:40 GMT
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
X-Frame-Options: SAMEORIGIN
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-Magento-Cache-Control: max-age=86400, public, s-maxage=86400
X-Magento-Cache-Control: max-age=86400, public, s-maxage=86400
< Pragma: no-cache
Pragma: no-cache
< Expires: -1
Expires: -1
< Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
< Age: 10224
Age: 10224
< X-Magento-Cache-Debug: HIT
X-Magento-Cache-Debug: HIT
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Connection: keep-alive
Connection: keep-alive

[gcampedelli]

Nginx - conf

upstream fastcgi_backend {
server unix:/var/run/php/php7.0-fpm.sock;
}
server {
server_name myurl;
listen 8080;
set $MAGE_ROOT /var/www/html/magento2/pub;
set $MAGE_MODE production; # or developer
root $MAGE_ROOT;
access_log /var/log/nginx/myurl.com-access.log;
error_log /var/log/nginx/myurl-error.log;

include /var/www/html/magento2/nginx.conf.sample;

}

server {

listen 443 ssl http2;
server_name myurl;
proxy_headers_hash_bucket_size 128;
proxy_headers_hash_max_size 1024;
ssl_certificate /etc/letsencrypt/live/myurl/fullchain.pem; # managed by Certbot    
ssl_certificate_key /etc/letsencrypt/live/myurl/privkey.pem; # managed by Certbot
ssl_protocols              TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers               'AES128+EECDH:AES128+EDH:!aNULL';
#ssl_session_cache    shared:SSL:10m;
#ssl_session_timeout 24h;
#keepalive_timeout 300s;

location / {
    proxy_pass http://127.0.0.1:80;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header      X-Forwarded-Proto https;
    proxy_set_header      X-Forwarded-Port 443;
    proxy_set_header X-Forwarded-Proto $scheme;
    include fastcgi.conf;

}

}

And configure Varnish as Magento 2 docs. But you may face problems with the VCL generated in Magento 2 backend. Please refer yourself to default.vcl I published here. Hope you to achieve that.

[csdougliss]

@gcampedelli

My Varnish is running on port 80, nginx on 8080 and 443.

I believe it may be Magento doing to the 302 redirect from HTTP to HTTPS as my base URL and secure URL are both HTTPS?

If this is the case, then this would not work?

Why would setting ttl to 0 make it work?

[gcampedelli]

It is really hard to say. In my Magento 2 configuration, both are https.
Curl gives 200 on http
Varnish settings are above and Nginx or Apache should have proxy_pass correctly configure 127.0.0.1:80
I had the same problem as you are facing right now. I tried every possible solution and it was a server configuration preventing menu to render.

[oviliz]

2016 and is a closed issue while the problem is still present in 2.2.4 at least...

[vetshopdeveloper]

2.2.7 and still an issue

[TimQSO]

I can confirm this is still an issue in 2.2.7 too. Removing the ttl solves it, but why doesn't it work with the ttl?

@piotrekkaminski I think this ticket should be reopened?

[TimQSO]

An update, I don't think that it is caused by core Magento in our case. What I did:

Change php.ini, to display errors:

display_errors = on

Then I noticed an exception error where normally the menu is. It was from a plugin.

I disabled the plugin, ran setup:di:compile and rechecked.

The error was gone, while ttl="3600" still was enabled in default.xml. Also I didn't need to use the Varnish 'p feature=+esi_ignore_https ' option.

When I enabled the plugin again, the problem came back. Then, when I removed the ttl="3600" from default.xml the plugin was working.

In my case the Bss_MetaTagManager module (BSS Commerce Meta Tags Template for Magento 2) was the module that was causing the issues.

[joshdavenport]

We're seeing this issue in a default environment provided to us by a mainstream Magento focussed hosting provider. Pretty poor state of affairs that the application should experience this issue so immediately.

Has anybody out there definitively found out the internal reason this happens without resorting to hacks?

[gcampedelli]

Please check it => https://pt.slideshare.net/TonyBrown44/magento-2-and-avoiding-the-rabbit-hole

[gcampedelli]

And it can also be a plugin. Have you checked if billing address is displaying all right.
In one of our install we had a problem with Mega Menu module overwriting menu trying to avoid ttl and it breaks details billing address in checkout

[joshdavenport]

Thanks for the link @gcampedelli, that was helpful. You're right, just found the controller in use was triggering a 500 as a result of something else.

Perhaps this will be useful for those with this issue and stumbling upon this ticket:

1. Check varnishlog for /page_cache/block/esi/blocks/ requests. Useful to run varnishlog -q 'ReqURL ~ "^/page_cache/block"' so you only see what you're interested in.
2. Check RespStatus and RespReason. Chances are you're seeing a 500 Internal Server Error.
3. Worthwhile hitting the URL directly here just to verify our theory, grab that from ReqURL. Behold, no results! This is the crux of the problem.
4. Next step is to nail the issue down, so get looking through logs, something will be causing an error somewhere. Can't guide this process.

IMO the real problem here is the obfuscation of this error during the ESI. It absolutely explains my despair at this issue, and I bet it also explains the frustration expressed by many in this ticket.

Is there a varnish limitation here that some alternative output can't be grabbed in this case? There really should be one of the following as some kind of hint here, otherwise it fails silently:

• Grab the block some other way as a fallback
  • Using <esi:remove>?
• Log that this happened to a Magento log where I found nothing during this whole process
  • Probably hard given <esi:include> is handled outside of control of Magento and if there is no log because of the error at the URL requested, that's no different to there being no error on some other page where there is no error in exception.log - i.e. not Magentos problem
• Output something to help hint something went wrong: <!-- ESI failed: 500 -->
  • By adding something that does this to the default Magento generated VCL - https://varnish-cache.org/lists/pipermail/varnish-misc/2016-February/024866.html)

[joshdavenport]

Also see varnishcache/varnish-cache#3264 which could affect the developer experience of this kind of issue in the future

[gcampedelli]

Hi Josh, you are right. Default Magento Varnish.vcl does not work in all kind of installs. I modified it as well to make Varnish work in some cases.

[gaiterjones]

I am seeing this with Varnish 6.x and Magento 2.3.x and 2.4.0. I followed @joshdavenport 's advice to filter the varnish logs with

varnishlog -q "ReqURL ~ '/page_cache/block/'"

And browsed to the ESI block

/page_cache/block/esi/blocks/%5B"catalog.topnav"%5D/handles/WyJkZWZhd...

This returned a 503 and the following errors in the varnish log

• VCL_return restart
• VCL_Error Too many restarts
• RespProtocol HTTP/1.1
• RespStatus 503

The too many restarts error lead me to issue #24353 where another possible solution is to add

if (req.restarts > 0) { set req.hash_always_miss = true; }

To the varnish vcl_recv.

I am reverse proxying to varnish with nginx using full https for Magento 2 and already implemented feature=+esi_ignore_https. It's worth mentioning that I only see the "mysterious disappearing menu" with Varnish 6.x not with Varnish 5.