Varnish ESI blocks don't load when HTTPS is used site-wide

[erikhansen]

This issue is related to #2657 that the OP closed.

Steps to reproduce

1. Install Magento 2.0.2 with sample data
2. Configure all urls to load from HTTPS (e.g., set "Base URL" and "Secure Base URL" to https://example.dev)
3. Load frontend and you'll see the navigation present.
4. Configure and enable Varnish (see documentation)
5. Load the frontend and you'll see the main navigation is missing:
   

Potential Solutions

1. Modify Varnish ESI to ignore HTTPS

Per this Stack Exchange answer, if you modify your /etc/sysconfig/varnish file and add -p feature=+esi_ignore_https to the arguments passed to Varnish, the problem will be solved. For example:

DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
             -f ${VARNISH_VCL_CONF} \
             -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
             -p feature=+esi_ignore_https \
             -p thread_pool_min=${VARNISH_MIN_THREADS} \
             -p thread_pool_max=${VARNISH_MAX_THREADS} \
             -S ${VARNISH_SECRET_FILE} \
             -s ${VARNISH_STORAGE}"

This solution is merely a workaround for the fact that Magento 2 returns HTTPS urls as ESI urls. Also, this option is only available in Varnish 4.

2. Modify Magento's \Magento\PageCache\Observer\ProcessLayoutRenderElement::_wrapEsi method to never return HTTP urls

Per the #2657 issue raised by @boldhedgehog, the proper solution is to modify Magento to only return HTTP urls to Varnish. How exactly this is handled is up for discussion, as I don't think it would be proper to simply replace "https" with "https".

[mazhalai]

@erikhansen Thank you for reporting, we have created MAGETWO-51149 to investigate and fix.

[rip057]

why on earth is varnish the "go to" app when there is no ssl support? and +1 for my navigation bar not showing up, with or without the work around. also someone should fix the "never return HTTP urls"
and the [simply replace "https" with "https"] it is a little confusing.

[erikhansen]

@rip057 You can use a server like Nginx (example config) or Pound as an SSL terminator to pass through requests to Varnish so that HTTPS requests can still take advantage of Varnish's memory-based cache system. So while it doesn't natively support SSL, it can still work with SSL by having a system sitting in front of it handling the SSL connection.

[rip057]

I've decided to dive in to apache mod_mem_cache, and mod_file_cache, and
customize it. it just sucks that magneto is kind of pointing you, me and
anyone who doesn't want to look around, towards something that is
inherently lacking right now. I don't know a single website that doesn't
want a full green bar on their web link name.
On Apr 18, 2016 11:40 AM, "Erik Hansen" notifications@github.com wrote:

@rip057 https://github.com/rip057 You can use an SSL terminator like
Nginx or Pound so that HTTPS requests can still take advantage of Varnish's
memory-based cache system.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#3897 (comment)

[rip057]

Also sorry for being harsh, I just spent quite a lot of time getting
magneto 2.0 in, then scratching and starting over, then again, only to
find out about varnish at the end, and the only reason I found out about
varnish is my effing navigational bar took a vacation, and I spent another
several hours trying to figure out if I caused it by disabling something.
Anyhow has anyone tried mod_mem_cache? Have any xp with it?
On Apr 18, 2016 12:05 PM, "Chris Clark" rip057@gmail.com wrote:

I've decided to dive in to apache mod_mem_cache, and mod_file_cache, and
customize it. it just sucks that magneto is kind of pointing you, me and
anyone who doesn't want to look around, towards something that is
inherently lacking right now. I don't know a single website that doesn't
want a full green bar on their web link name.
On Apr 18, 2016 11:40 AM, "Erik Hansen" notifications@github.com wrote:

@rip057 https://github.com/rip057 You can use an SSL terminator like
Nginx or Pound so that HTTPS requests can still take advantage of Varnish's
memory-based cache system.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#3897 (comment)

[erikhansen]

@rip057 There is certainly a learning curve with Magento 2, but after having worked with it for ~9 months (after 7 years on M1), I think it is a great platform that will server merchants well for many years to come.

Magento 2 decided to ship with Varnish support natively as many merchants and solutions partners implemented Varnish on M1 to build highly scalable and performant sites. However using Varnish as a cache system, while "recommended", is optional. For merchants that don't have large amounts of traffic (especially burst traffic), they can just use the native Magento caching system and forego Varnish altogether.

[rip057]

But still you got no opinion on mod_cache? It is also a memory cache, I
mean basically the same thing as varnish
On Apr 18, 2016 12:15 PM, "Erik Hansen" notifications@github.com wrote:

@rip057 https://github.com/rip057 There is certainly a learning curve
with Magento 2, but after having worked with it for ~9 months (after 7
years on M1), I think it is a great platform that will server merchants
well for many years to come.

Magento 2 decided to ship with Varnish support natively as many merchants
and solutions partners implemented Varnish on M1 to build highly scalable
and performant sites. However using Varnish as a cache system, while
"recommended", is optional. For merchants that don't have large amounts of
traffic (especially burst traffic), they can just use the native Magento
caching system and forego Varnish altogether.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#3897 (comment)

[erikhansen]

@rip057 Many people have discontinued using Apache in favor of Nginx, so if Apache isn't already a part of your technology stack, it would make more sense to use an application designed for the specific purpose of being a memory-based cache, rather than a module for a more general-purpose server.

Since Magento natively integrates with Varnish, I wouldn't spend my time investigating an alternative cache system like Apache+mod_cache.

[rip057]

Maybe, it probably like what flavor do you run kde or gnome, personal
choice I guess, I just have an affinity for apache. They have done so much
for the world in general
On Apr 18, 2016 4:20 PM, "Erik Hansen" notifications@github.com wrote:

@rip057 https://github.com/rip057 Many people have discontinued using
Apache in favor of Nginx, so if Apache isn't already a part of your
technology stack, it would make more sense to use an application designed
for the specific purpose of being a memory-based cache, rather than a
module for a more general-purpose server.

Since Magento natively integrates with Varnish, I wouldn't spend my time
investigating an alternative cache system like Apache+mod_cache.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#3897 (comment)

[davidalger]

@rip057 One thing to keep in mind here is also that Magento 2 is designed to scale, and the web stack that's officially supported must keep that in mind. Varnish is a scaling tool. In a clustered deployment scenario, it's very common to terminate SSL in the load-balancer. In that situation, varnish lacking SSL termination support doesn't really matter since it's not needed in the reverse proxy layer. Where SSL termination is not in the load-balancer, we are setting up an nginx -> varnish -> nginx sandwich, using nginx for SSL termination, then passing on to varnish for the app acceleration, and then on to nginx + php-fpm to actually run the application. This results in a still very lightweight setup. Similar could be accomplished with apache were one to take the time to build appropriate configuration files.

[rip057]

So does anyone have any experience with both? Anyone have experience at all
with mod_cache or mod_socache_shmcb. I get they are relatively new to the
release section of apache. The whole navi bar, is the only reason for all
ofthis, although I'm sure there are other broken parts

[davidalger]

@rip057 That is really a question better suited for the forums, as we need to keep the issue tracker on it's core focus, technical issues / bugs with the Magento 2 platform. I would recommend you take that over to the Community Forums or Magento Stack Exchange site. Thanks!