Adds support for purging varnish cache based on an X-Pool header

[davidalger]

This adds the ability to default vcl templates to issue PURGE requests using an X-Pool header instead of the various headers Magento sets on it's responses.

Use case is flushing a specifc "pool" of cache records during automated deployments. It would be expected that the X-Pool header would be set in the Apache/Nginx config files if used.

[mazhalai]

@davidalger can you please resync with develop and restart builds?

[davidalger]

@mazhalai Rebased on latest develop. Let me know if you need anything else.

[mazhalai]

@davidalger we have created ticket MAGETWO-52293 to process this PR.

[piotrekkaminski]

@maksek can we process it?

[bwaters]

I tried googling for x-pool http header to see if this technique was used by others and didn't see any reference to this header. I want to understand the use case a little better.

Is the purpose to configure a sets of upstream http servers with different x-pool headers so that you can flush the caches produced by different sets of servers? Or tag the cache with version / deploy numbers so you can flush the cache associated with the previous deployment code?

[davidalger]

@bwaters Theoretically this could be used to accomplish nearly any varnish flushing behavior one desired, as it's value is entirely controlled by the server configurations, allowing a lot of flexibility in what one is flushing. This introduces a flexibility into the standard VCL so that publicly available tools can work completely with out-of-the-box configuration files.

Support for using this has been built into the davidalger/capistrano-magento2 gem which I've built and released for public use. It's not quite yet documented in the readme for the gem yet, but the support is there to set the :varnish_cache_hosts and :ban_pools arrays in the capistrano deployment configuration, and then after the application is deployed, the gem will send the given varnish hosts a PURGE request with the specified X-Pool values.

As for how I am specifically using it: We are using it to ban everything related to a specific site from the varnish cache when the deploy is complete. The Nginx server configuration is using an add_header directive to set the header on all responses from the site, enabling the PURGE to ban only the records for that particular domain.

Regarding the X-Pool header not being found around the web…no, it's not a "standard" header name. It's a custom header just like the ones which Magento uses to tag it's responses for tag based varnish flushing, but in this case, it's application agnostic, and more relatable to the X-Server header in that it's identifying a sub-set of what is being served up by a given server and/or group of servers.

I'm really looking forward to seeing this merged in!

[erikhansen]

+1 I'm also interested in seeing an X-Pool header get added to the default Magento VCLs.

@davidalger You might be interested to check out this somewhat related pull request #4905

[buskamuza]

Can X-Magento-Tags be utilized for the same purpose? E.g., make Nginx add another tag for each request and then purge the cache by this tag?
The concern to the PR is that X-Pool is not used anywhere in Magento itself, so it's hard to track down why it's needed here for a person w/o the context.

[davidalger]

@buskamuza Sorry, I either did not see your question until now or it slipped by me…last few months have been crazy over here!

Can X-Magento-Tags be utilized for the same purpose?

In short, no, not really. Modifying that header with Nginx would be difficult, and wouldn't be something which would then serve the sole purpose of identifying which "pool" served the request.

The concern to the PR is that X-Pool is not used anywhere in Magento itself, so it's hard to track down why it's needed here for a person w/o the context.

You are correct that it's not used by Magento itself. This is a VCL file though, and closely related to server configuration. It is needed to support the increasingly widely used Capistrano based deployment setup using the capistrano-magento2 gem. There is functionality in this gem which I've been holding off on documenting until this PR is merged in. Folks are already trying to use the undocumented functionality in the deploy tool and finding that the VCL built-in to Magento 2 does not support the X-Pool header needed for the tool to flush varnish by pool to work.

As for me personally, I don't use the stock VCL anymore, as things like this not being merged in have forced me to maintain my own VCL for M2 hosting setups. Please consider getting this simple enhancement merged in!

[erikhansen]

+1 vote for adding this to the default Magento VCL.

@buskamuza One of the beautiful things about M2 being an open-source project is that if developers are curious about the origin of a section of code, they can always run a git blame against a file to see commit history about certain lines. For example: https://github.com/davidalger/magento2/blame/daddb881cbe966e16f237266d29036dd0d2aecd6/app/code/Magento/PageCache/etc/varnish4.vcl#L20

[buskamuza]

@davidalger , sorry, looks like I missed your answer too.
I'm OK with accepting the PR. Would you mind adding some explanation comments in the vcl files, so that it will be more clear why those lines are needed though they're not used in Magento by default?

[davidalger]

@buskamuza I will add comments. Thanks!

[vrann]

@davidalger the change looks awesome! Can you please update the source code? Magento does not have vanish3.vcl in the mainline, instead, there is varnish5.vcl

[davidalger]

@vrann I've just rebased on develop and updated to affect the varnish5.vcl vs the old v3 config file. Also added a comment to each file as @buskamuza had requested.

Let me know if anything else is needed to see this merged in. Thanks!

[vrann]

@davidalger perfect, thanks, will be merged.