What varnish module to use?

[fballiano]

Hi everybody, openmage + php7.4 is extremely fast and, with my customers, we're super happy about it.

I've been using different FullPageCache solutions over the years (turpentine+varnish, some proprietary php based PFC extensions etc) and every time they were 90% working but we were constantly having small problems here and there.

So I'm here to ask you: what varnish extension do you think it's the best and most modern? Is it still turpentine? the official repo or some fork/branch (because the official one doesn't get an update since looooong time)?

Thanks everybody!

[addison74]

This question should be in Discussions section.

For many years I was close to the Nexcess Turpentine project which was a very active one. In my opinion, the extension offered high performance when loading pages on crowded servers. If with an optimized PHP you could reach 76 requests per second with Varnish you could reach 2,000. I also used HHVM for a while but gave up when PHP 7.0 came out.

Unfortunately, those who maintained the project gave up without even handing over the baton to other people. Lately, Miguel Balparada was very bored to answer more questions, and if he did, he would not offer a solution. Thus, there are a lot of PRs that have not been integrated in the same way, there are problems where no answer has been given since mid of 2018. I have a great regret for what happened there.

The only issues I identified and could correct with ESI policy were related to the use of AJAX in pages. Otherwise it still works today, I even have an analysis of the extension with Varnish 6.1.1 where I identified only small correctable issues. FPC from different developers is not quite fast as Varnish based on a good configuration.

One person who supported the project is Jeroen Vermeulen who has been creating many PR's and seeing that they are not integrated has incorporated them into his own fork. You can find here his version: https://github.com/jeroenvermeulen/magento-turpentine.

If this project were resumed it would be a solution for OM especially since now VDS or DS have become cheaper than years ago and you can install ElasticSearch, Sphinx, Varnish to improve performance.

[m-overlund]

Do you know whether the said Varnish 6.1.1 patches is fixed in Jeroen's fork?

[addison74]

Here is one of my posts in Nexcess Turpentine repository: nexcess/magento-turpentine#1528. I did not test Joroen's fork but it seems he integrated important PR's he promoted. Anyway let's say it is a little bit updated then original.

If there is interest we can make a fork and continue where Turpentine was left back in 2018. We need maintainers and people with enthusiasm. I think that if a message were posted on the extension page that it continues "somewhere", those who were disappointed with the way this project was managed will surely come back.

Lesti abandoned FPC, but there are still good paid extensions around 150 - 250 USD. There are many articles for comparison between these extensions (Amasty if I remember) and Varnish. All say that it is not a true cache because always there is a first request which is not cached. It is a long discussion. From my experience I can say Varnish was able to reject a DDoS not big on one of my servers and it is a precious tool.

[fballiano]

One person who supported the project is Jeroen Vermeulen who has been creating many PR's and seeing that they are not integrated has incorporated them into his own fork. You can find here his version: https://github.com/jeroenvermeulen/magento-turpentine.

I've checked te jeroenvermeulen repo and it's great to have a new version but his repo doesn't allow for issues/pr so it's a bit limited, mmmm wondering if there's a way to resume something (maybe talking to jeroenvermeulen?) cause OM needs a reliable FPC module for the future (although the performance is very very good)

[addison74]

It is noteworthy that the Nexcess/Turpentine project has almost 270 forks. It is below OM by 80, which means that the project itself was of great interest to those who used M1. As I said the project can be invigorated if desired. It could start from a new variant and directs the community to the place with activity.

Placing Varnish in front of the web server involves configurations that are not for everyone. Personally, I will have to regain my knowledge about this project because it's been a while since I've used it. I was also the one who recommended using Pound in front of Varnish for SSL termination, there are still my configurations in the help pages. Now I can go with the new Pound version (not in Debian 10 repository - but 11), but it must be compiled from source, with HAProxy or Nginx. A problem that appears in Magento and that I solved with a configuration file is related to the fact that there is no longer just one IP in the order page but a chain of IP's. Magento itself was not provided for such situations.

[jfksk]

A bit OT and generalized; but as it was mentioned:

Integrated FPCs
There is unfortunately no straight forward way to implement them.

• A half way initialized framework introduces quiet a few problems. That is IMHO not the desirable way to do it even if the core suggests it.
• Initializing the framework still does not solve it as it will produce a lot of overhead if not handled.

I wrote one and ended up w/ lazy rendering/block construction and lazy init/loading for models. It is, THB, a monstrosity but it handles many cases while keeping the core functionality intact (memorized toolbar-params are ATM a wont fix) and it wont punish you w/ a broken factory and an unavailable config.
I'm still scratching my head if it could make sense or even would be possible, to integrate some of this into the core.
A server-side FPC is very hard to get right and is hard to integrate/configure right.

A server-side FPC is in theory more efficient but requires some work. One should not expect plug and play functionality - even if marketing suggest it. Scalability-wise, both solutions can make sense.
But, in the end, if you are selling super popular products, like the PS5, neither will help you to get thru this on it's own.
Scaling and caching is tricky.
Unfortunately I do not have a Varnish solution.

[fballiano]

I've tried Lesti_FPC and I've to say I'm pretty pleased, worked like charm and it's really fast.

[addison74]

Indead Lesti_FPC was a great extension. Even it worked for you please take a look here: https://github.com/GordonLesti/Lesti_Fpc/issues. I faced some of the them. I didn't have the patience to try solving them. Varnish is the best way and Magento 2 has it already. If we could continue the Trupentine project it will be a great addition to OpenMage and an excellent promotion tool for big stores.

[albertobraschi]

turpentine-varnish
This is a fork. Work in progress.

To install, run:

composer require "luigifab/openmage-turpentine-varnish":"dev-master" [--ignore-platform-reqs]
Compatibility: OpenMage 19.x / 20.x / 21.x, PHP 7.2 / 7.3 / 7.4 / 8.0 / 8.1
License: GNU GPL 2+

https://github.com/luigifab/openmage-turpentine-varnish

[addison74]

@luigifab is ours, He read our discussion and made a fork to joernvermulen.