L2 caching information and optimization

[onlinebizsoft]

Summary

We are a website with many store views and many products, our Magento redis cache is really big because how big our system is. I'm looking at L2 caching and hoping it will improve a bit our network transfer, network latency,......

However the document about L2 caching seems to be lacking technical information, it is not clear how it works because it is not possible to just store whole remote cache on the L2 cache and if the L2 cache is too small then even using L2 cache will just add extra load on the top (to save and remove L2 cache).

Can someone explain or share the experience with using this?
CC @hostep @georgebabarus @Nuranto @jonathanribas @vzabaznov @Adel-Magebinary @igorwulff @drew7721 @ihor-sviziev

Examples

L2 cache seems to just work as an extra cache level on local web server machine

Proposed solution

We should be able to configure L2 cache to keep some certain cache keys or find a way to make it more effective, otherwise it will be almost useless for a website with big Redis cache size.

Release note

No response

Triage and priority

• Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

[m2-assistant]

Hi @onlinebizsoft. Thank you for your report.
To speed up processing of this issue, make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, Add a comment to the issue:

• @magento give me 2.4-develop instance - upcoming 2.4.x release
• For more details, review the Magento Contributor Assistant documentation.
• Add a comment to assign the issue: @magento I am working on this
• To learn more about issue processing workflow, refer to the Code Contributions.

---

Join Magento Community Engineering Slack and ask your questions in #github channel.
⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.
🕙 You can find the schedule on the Magento Community Calendar page.
📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

[ihor-sviziev]

Sorry, we never user L2 cache in Magento

[onlinebizsoft]

P/S : In my user case, we have some quite big cache keys because number of websites/store views and number of different modules so our cache size is very big and L2 seems to be not applicable however we plan to update the logic to keep only certain cache key in L2 caching and the rest will go to remote cache storage. This will help to reduce data transfer over network and reduce remote Redis engine CPU to serve big keys
Just sharing our plan, if any one has a better idea, please feel free to discuss.

At beginning, I expected a magic things like "preload keys" should work specially with L2 caching but it doesn't. There is nothing specially behind the L2 caching logic.

[Nuranto]

Hi @onlinebizsoft

I'm not sure to understand why you don't want to use fully L2 caching. What would be the benefit of using it only for a part of cache ? A network call is costly, for small entries too.

[onlinebizsoft]

@Nuranto 1. We can't use it fully because our total Redis size is very big and it is much bigger than the total web server memory so it makes no sense to replicate everything on L2 which lead to save/remove operation very often
2. I know the main thing is network call (even we have all components within a private network). However I'm thinking some very big keys will have much more effect than other small ones so at least I can eliminate big keys from network transfer and I will
monitor the result from there.

Do you have a contact, I can connect you and we can share some experience how it is going?

[Nuranto]

Maybe the issue is that you have too much cache than necessary ? How much memory is redis using ?
Do you use remote storage ? There's a known bug that produce huge amount of unnecessary cache : #35820

[onlinebizsoft]

@Nuranto you know a website with 100K products can be few GB of cache and the website has like 100 websites on the same installation so you can imagine how big it can be. And of course, we don't have any kind of web server with so much memory and memory should need to be spent for PHP processes instead of caching.

[ihor-sviziev]

I just wonder, maybe separating the cache for each website might help you?

[onlinebizsoft]

@ihor-sviziev yes actually it was an option I was thinking about. However it will make the infrastructure more complex (deployment process,....) and it will lead to ineffective sharing resource, we always need to reserve enough resource for each group of websites. So I'm trying with simpler options first.

[Nuranto]

We have as much products, but only 3 websites indeed. I suppose that makes the difference. Also, we have servers nodes with 192Gb of RAM, which surely helps...
I think there's no other option left for you than rewrite (or write a new) L2 caching (see RemoteSynchronizedCache class)

[onlinebizsoft]

An off-topic question, do you use persistent database connection (and/or Redis persistent connection) @Nuranto @ihor-sviziev ?

[ihor-sviziev]

nope :(

[jonathanribas]

Hi @onlinebizsoft, sorry for the late reply, we did some tests yesterday enabling L2 cache on EFS (not /dev/shm).

We have very good results regarding network usage on Redis side, it has clearly dropped!

Network usage without L2 cache:

Network usage with L2 cache enabled:

BUT we have very poor results in terms of PHP response consumption when enabling L2 cache:

PHP transaction time without L2 cache:

PHP transaction time with L2 cache:

[onlinebizsoft]

@jonathanribas it is a bit strange, currently we have tried to enable L2 on our development environment and we didn't notice such slowness. We are not ready to go live yet (our L2 implementation is a custom one to apply our logic for which key to be on L2 only)

[jonathanribas]

@onlinebizsoft, I've run this same test several times: without / with / without / with L2 cache and I got same results. Those environments are clean and without traffic.

[m2-assistant]

Hi @engcom-Hotel. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue.
• 3. Add Area: XXXXX label to the ticket, indicating the functional areas it may be related to.
• 4. Verify that the issue is reproducible on 2.4-develop branch
  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
• 5. Add label Issue: Confirmed once verification is complete.
• 6. Make sure that automatic system confirms that report has been added to the backlog.

[engcom-Hotel]

Hello @onlinebizsoft,

Thanks for the report and collaboration!

After going through the main description, seems like it can be a good feature to have, to configure the L2 cache to keep some certain cache keys or find a way to make it more effective.

Hence marking this issue as a feature request.

Thanks

[JamesFX2]

@jonathanribas why did you choose to use EFS rather than /dev/shm - this is surely built around use of in-memory store rather than disk reads of up to hundreds of identifiers?

@onlinebizsoft did you ever follow up on this? To get what you'd want would probably involve storing last access times and purging based on that.

[onlinebizsoft]

@jonathanribas why did you choose to use EFS rather than /dev/shm - this is surely built around use of in-memory store rather than disk reads of up to hundreds of identifiers?

@onlinebizsoft did you ever follow up on this? To get what you'd want would probably involve storing last access times and purging based on that.

I dont think it is a good approach because it will add too much logic and extra processing.

[JamesFX2]

@onlinebizsoft depends how you implement it, I agree adding a write to every read isn't the best but there are new approaches for storing in-memory cache. I am not an expert in this field but I've heard people talking enthusiastically about Swoole (in the context of explaining Laravel Octane).

[jonathanribas]

@JamesFX2, we run M2 on k8s and we don't want to change k8s default size setting for /dev/shm which is 64mb.

[onlinebizsoft]

@engcom-Hotel do we have any move on this? Im sure that the strongest point of Magento would be supporting complex system so optimizing for 100 stores system would be considered as high priority.

[Adel-Magebinary]

There might be a bug in paypal module. See here #32118 (comment)

The issue was fixed after disabling the module. CC: @onlinebizsoft

[Adel-Magebinary]

This is how you use L2 caching to save some network traffic.

    'cache' => [
        'frontend' => [
            'default' => [
                'backend' => '\\Magento\\Framework\\Cache\\Backend\\RemoteSynchronizedCache',
                'backend_options' => [
                    'remote_backend' => '\\Magento\\Framework\\Cache\\Backend\\Redis',
                    'remote_backend_options' => [
                        'server' => 'redis-cluster-node-3',
                        'database' => '3',
                        'port' => '6379',
                        'compress_data' => '0',
                        'timeout' => '10'
                    ],
                    'local_backend' => 'Cm_Cache_Backend_File',
                    'local_backend_options' => [
                        'cache_dir' => '/dev/shm/'
                    ],
                    'use_stale_cache' => false
                ],
                'frontend_options' => [
                    'write_control' => false
                ]
            ],
            'type' => [
                'default' => [
                    'frontend' => 'default'
                ]
            ],
            'page_cache' => [
                'id_prefix' => 'ae1_',
                'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
                'backend_options' => [
                    'server' => 'redis-cluster-node-3',
                    'database' => '2',
                    'port' => '6379',
                    'compress_data' => '0',
                    'timeout' => '10'
                ]
            ]
        ],