Data transfer between web server and redis cache is too high

[onlinebizsoft]

Summary (*)

We run a Magento site on multiple servers on multiple aws regions, the website have multple domains and multiple languages and the catalog has many products as well.

We relealized that the data transfer cost on the aws bill is quite much higher than normal. It cover up to 50-60% the total.

We figured out that most of data transfer was from Elasticache (one region) to EC2.

Examples (*)

Proposed solution

From our side we are looking at 2 things

1. Setup multiple regions for elasticache
   https://aws.amazon.com/blogs/database/reduce-cost-and-boost-throughput-with-global-datastore-for-amazon-elasticache-for-redis/
2. Enable L2 caching
   https://devdocs.magento.com/guides/v2.4/config-guide/cache/two-level-cache.html

I think the code core need to be re-worked.

---

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

• Severity: S0 - Affects critical data or functionality and leaves users with no workaround.
• Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

[m2-assistant]

Hi @onlinebizsoft. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[onlinebizsoft]

P/S Im on 2.4.2 so it is not same as #32118

[pmonosolo]

P/S Im on 2.4.2 so it is not same as #32118

^^^That issue is still present on 2.4.2-p1

[onlinebizsoft]

@matiashidalgo @Vinai @hostep @mrtuvn @DavorOptiweb

[mrtuvn]

Seem much complex with use-case multiple domain? Sorry but i dont have much experiences at this case ! You should add some details. What version that you are experiencing this problem. Any customises modules used

[onlinebizsoft]

@mrtuvn yes, multiple domain, multiple websites

Im on 2.4.2 with many customization however I believe there is nothing causing this. Our Magento installation is around 20GB (without media and without databases

[IbrahimS2]

@onlinebizsoft Have you configured FPC to utilise redis?

[onlinebizsoft]

@IbrahimS2 no, we end up with using single zone for the system now so all EC2 and elasticache instances are on same zone (cut down 50%-60% the bill)

[IbrahimS2]

@onlinebizsoft Please share the Redis configuration section from your env.php.

[onlinebizsoft]

@IbrahimS2
............
'x-frame-options' => 'SAMEORIGIN',
'MAGE_MODE' => 'production',
'cache' => [
'frontend' => [
'default' => [
'backend' => 'Cm_Cache_Backend_Redis',
'backend_options' => [
'server' => 'xxxxxxxxxxxxxxxxxxxxxxxxx',
'port' => '6379',
'persistent' => '',
'database' => 1,
'password' => '',
'force_standalone' => 0,
'connect_retries' => 2,
'read_timeout' => 10,
'automatic_cleaning_factor' => 0,
'compress_tags' => 1,
'compress_data' => 1,
'compress_threshold' => 20480,
'compression_lib' => 'gzip'
]
]
]
],
'session' => [
'save' => 'redis',
'redis' => [
'host' => 'xxxxxxxxxxxxxxxx',
'port' => '6379',
'password' => '',
'timeout' => '2.5',
'persistent_identifier' => '',
'database' => '0',
'compression_threshold' => '2048',
'compression_library' => 'gzip',
'log_level' => '1',
'max_concurrency' => '16',
'break_after_frontend' => '5',
'break_after_adminhtml' => '30',
'first_lifetime' => '600',
'bot_first_lifetime' => '60',
'bot_lifetime' => '7200',
'disable_locking' => '1',
'min_lifetime' => '60',
'max_lifetime' => '2592000'
]
],
'cache_types' => [
'config' => 1,
'layout' => 1,
'block_html' => 1,
'collections' => 1,
'reflection' => 1,
'db_ddl' => 1,
'compiled_config' => 1,
'eav' => 1,
'customer_notification' => 1,
'config_integration' => 1,
'config_integration_api' => 1,
'full_page' => 1,
'config_webservice' => 1,
'translate' => 1,
'vertex' => 1
],..........................

[mrtuvn]

cc: @vzabaznov may know better at this section

[onlinebizsoft]

Please keep in mind that data transfer between EC2 and Redis it is much much bigger than response data to traffic (from both nginx and varnish)

I'm thinking it can be possible that some private data ajax action could cause data transfer? So these ajax actions might serve very small data but it still request and transfer big data from redis?

[mrtuvn]

about ajax request seem we have fixed this case here
#31933 Landed in 2.4.3