High Database Load for Sales Rule Validation

[rauberdaniel]

Preconditions (*)

1. Magento 2.2.5
2. PHP 7.1 / MySQL 5.6.41

Steps to reproduce (*)

1. Create multiple Sales Rules with multiple coupons each
2. Get a lot of customers to redeem codes

Expected result (*)

1. High Performance, as finding sales rules is fast using indexes on Database

Actual result (*)

1. Incredibly Low Performance, as SQL query is badly constructed so it cannot use indexes

Description

We’re currently having about 400 sales rules (most of them are inactive or expired) as well as about 600.000 coupons. The query, that is currently being used by the sales-rule module is extremely inefficient on the database and therefore results in massive database CPU consumption when having several orders with coupons per minute. However, this query can greatly be optimized by splitting it into two queries.

The query that is generated by the setValidationFilter method in magento/vendor/magento/module-sales-rule/Model/ResourceModel/Rule/Collection.php currently looks like this:

SELECT `main_table`.*, `rule_coupons`.`code`
FROM `salesrule` AS `main_table`
INNER JOIN `salesrule_website` AS `website`
ON website.website_id IN ('1') AND main_table.rule_id = website.rule_id
INNER JOIN `salesrule_customer_group` AS `customer_group_ids`
ON main_table.rule_id = customer_group_ids.rule_id AND customer_group_ids.customer_group_id = 0
LEFT JOIN `salesrule_coupon` AS `rule_coupons` ON main_table.rule_id = rule_coupons.rule_id AND main_table.coupon_type != 1

WHERE (from_date is null or from_date <= '2018-11-01')
AND (to_date is null or to_date >= '2018-11-01')
AND (`is_active` = '1')
AND
(
    main_table.coupon_type = 1
    OR
    (
        (
            (main_table.coupon_type = 3 AND rule_coupons.type = 1)
            OR
            (main_table.coupon_type = 2 AND main_table.use_auto_generation = 1 AND rule_coupons.type = 1)
            OR
            (main_table.coupon_type = 2 AND main_table.use_auto_generation = 0 AND rule_coupons.type = 0)
        )
        AND
        rule_coupons.code = 'COUPONCODE'
        AND
        (rule_coupons.expiration_date IS NULL OR rule_coupons.expiration_date >= '2018-11-01')
    )
)
AND (`is_active` = '1') ORDER BY sort_order ASC;

The problematic part in this query is line 14 main_table.coupon_type = 1 OR. This part forces the database to filter the joined tables based on coupon_type. By simply removing that line, the rule_coupons.code can be used as an index to find exactly the one matching sales rule without having to join and filter a massive amount of rows. The line 14 is a completely separate case than the rest in that scope (line 13 to 29) and therefore should be treated separately e.g. by a second query in a UNION operation.

Some details using the MySQL EXPLAIN function:

Original Query:

Query without line 14/15 (not added in a second query in this example, therefore ignoring sales rules without coupons):

This is a major performance improvement especially for larger amounts of sales rules and coupons and absolutely critical.

[magento-engcom-team]

Hi @rauberdaniel. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento-engcom-team give me $VERSION instance

where $VERSION is version tags (starting from 2.2.0+) or develop branches (for example: 2.3-develop).
For more details, please, review the Magento Contributor Assistant documentation.

@rauberdaniel do you confirm that you was able to reproduce the issue on vanilla Magento instance following steps to reproduce?

• yes
• no

[rauberdaniel]

To be a bit more precise, the alternative recommended query that has the equal result but split into two queries that are very fast because of index usage is:

SELECT `main_table`.*, `rule_coupons`.`code`
FROM `salesrule` AS `main_table`
INNER JOIN `salesrule_website` AS `website`
ON website.website_id IN ('1') AND main_table.rule_id = website.rule_id
INNER JOIN `salesrule_customer_group` AS `customer_group_ids`
ON main_table.rule_id = customer_group_ids.rule_id AND customer_group_ids.customer_group_id = 0
LEFT JOIN `salesrule_coupon` AS `rule_coupons` ON main_table.rule_id = rule_coupons.rule_id AND main_table.coupon_type != 1

WHERE (from_date is null or from_date <= '2018-11-08')
AND (to_date is null or to_date >= '2018-11-08')
AND (`is_active` = '1')
AND
   (
        (
            (main_table.coupon_type = 3 AND rule_coupons.type = 1)
            OR
            (main_table.coupon_type = 2 AND main_table.use_auto_generation = 1 AND rule_coupons.type = 1)
            OR
            (main_table.coupon_type = 2 AND main_table.use_auto_generation = 0 AND rule_coupons.type = 0)
        )
        AND
        rule_coupons.code = 'COUPONCODE'
        AND
        (rule_coupons.expiration_date IS NULL OR rule_coupons.expiration_date >= '2018-11-08')
    )
AND (`is_active` = '1')

UNION

SELECT `main_table`.*, NULL as code
FROM `salesrule` AS `main_table`
INNER JOIN `salesrule_website` AS `website`
ON website.website_id IN ('1') AND main_table.rule_id = website.rule_id
INNER JOIN `salesrule_customer_group` AS `customer_group_ids`
ON main_table.rule_id = customer_group_ids.rule_id AND customer_group_ids.customer_group_id = 0
WHERE (from_date is null or from_date <= '2018-11-08')
AND (to_date is null or to_date >= '2018-11-08')
AND (`is_active` = '1')
AND (main_table.coupon_type = 1)

ORDER BY sort_order ASC;

The first query will then use rule_coupons.code as the index to retrieve that one coupon as well as the respective sales rule, the second query has to filter the sales rules table which however is smaller than the coupon table (on any joined combination). In our case by a very big factor as one sales rule can have multiple thousands of individual coupons.

[orlangur]

@rauberdaniel great finding and analysis! Are you interested in preparing a pull request maybe?

[rauberdaniel]

@orlangur I absolutely would, however, I’m not very into Magento code (or PHP at all) and the class that defines this query as well as the abstract are pretty confusing to me.

[orlangur]

@rauberdaniel got it, no problem. Let's wait for a volunteer then.

[magento-engcom-team]

Hi @engcom-backlog-nazar. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 5. Verify that the issue is reproducible on 2.2-develop branch.

  Details

  - Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

• 6. Add label Issue: Confirmed once verification is complete.

• 7. Make sure that automatic system confirms that report has been added to the backlog.