Carrier codes with '_' (underscores) break several payment API's

[thlassche]

Preconditions

1. Magento version >= 2.2

Steps to reproduce

1. Create a carrier with carrier code 'test_carrier' for example config.xml:

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:module:Magento_Store:etc/config.xsd">
    <default>
        <carriers>
            <test_carrier>
                <active>1</active>
                <title>Flatrate</title>
                <name>Flatrate 1</name>
                <price>0</price>
                <model>Namespace\Vendor\Model\Carrier\Flatrate1</model>
                <sallowspecific>0</sallowspecific>
                <specificerrmsg>This shipping method is not available.</specificerrmsg>
            </test_carrier>
      </carriers>
</default>
</config>

2. Login as customer
3. Go to the checkout, select created carrier and see that for example POST set-payment-information fails

Expected result

1. Order placement works

Actual result

1. Several web api request fail with 400 bad request, please specify a shipping method

The cause of this is several explode's by '_' in the code, where $addressInformation->getShippingCarrierCode() should have been used in my opinion. Example: https://github.com/magento/magento2/blame/2.3-develop/app/code/Magento/Checkout/Model/PaymentInformationManagement.php#L118

[magento-engcom-team]

@thlassche, thank you for your report.
We were not able to reproduce this issue by following the steps you provided. Please provide more detailed steps to reproduce.

[thlassche]

@magento-engcom-team I saw this when using it in combination with onestepcheckout. Please see here for the code that's definitely wrong in my opinion:

https://github.com/magento/magento2/blame/2.3-develop/app/code/Magento/Checkout/Model/PaymentInformationManagement.php#L118

You can't explode by '_', compare it with this one:

https://github.com/magento/magento2/blob/2.3-develop/app/code/Magento/Checkout/Model/ShippingInformationManagement.php#L160

[kanduvisla]

I have this same problem with Magento 2.1.12. And only with logged in customers. This problem does not seem to affect guest checkout.

The bpost shipping carrier has carrier codes with underscores, like:

• bpost_homedelivery
• bpost_pickuppoint
• bpost_clickcollect
• etc.

The code in Magento\Checkout\Model\PaymentInformationManagement::savePaymentInformation() explodes the selected shipping carrier by an underscore character and sets this as limit_carrier:

// Shipping method is bpost_homedelivery_bpost_homedelivery for example:
$shippingDataArray = explode('_', $shippingAddress->getShippingMethod());
$shippingCarrier = array_shift($shippingDataArray);
// Carrier is now 'bpost', which does not exist:
$shippingAddress->setLimitCarrier($shippingCarrier);

I'm still looking where something like getLimitCarrier is used to re-attach the shipping method to the quote address (and collect the proper rates), but for now this appears to be a serious issue. It effectively renders the checkout unable to complete an order for carriers that have an underscore in their name.

In my opinion there are two options here:

• Fix savePaymentInformation() to do extra checks for carriers with more underscores.
• Do some validation on carrier codes upfront for developers (like, immediately throw an exception that underscores are not allowed in carrier codes for example).

[kanduvisla]

As a temporary solution (at least for 2.1.12) I came op with a plugin that fixes this for 99.9%.

di.xml:

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:ObjectManager/etc/config.xsd">
    <!--
        Addresses GITHUB-13902, where logged in customers cannot use shipping carriers that contain an underscore
        @see https://github.com/magento/magento2/issues/13902
    -->
    <type name="Magento\Checkout\Model\PaymentInformationManagement">
        <plugin name="fix-carriers-with-underscore-bug" type="Vendor\Module\Plugin\Magento\Checkout\Model\PaymentInformationManagement"/>
    </type>
</config>

app/code/Vendor/Module/Plugin/Magento/Checkout/Model/PaymentInformationManagement.php:

<?php

namespace Vendor\Module\Plugin\Magento\Checkout\Model;

use Magento\Framework\App\Config\ScopeConfigInterface;
use Magento\Quote\Api\CartRepositoryInterface;
use Magento\Quote\Api\Data\AddressInterface;
use Magento\Quote\Api\Data\PaymentInterface;
use Magento\Quote\Api\PaymentMethodManagementInterface;
use Magento\Store\Model\ScopeInterface;

/**
 * Class PaymentInformationManagement
 * @package Vendor\Module\Plugin\Magento\Checkout\Model
 */
class PaymentInformationManagement
{
    /**
     * @var PaymentMethodManagementInterface
     */
    protected $paymentMethodManagement;

    /**
     * @var ScopeConfigInterface
     */
    protected $scopeConfig;

    /**
     * @var CartRepositoryInterface
     */
    protected $cartRepository;

    /**
     * PaymentInformationManagement constructor.
     * @param PaymentMethodManagementInterface $paymentMethodManagement
     * @param ScopeConfigInterface $scopeConfig
     * @param CartRepositoryInterface $cartRepository
     */
    public function __construct(
        PaymentMethodManagementInterface $paymentMethodManagement,
        ScopeConfigInterface $scopeConfig,
        CartRepositoryInterface $cartRepository
    ) {
        $this->paymentMethodManagement = $paymentMethodManagement;
        $this->scopeConfig = $scopeConfig;
        $this->cartRepository = $cartRepository;
    }

    /**
     * @param \Magento\Checkout\Model\PaymentInformationManagement $subject
     * @param callable $proceed
     * @param $cartId
     * @param PaymentInterface $paymentMethod
     * @param AddressInterface|null $billingAddress
     * @return bool
     * @throws \Magento\Framework\Exception\NoSuchEntityException
     * @throws \Magento\Framework\Exception\State\InvalidTransitionException
     */
    public function aroundSavePaymentInformation(
        \Magento\Checkout\Model\PaymentInformationManagement $subject,
        callable $proceed,
        $cartId,
        PaymentInterface $paymentMethod,
        AddressInterface $billingAddress = null
    ) {
        if ($billingAddress) {
            /** @var \Magento\Quote\Model\Quote $quote */
            $quote = $this->cartRepository->getActive($cartId);
            $quote->removeAddress($quote->getBillingAddress()->getId());
            $quote->setBillingAddress($billingAddress);
            $quote->setDataChanges(true);
            $shippingAddress = $quote->getShippingAddress();
            if ($shippingAddress && $shippingAddress->getShippingMethod()) {
                $shippingDataArray = explode('_', $shippingAddress->getShippingMethod());
                $shippingCarrier = array_shift($shippingDataArray);
                // Start fix for GITHUB-13902
                while (!$this->isExistingCarrier($shippingCarrier)) {
                    $shippingCarrier .= '_' . array_shift($shippingDataArray);
                }
                // End fix for GITHUB-13902
                $shippingAddress->setLimitCarrier($shippingCarrier);
            }
        }
        $this->paymentMethodManagement->set($cartId, $paymentMethod);

        return true;
    }

    /**
     * @param string $shippingCarrier
     * @return bool
     */
    protected function isExistingCarrier(string $shippingCarrier): bool
    {
        $carrierConfig = $this->scopeConfig->getValue('carriers/' . $shippingCarrier, ScopeInterface::SCOPE_STORE);
        return !empty($carrierConfig);
    }
}

The 0.1% that is not fixed is if there are carriers that have matching names with underscores. For example: carrier_foo and carrier_foo_bar. In this case, when carrier_foo_bar is selected, it will be rebuild and matched with carrier_foo. But as I said: it's a temporary fix and it's currently design to only work in my situation but perhaps it will help someone.

Another risk is an infinite loop if the store owner deletes or disables the selected carrier in between selecting the shipping method and completing the order :-P

[thlassche]

@kanduvisla Thanks, added the login step to the steps to reproduce section

[davidverholen]

having the same problem in 2.1.11 since the carrier code is just determined by exploding the shipping method with '_'. For every shipping carrier containing an underscore the checkout does not work anymore

see: https://github.com/magento/magento2/blob/2.1-develop/app/code/Magento/Checkout/Model/PaymentInformationManagement.php#L124

For 2.1 this was introduced in the update from 2.1.10 to 2.1.11

[hostep]

@irenelagno or @nmalevanec: since you guys introduced this code in 3495e60 and 179f984, would you be so kind to review the above discussion and see if there is a good solution for this problem?

@kanduvisla: would using a regular expression to try and extract the carrier work as well? Something like (.+)_\1 for example?
I haven't debugged this yet, so I don't know for certain if the first part and second part of the shipping method are always exactly the same with all shipping methods, but if they are the same, the regex should work I think?

[kanduvisla]

@hostep: I don't exactly understand what you mean. The problem is that Magento concatenates the carrier code and the shipping method as one string, and later on tries to 'guess' it back, using a string. This works great for codes like flatrate_flatrate, or dhl_express, but what if you have:

• A carrier foo with the method foo_foo, and:
• A carrier foo_foo with the method foo.

Now Magento gets the string foo_foo_foo. How can it determine which carrier / method are chosen here? I think the main issue here is an architectural one: because the shipping_method is saved as a concatenated string, issues like this are unavoidable. This can be fixed in various ways:

• Add a validation to carrier_code and shipping_method, stating that it cannot contain an underscore (might break several carrier modules).
• Use something else as glue for the concatenating (for example: carrier_code::shiping_method). Might break BC for older quotes / orders that already exist in the database).
• Split shipping_method-column in database with extra carrier_code-column. Might be a big commit to the code, but is more in line with the carrier/method-model, and BC can be achieved by checking if carrier_code is null in the database.

edit: @hostep: upon reading your comment again I do know what you mean and no: carrier code and shipping method are not always the same. They are often the same if a carrier only has one method, but it's perfectly fine for a carrier to have multiple shipping methods. In that case the code of the shipping method differs.

[koenner01]

Imo this is a pretty serious issue; If Magento would choose not to allow an underscore in the carrier code & method there will be a lot of people unhappy and a lot of extra work will be needed for people to fix this.

Of the above solutions I would prefer the 3th option as it indeed follows carrier/method model.

[pocallaghan]

In M1, I've seen shipping method codes with _ in, but never a carrier code. This was partially enforced by the code in Mage_Sales_Model_Order::getShippingMethod, which does this list($carrierCode, $method) = explode('_', $shippingMethod, 2); (looks like it does similar in M2). Whilst there may be different usage in other places, this code sort of implies theres a contract that a carrier code can't include an underscore.

[hostep]

Thanks for the feedback all! I've had some chance to dig into this a bit further in the mean time.

So here's my opinion:
If Magento allowed the carrier code to have underscores in version 2.1.10 and before, then it should still allow to do this, otherwise we have BC breakage (like we have now in 2.1.11, 2.1.12 & 2.2.x).

So we need to replace the splitting of carrier & shipping method with underscore by something else, that was indeed M1 behavior, but I don't think it applies to M2 anymore. The carrier code should be available without extracting it out of another string.

The only thing we need is figure out a way to replace these lines with the carrier code which we need to get from somewhere.

The question is: where should we get it from? Possible solutions:

• when saving the payment information, send the carrier code from the frontend, as an extra param in the ajax request
• when storing the shipping method in the first step of the checkout, store it separately either in the database or maybe just in the session and get it from there when saving the payment information

Just my 2 cents.

Maybe @magento-engcom-team can try to invite some core devs in here to figure out the best solution for this problem?

[davidverholen]

although it might be a bigger change, I think it would solve the problem for good to separate carrier code and shipping method code.

They are separated (and also needed to be separated) nearly everywhere. Only at this point in the database for some reason they are merged. This really makes no sense.