New Indexer for StockItems

Table of Contents

• What we need to index
• Index Dimensions
• Index Aliases and Zero Downtime
• Pull Request for StockItem Index

What we need to index

Building new Inventory implementation which supports multi source/stock capabilities. We need to calculate the stock quantities for SKUs per each particular Sales Channel (website or store). For these purposes, we have StockItem entity which represents an Aggregated Virtual Stock. Which is used as Read-only data generated in a result of the re-indexation process based on pre-defined mapping which defines a binding between Stock objects and Sources assigned to them (SourceStockLink on the diagram above).

For example, there are 3 physical sources (A, B and C) which assigned to specific sales channel (for example, Magento Website). If Customer comes to this website and wants to buy some product (Product1), we should show him that we have next amount of this product: Qty of Product1 on A + Qty of Product1 on B + Qty of Product1 on C This is what StockItem does.

Read more about this in MSI Technical Vision document.

As rules (SourceStockLink) we use for StockItems creation are added dynamically and could be changed anytime. It's convenient to introduce dedicated index which would store StockItem data. Doing so we offload our front-end as we no need to calculate aggregations dynamically at a time of users' request. All the aggregations would be pre-calculated in advance and we need just to refer these data to use. So that we would not affect performance adding multiple sources functionality.

Index Dimensions

Taking into account that potentially StockItem index could contain a lot of data, as we have a multiplication of sales channels on a number of SKUs.

For Big Merchants, who have a lot of SKUs and a lot of Sales channels (scopes in which products would be sold). We want to prevent a multiplication of data in the index table : SKUs Qty * Number of Sales Channels

Because that could introduce performance impact both on Read and Write index operations.

That's why it's proposed to split indexes by the Scope. Taking into account that we have 1 - 1 relation between Scope (Sales Channel) and Stock (StockSalesChannelLink on the diagram above). We could Build independent index table per each Stock (Stock_id).

Stock Id could be used as Dimension which will help us to resolve correct index we need to refer to get Product Qty.

For example, having 5 websites with ids (1 .. 5) we will end up creating 5 different index tables (with the same structure):

inventory_stock_item_stock_1
inventory_stock_item_stock_2
inventory_stock_item_stock_3
inventory_stock_item_stock_4
inventory_stock_item_stock_5

There is already similar implementation in the CatalogSearch Index Magento\CatalogSearch\Model\Indexer\Fulltext Where we use Store View Id as a Scope and create independent indexes per each Store View. Which are usually used to store data per each language, so that index which holds data for Eglish locale doesn't overlap with index holding data for German one. To resolve which index (table) we should refer - we use current scope (Store View Id).

We already have an SPI interface Magento\Framework\Indexer\SaveHandler\IndexerInterface which helps us to work with Indexes which are segregated by dimensions.

namespace Magento\Framework\Indexer\SaveHandler;

use Magento\Framework\Search\Request\Dimension;

/**
 * Indexer persistence handler
 *
 * @api
 */
interface IndexerInterface
{
    /**
     * Add entities data to index
     *
     * @param Dimension[] $dimensions
     * @param \Traversable $documents
     * @return IndexerInterface
     */
    public function saveIndex(array $dimensions, \Traversable $documents);

    /**
     * Remove entities data from index
     *
     * @param Dimension[] $dimensions
     * @param \Traversable $documents
     * @return IndexerInterface
     */
    public function deleteIndex(array $dimensions, \Traversable $documents);

    /**
     * Remove all data from index
     *
     * @param Dimension[] $dimensions
     * @return IndexerInterface
     */
    public function cleanIndex(array $dimensions);

    /**
     * Define if engine is available
     *
     * @return bool
     */
    public function isAvailable();
}

Index Aliases and Zero Downtime

We have a requirement to have index aliasing for all Magento indexes during the full reindex happened, so we need to have switchable indexes to switch from an old index (current index, which exists before full reindex operation has been launched) to a new index (created when the full reindex operation finished) with zero downtime to make Front-End responsive while Full Reindex being worked.

Similar Implementation for Full Text Index:

namespace Magento\CatalogSearch\Model\Indexer;

/**
 * Provide functionality for Fulltext Search indexing.
 */
class Fulltext implements \Magento\Framework\Indexer\ActionInterface
{
...
    /**
     * Execute full indexation
     *
     * @return void
     */
    public function executeFull()
    {
        $storeIds = array_keys($this->storeManager->getStores());
        /** @var IndexerHandler $saveHandler */
        $saveHandler = $this->indexerHandlerFactory->create([
            'data' => $this->data
        ]);
        foreach ($storeIds as $storeId) {
            $dimensions = [$this->dimensionFactory->create(['name' => 'scope', 'value' => $storeId])];
            $this->indexScopeState->useTemporaryIndex();

            $saveHandler->cleanIndex($dimensions);
            $saveHandler->saveIndex($dimensions, $this->fullAction->rebuildStoreIndex($storeId));

            $this->indexSwitcher->switchIndex($dimensions);
            $this->indexScopeState->useRegularIndex();
        }
    }
...
}

Also, please take into account that when the SourceItem data changed -> we need to update StockItem Indexes for each of the Stock to which given Source was assigned. If Source was assigned to 3 stocks -> we need to refresh data in the 3 StockItem indexes

For this operation we need to introduce new API service

/**
 * Get assigned Stocks for Source command
 *
 * @api
 */
interface GetAssignedStocksForSourceInterface
{
    /**
     * Get Stocks assigned to Source
     *
     * @param int $sourceId
     * @return \Magento\InventoryApi\Api\Data\StockInterface[]
     * @throws \Magento\Framework\Exception\InputException
     * @throws \Magento\Framework\Exception\LocalizedException
     */
    public function execute($sourceId);
}

Pull Request for StockItem Index

Work in Progress. Current code responsible for StockItem idex could be found https://github.com/magento/inventory/pull/49