427: System configuration full and updates export. #15

owlsmage · 2020-11-18T19:59:07Z

Module to observe and export configuration changes to message queue.

Description (*)

Allows full export manually or automatically (recurring upgrade data script)
Observe changed values in system configuration to export only updates
Exports values for affected stores (only store view level)

Related Pull Requests

https://github.com/magento-commerce/magento2-infrastructure/pull/1395v

Fixed Issues (if relevant)

Fixes Configuration propagation catalog-storefront#427

Code Review Checklist (*)

Story AC is completed
proposed changes correspond to Magento Technical Vision
new or changed code is covered with web-api/integration tests (if applicable)
no backward incompatible changes

app/code/Magento/ConfigurationDataExporter/Console/Command/FullSyncCommand.php

app/code/Magento/ConfigurationDataExporter/Model/ChangedConfigMessageBuilder.php

app/code/Magento/ConfigurationDataExporter/Observer/ConfigChange.php

app/code/Magento/ConfigurationDataExporter/Plugin/ConfigUpdateExport.php

app/code/Magento/ConfigurationDataExporter/Setup/RecurringData.php

mslabko · 2020-11-18T22:53:13Z

Looks good to me!

Regarding AC
"collected configuration paths should be described in di.xml"
I think we may need some Filter class, which will accept only the system path, described in di.xml. This class should filter paths during "observer" and "full sync" part.
We have to filter it to prevent potential security issues and simplify security audit in the future

Ideally need to have possibility to accept paths from env variables (can be done be adding "init_paramter" const)
<argument name="exportConfigPaths" xsi:type="init_parameter">some\corresponding\path::CONST_NAME</argument>) or create new system configuration which can be dynamically configured per each instance

mslabko · 2020-11-19T23:32:42Z

app/code/Magento/ConfigurationDataExporter/Model/ChangedConfigMessageBuilder.php

+        $whitelistedPaths = $this->whitelistProviderPool->getWhitelist();
+
+        foreach ($configData as $item) {
+            if (in_array($item['path'], $whitelistedPaths)) {


can we add the ability to add configuration with partial match, like "path/"?
E.g. to provide all paths for layered navigation from app/code/Magento/LayeredNavigation/etc/config.xml we can use "catalog/layered_navigation/" path

@mslabko partial match added

mslabko · 2020-11-19T23:34:33Z

app/code/Magento/ConfigurationDataExporter/Model/ConfigExportCallback.php

+     */
+    private function publishMessage(array $configData, string $evenType): void
+    {
+        $message = $this->messageBuilder->build($evenType, $configData);


need to add check if message present, since we can receive empty results after filtration by whitelist

@mslabko added check to do not publish empty message to queue

mslabko · 2020-11-19T23:37:39Z

app/code/Magento/ConfigurationDataExporter/Console/Command/FullSyncCommand.php

+ */
+class FullSyncCommand extends \Symfony\Component\Console\Command\Command
+{
+    const COMMAND_NAME = 'config-export:sync:full';


lets rename command to "commerce-data-export::config:export"

mslabko · 2020-11-19T23:38:35Z

app/code/Magento/ConfigurationDataExporter/Model/Whitelist/EnvironmentProvider.php

+    {
+        if (empty($this->whitelist)) {
+            try {
+                $this->whitelist = $this->deploymentConfig->get(self::WHITELIST_CONFIG_KEY, []);


Could you please add CLI command which will add configuration to env.php ? `bin/magento commerce-data-export::config:add-paths-to-whitelist"

@mslabko command added.

Avoid publishing message to queue if message is empty after filtration by whitelist.

… receiving of incorrect message

owlsmage · 2020-11-24T08:07:38Z

@mslabko i added clean up queue before test run.

…on' into story-427-configuation-propagation

RuslanKostiv1 · 2020-12-07T15:31:41Z

@owlsmage please add https://github.com/magento-commerce/magento2-infrastructure/pull/1395 in the list of Related Pull Requests

…ion-propagation

RuslanKostiv1 · 2020-12-15T04:19:17Z

@magento import pr to magento-commerce/commerce-data-export

magento-engcom-team · 2020-12-15T04:20:13Z

@RuslanKostiv1 the pull request successfully imported.

MDEE-57: Handle is_deleted updates. Unassign by chunks

427: System configuration full and updates export.

65d8ae6

owlsmage mentioned this pull request Nov 18, 2020

Configuration propagation magento/catalog-storefront#427

Closed

RuslanKostiv1 reviewed Nov 18, 2020

View reviewed changes

owlsmage added 2 commits November 19, 2020 14:48

427: code style refactor and change data dto implementation

07ebaa4

427: Add path whitelist

d917a7c

owlsmage requested a review from RuslanKostiv1 November 19, 2020 19:51

mslabko reviewed Nov 19, 2020

View reviewed changes

427: Add cli command to add paths to whitelist in env.php.

11d8201

Avoid publishing message to queue if message is empty after filtration by whitelist.

owlsmage requested a review from mslabko November 22, 2020 16:51

owlsmage added 2 commits November 23, 2020 22:18

427: Add configuration data export integration test

5adeedd

427: Purge system configuration export queue before test run to avoid…

3a96e08

… receiving of incorrect message

owlsmage mentioned this pull request Nov 29, 2020

427: configuration updates message broker magento/storefront-message-broker#4

Merged

6 tasks

mslabko mentioned this pull request Dec 2, 2020

PR to run tests for 427_confiuration_import_message_broker magento/catalog-storefront#456

Closed

7 tasks

mslabko approved these changes Dec 2, 2020

View reviewed changes

lykhachov and others added 10 commits December 4, 2020 10:17

static tests

137f3dd

static tests

0381b1a

static tests

ee91b46

added additional check

09a7d4a

Merge branch 'main' into story-427-configuation-propagation

1df7b75

static tests

c81289d

Merge remote-tracking branch 'origin/story-427-configuation-propagati…

64a95dd

…on' into story-427-configuation-propagation

static tests

e5259d6

added blacklist/undeclared_dependency

7e520a0

Integration test resolve

8cf25b4

RuslanKostiv1 added 3 commits December 7, 2020 13:56

Merge remote-tracking branch 'commerce/main' into story-427-configuat…

b9f27e2

…ion-propagation

Merge remote-tracking branch 'commerce/main' into story-427-configuat…

61446b6

…ion-propagation

Static tests fixes

f592c4a

mmansoor-magento merged commit 933b513 into magento:main Dec 16, 2020

magento-devops-reposync-svc pushed a commit that referenced this pull request Nov 4, 2021

Merge pull request #15 from magento-troll/unassign-by-chunk

cbab2e1

MDEE-57: Handle is_deleted updates. Unassign by chunks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

427: System configuration full and updates export. #15

427: System configuration full and updates export. #15

owlsmage commented Nov 18, 2020 •

edited

Loading

mslabko commented Nov 18, 2020 •

edited

Loading

mslabko Nov 19, 2020

owlsmage Nov 22, 2020

mslabko Nov 19, 2020

owlsmage Nov 22, 2020

mslabko Nov 19, 2020

mslabko Nov 19, 2020

owlsmage Nov 22, 2020

owlsmage commented Nov 24, 2020

RuslanKostiv1 commented Dec 7, 2020

RuslanKostiv1 commented Dec 15, 2020

magento-engcom-team commented Dec 15, 2020

427: System configuration full and updates export. #15

427: System configuration full and updates export. #15

Conversation

owlsmage commented Nov 18, 2020 • edited Loading

Description (*)

Related Pull Requests

Fixed Issues (if relevant)

Code Review Checklist (*)

mslabko commented Nov 18, 2020 • edited Loading

mslabko Nov 19, 2020

Choose a reason for hiding this comment

owlsmage Nov 22, 2020

Choose a reason for hiding this comment

mslabko Nov 19, 2020

Choose a reason for hiding this comment

owlsmage Nov 22, 2020

Choose a reason for hiding this comment

mslabko Nov 19, 2020

Choose a reason for hiding this comment

mslabko Nov 19, 2020

Choose a reason for hiding this comment

owlsmage Nov 22, 2020

Choose a reason for hiding this comment

owlsmage commented Nov 24, 2020

RuslanKostiv1 commented Dec 7, 2020

RuslanKostiv1 commented Dec 15, 2020

magento-engcom-team commented Dec 15, 2020

owlsmage commented Nov 18, 2020 •

edited

Loading

mslabko commented Nov 18, 2020 •

edited

Loading