diff --git a/design-documents/asynchronous-import/AsyncImportUML.png b/design-documents/asynchronous-import/AsyncImportUML.png new file mode 100644 index 000000000..eb2e0e9f0 Binary files /dev/null and b/design-documents/asynchronous-import/AsyncImportUML.png differ diff --git a/design-documents/asynchronous-import/asycnhronous-import-service.md b/design-documents/asynchronous-import/asycnhronous-import-service.md deleted file mode 100644 index ca638f019..000000000 --- a/design-documents/asynchronous-import/asycnhronous-import-service.md +++ /dev/null @@ -1,16 +0,0 @@ - -# Asynchronous Import as separate service -## Context - -Main idea of extension is to replace current Import module with new implementation that will allow to users import objects in Magento by using Asynchronous approach. - -Basic idea: -- User upload *.csv (or any other) file format -- Extension receive file, validate it and return to user File UUID -- By using this File UUID user can start import with customer parameters (if applicable) -- Module will parse file, split it on single messages and sends to Asynchronous API. -- Later on user can request back status of import and resubmit objects which were failed during the import - -Proposed to split on several phases -- [Phase 1](base-extension.md) -- [Phase 2](import-ui.md) \ No newline at end of file diff --git a/design-documents/asynchronous-import/asycnhronous-import.md b/design-documents/asynchronous-import/asycnhronous-import.md new file mode 100644 index 000000000..f3eb2800e --- /dev/null +++ b/design-documents/asynchronous-import/asycnhronous-import.md @@ -0,0 +1,48 @@ +# Asynchronous Import as separate service + +Main idea of extension is to replace current Import module with new implementation that will allow to users import objects in Magento by using Asynchronous approach. + +## Workflow + +- Upload source data to instance storage (optional step); +- Retrieve source data from Source (partial reading); +- Parse source data; +- Apply data converting rule; +- Exchange data with Magento instance via Bulk API Service contracts / Bulk API REST (partial processing); +- Later an user can request balk status of import and resubmit objects which were failed during the import; + +## REST API + +- [REST API](rest-api.md) + +## Modularity (API / Extension points) + +- [Start Import based on CSV data (main entry point)](modularity/import-csv.md) +- [Source data retrieving](modularity/source-data-retrieving.md) +- [Data converting before import](modularity/data-converting.md) +- [Data exchanging with Magento instance](modularity/data-exchanging.md) +- [Import advanced pricing](modularity/advanced-pricing.md) + +## MVP + +[MVP Board](https://app.zenhub.com/workspaces/async-import-5b5f349bd6768d6255917727/reports/burndown?milestoneId=4625823) + +- Sources data retrieving: *HTTP, HTTPS, base64*; +- CSV reader; +- Data Converting Rules; +- Import Data Exchanging; +- Import Configuration; +- Product import; +- Stock import; +- Advanced pricing import; +- Documentation; + +## UML + +![AsyncImportUML@2x](AsyncImportUML.png) + +## TODO +- Design of Import configuration; +- Design of Get Import status; +- Design of Import processing - Sync / Async +- Design of Restart failed operations diff --git a/design-documents/asynchronous-import/base-extension.md b/design-documents/asynchronous-import/base-extension.md deleted file mode 100644 index e3e166973..000000000 --- a/design-documents/asynchronous-import/base-extension.md +++ /dev/null @@ -1,374 +0,0 @@ -# Phase 1 - -With phase 1 we are planning to develop next functionality: -- Endpoint to receive file -- Endpoint to start processing -- Endpoint to receive status - -## File Upload Endpoint -### Source Upload endpoint - -So import will start from uploading import Source file. Currently we will support "csv" files format - -POST `/V1/import/source/csv` - -This request can accept files from different sources: -- Local file path -- Direct link to the file -- Base64 encoded file content - -#### Local file path - -Path is relative from Magento Root folder - -``` -{ - "source": { - "import_data": "var/catalog_product.csv", - "import_type": "local_path", - "uuid": "UUID", - "format": { - "csv_separator": "string", - "csv_enclosure": "string", - "csv_delimiter": "string", - "multiple_value_separator": "string" - } - } -} -``` - -#### Direct link to the file - -``` -{ - "source": { - "import_data": "http://some.domain/file.csv", - "import_type": "external", - "uuid": "UUID", - "format": { - "csv_separator": "string", - "csv_enclosure": "string", - "csv_delimiter": "string", - "multiple_value_separator": "string" - } - } -} -``` - -#### Base64 encoded file content - -``` -{ - "source": { - "import_data": "c2t1LHN0b3JlX3ZpZXdfY29kZSxhdHRyaWJ1dGVfc2V0X2NvZGUscHJvZHVjdF90eXBlLGNhdGVnb3JpZXMscHJvZHVjdF93ZWJzaXRlcyxuYW1lLGRlc2NyaXB0aW9uLHNob3J0X2Rlc2NyaXB0aW9uLHdlaWdodCxwcm9kdWN0X29ubGluZSx0YXhfY2xhc3NfbmFtZSx2aXNpYmlsaXR5LHBya......", - "import_type": "base64_encoded_data", - "uuid": "UUID", - "format": { - "csv_separator": "string", - "csv_enclosure": "string", - "csv_delimiter": "string", - "multiple_value_separator": "string" - } - } -} -``` - -Import of big file also can be divided in several parts. -For this case we have separate endpoint - -POST `/V1/import/source/csv/partial/` - -Input request will looks like: - -``` -{ - "source": { - "import_data": "c2t1LHN0b3JlX3ZpZXdfY29kZSxhdHRyaWJ1dGVfc2V0X2NvZGUscHJvZHVjdF90eXBlLGNhdGVnb3JpZXMscHJvZHVjdF93ZWJzaXRlcyxuYW1lLGRlc2NyaXB0aW9uLHNob3J0X2Rlc2NyaXB0aW9uLHdlaWdodCxwcm9kdWN0X29ubGluZSx0YXhfY2xhc3NfbmFtZSx2aXNpYmlsaXR5LHBya...", - "data_hash" : "sha256 encoded data of the full 'import_data' value" - "pieces_count": "5" - "piece_number": "1", - "import_type": "base64_encoded_data", - "uuid": "UUID", - "format": { - "csv_separator": "string", - "csv_enclosure": "string", - "csv_delimiter": "string", - "multiple_value_separator": "string" - } - } -} -``` -where *import_data* is a 1/N part of the whole content, and *data_hash* contains sha256 hash of full import_data body. - -`pieces_count` - its an amount of pieces that will be transferred for 1 file. We need it to be sure that import is completed and then we could detect if it was successfully finished or failed - -`piece_number` - its a number that detects which part of file currently transferred. This is required to have to support Asynchronous File import when we dont need to send parts in correct sequence - -Those parts could be send asynchronously. They will be merged together after all data are transferred. - -### Return values - -As return user will receive: - -| Key | Value | -| --- | --- | -| uuid | Imported File ID | -| status | Status of this file. Possible values: completed, uploaded, failed | -| error | Error message if exists | - -Example: - -``` -{ - "uuid": null, - "status": null, - "error": null, - "source": { - // Source object is coming here - } -} -``` - -### Update Imported Source Format - -Its possible also to Update Format - -PUT `/V1/import/source/csv/:uuid` - -``` -{ - "source": { - "uuid": "uuid", - "format": { - "csv_separator": "string", - "csv_enclosure": "string", - "csv_delimiter": "string", - "multiple_value_separator": "string" - } - } -} -``` - -### Delete Imported Source Format - -DELETE `/V1/import/source/:uuid` - -### Get List of sources - -GET `/V1/import/sources/?searchCriteria` - -Will return list of Source that was uploaded before. - -``` -{ - "sources": [ - { - "uuid": "uuid", - "format": { - "csv_separator": "string", - "csv_enclosure": "string", - "csv_delimiter": "string", - "multiple_value_separator": "string" - } - }, - { - "uuid": "uuid", - "format": { - "csv_separator": "string", - "csv_enclosure": "string", - "csv_delimiter": "string", - "multiple_value_separator": "string" - } - } - ..... - ], - "search_criteria": { - "filter_groups": [ - { - "filters": [ - { - "field": "string", - "value": "string", - "condition_type": "string" - } - ] - } - ], - "sort_orders": [ - { - "field": "string", - "direction": "string" - } - ], - "page_size": 0, - "current_page": 0 - }, - "total_count": 0 -} -``` - -## Start File Import Endpoint -### Main endpoint - -Current Endpoint starts import process based on FileID. Where Module will read file, split it into message and send to Async API - -POST `/V1/import/start/{uuid}` - -`type` - its an import type like: catalog_product, catalog_category, customer, order ... etc... - -Start File Import - -``` -{ - "importConfig": { - "import_type": "catalog_product", - "import_strategy": "add_update, delete, replace", - "validation_strategy": "string", - "allowed_error_count": 0, - "import_image_archive": "string", - "import_images_file_dir": "string", - "mapping": [ - { - "name": "string", - "source_path": "string", - "target_path": "string", - "target_value": "mixed", - "processing_rules": [ - { - "sort": int, - "function": "string", - "args": "array" - } - ] - } - ] - } -} -``` - -| Key | Value | -| --- | --- | -| uuid | UUID that was returned by source upload call | -| behaviour | Import behaviour (add_update, delete, update, add, replace) | -| import_image_archive | Relative path to product images archive file | -| import_images_file_dir | Relative path to product images files | -| validation_strategy | Moved from main standard Import, not sure if we will use if | -| allowed_error_count | How many errors allowed to be during the import | -| mapping | Data format mapping | - - -#### Return - -``` -{ - "uuid": string - "status": "proccessing", - "error": "string" -} -``` - -### Get import status - -Receive information about imported file -GET `/V1/import/:uuid` - -#### Return - -Will be returned list of objects that we tried to import - -``` -{ - "status": "string", - "error": "string", - "uuid": 0, - "entity_type": "catalog_product, customers ....", - "user_id": "User ID who created this request", - "user_type": "User Type who created this request", - "items": [ - { - "uuid": 0, - "status": "", - "serialized_data": "", - "result_serialized_data": "", - "error_code":"", - "result_message":"" - }] -} -``` - -| Key | Value | -| --- | --- | -| uuid | Imported File ID | -| status | Status of this file. Possible values: completed, not_completed, error | -| error | Error message if exists | -| entity_type | Import type: eg. customers, products, etc ... | -| items | List of items that were imported. As an array | - -### Get single import operation status - -Receive information about imported file -GET `/V1/import/operation/:uuid` - -#### Return - -Will be returned specific object that we tried to import - -``` -{ - "status": "string", - "error": "string", - "uuid": 0, - "entity_type": "catalog_product, customers ....", - "user_id": "User ID who created this request", - "user_type": "User Type who created this request", - "items": [ - { - "uuid": 0, - "status": "", - "serialized_data": "", - "result_serialized_data": "", - "error_code":"", - "result_message":"" - }] -} -``` - -| Key | Value | -| --- | --- | -| uuid | Imported File ID | -| status | Status of this file. Possible values: completed, not_completed, error | -| error | Error message if exists | -| entity_type | Import type: eg. customers, products, etc ... | -| items | Item that were requested. For do not change interfaces it will be still as an array, but always one item | - -##### And Item object will contain - -This values are based on magento "magento_operation" table - -| Key | Value | -| --- | --- | -| uuid | Entity UUID - defined by customer OR auto-generated by system | -| status | Import status. "pending, failed, processing, completed" | -| serialized_data | Data that was send to Magento, via Async endpoint | -| result_serialized_data | Data that Magento returned for this object after import | -| error_code | Error code | -| result_message | Result message of operation execution | - - -## Repeatable call endpoint - -Main idea, is in case some operation import failure, that user could align input data and repeat import for this particular item - -PUT `/V1/import/operation-id/{uuid}` - -``` -{ - "serialized_data":"" -} -``` - -### Return - -``` -[] -``` diff --git a/design-documents/asynchronous-import/import-ui.md b/design-documents/asynchronous-import/import-ui.md deleted file mode 100644 index a31fab418..000000000 --- a/design-documents/asynchronous-import/import-ui.md +++ /dev/null @@ -1,2 +0,0 @@ -# Phase 2 -In a second phase planned to build UI for Asynchronous Import. It will be separate Magento module that will use API to communicate with Asynchronous import module. diff --git a/design-documents/asynchronous-import/modularity/advanced-pricing.md b/design-documents/asynchronous-import/modularity/advanced-pricing.md new file mode 100644 index 000000000..1611bec02 --- /dev/null +++ b/design-documents/asynchronous-import/modularity/advanced-pricing.md @@ -0,0 +1,4 @@ +# Import advanced pricing + +## Modules +- `AsynchronousImportAdvancedPricing` diff --git a/design-documents/asynchronous-import/modularity/data-converting.md b/design-documents/asynchronous-import/modularity/data-converting.md new file mode 100644 index 000000000..b00880244 --- /dev/null +++ b/design-documents/asynchronous-import/modularity/data-converting.md @@ -0,0 +1,154 @@ +# Data converting before import + +## Modules + +- `AsynchronousImportDataConvertingApi` +- `AsynchronousImportDataConverting` + +## Implementation + +### API + +```php +namespace Magento\AsynchronousImportDataConvertingApi\Api; + +use Magento\AsynchronousImportDataConvertingApi\Api\Data\ConvertingRuleInterface; +use Magento\Framework\Validation\ValidationException; + +/** + * Apply converting rules to import data operation. Uses differect strategies for rules applying + * Responsible for data changing before import + * + * @api + */ +interface ApplyConvertingRulesInterface +{ + /** + * Apply converting rules to import data operation. Uses differect strategies for rules applying + * + * @param array $importData + * @param ConvertingRuleInterface[] $convertingRules + * @return array + * @throws ValidationException + * @throws ApplyConvertingRulesException + */ + public function execute( + array $importData, + array $convertingRules + ): array; +} +``` + +```php +namespace Magento\AsynchronousImportDataConvertingApi\Api\Data; + +/** + * Describes how to change data before import + * + * @api + */ +interface ConvertingRuleInterface +{ + /** + * Get rule identifier + * + * @return string + */ + public function getIdentifier(): string; + + /** + * Get rule parameters + * + * @return string[]|null Null value is needed fro SOAP parser + */ + public function getParameters(): array; + + /** + * Get sort + * + * @return int|null + */ + public function getSort(): ?int; + + /** + * Get apply to + * + * @return string[]|null Null value is needed fro SOAP parser + */ + public function getApplyTo(): array; +} +``` + +Converting rules are transforming incoming data in a way, that Magento API can correctly understand them. + +#### Example: +Magento default Advanced Pricing import *.csv file contains Magento Website assignment as a string value: "All Websites [ALL]". + +Magento API cannot recognise correct website base on this string representation, so this means we have to create converting rule that will define correct transformation from provided text to Magento Website ID. + + +### Extension points + +```php +namespace Magento\AsynchronousImportDataConvertingApi\Model; + +use Magento\AsynchronousImportDataConvertingApi\Api\Data\ConvertingRuleInterface; + +/** + * Extension point for adding converting rule applying algorithms + * Represents concrete strategy + * + * @api + */ +interface ApplyConvertingRuleStrategyInterface +{ + /** + * Converting rule applying strategy + * + * @param array $importData + * @param ConvertingRuleInterface $convertingRule + * @return array + */ + public function execute( + array $importData, + ConvertingRuleInterface $convertingRule + ): array; +} +``` + +```php +namespace Magento\AsynchronousImportDataConvertingApi\Model; + +/** + * Extension point for adding converting rule validators via DI configuration + * + * @api + */ +class ConvertingRuleValidatorChain implements ConvertingRuleValidatorInterface +{ + ... +} +``` + +```php +namespace Magento\AsynchronousImportDataConvertingApi\Model; + +use Magento\AsynchronousImportDataConvertingApi\Api\Data\ConvertingRuleInterface; +use Magento\Framework\Validation\ValidationResult; + +/** + * Extension point for adding converting rule validators + * + * @api + */ +interface ConvertingRuleValidatorInterface +{ + /** + * Validate converting rule + * + * @param ConvertingRuleInterface $convertingRule + * @return ValidationResult + */ + public function validate(ConvertingRuleInterface $convertingRule): ValidationResult; +} +``` diff --git a/design-documents/asynchronous-import/modularity/data-exchanging.md b/design-documents/asynchronous-import/modularity/data-exchanging.md new file mode 100644 index 000000000..44307aea4 --- /dev/null +++ b/design-documents/asynchronous-import/modularity/data-exchanging.md @@ -0,0 +1,211 @@ +# Data exchanging with Magento instance + +## Modules + +- `AsynchronousImportDataExchangingApi` +- `AsynchronousImportDataExchanging` + +## Implementation + +### API + +```php +namespace Magento\AsynchronousImportDataExchangingApi\Api; + +use Magento\AsynchronousImportDataExchangingApi\Api\Data\ImportInterface; +use Magento\Framework\Validation\ValidationException; + +/** + * Operation for exchanging import data with destination instance. Uses differect strategies for data import + * + * @api + */ +interface ExchangeImportDataInterface +{ + /** + * Operation for exchanging import data with destination instance + * + * @param ImportInterface $import + * @param array $importData + * @return void + * @throws ValidationException + * @throws ImportDataExchangeException + */ + public function execute(ImportInterface $import, array $importData): void; +} +``` + +```php +namespace Magento\AsynchronousImportDataExchangingApi\Api\Data; + +use Magento\Framework\Api\ExtensibleDataInterface; + +/** + * Describes how to import data + * + * @api + */ +interface ImportInterface extends ExtensibleDataInterface +{ + /** + * Get import uuid + * + * @return string|null + */ + public function getUuid(): ?string; + + /** + * Get import type + * + * @return string + */ + public function getImportType(): string; + + /** + * Get import behaviour + * + * @return string + */ + public function getImportBehaviour(): string; + + /** + * Get existing extension attributes object + * + * Used fully qualified namespaces in annotations for proper work of extension interface/class code generation + * + * @return \Magento\AsynchronousImportDataExchangingApi\Api\Data\ImportExtensionInterface|null + */ + public function getExtensionAttributes(): ?ImportExtensionInterface; +} +``` + +```php +namespace Magento\AsynchronousImportDataExchangingApi\Api; + +use Magento\AsynchronousImportDataExchangingApi\Api\Data\ImportStatusInterface; +use Magento\Framework\Exception\NotFoundException; + +/** + * Get import status operation + * + * @api + */ +interface GetImportStatusInterface +{ + /** + * Get import status operation + * + * @param string $uuid + * @return ImportStatusInterface + * @throws NotFoundException + */ + public function execute(string $uuid): ImportStatusInterface; +} +``` + +```php +namespace Magento\AsynchronousImportDataExchangingApi\Api\Data; + +/** + * Represents import status + * + * @api + */ +interface ImportStatusInterface +{ + public const STATUS_RUNNING = 'running'; + public const STATUS_COMPLETED = 'completed'; + public const STATUS_FAIL = 'fail'; + + /** + * Get status + * + * @return string One of const STATUS_* + */ + public function getStatus(): string; + + /** + * Get Errors + * + * @return array + */ + public function getErrors(): array; + + /** + * Get created at + * + * @return string|null + */ + public function getCreatedAt(): ?string; + + /** + * Get finished at + * + * @return string|null + */ + public function getFinishedAt(): ?string; +} +``` + +### Extension points + +```php +namespace Magento\AsynchronousImportDataExchangingApi\Model; + +use Magento\AsynchronousImportDataExchangingApi\Api\Data\ImportInterface; + +/** + * Extension point for adding data import algorithms + * Represents concrete strategy + * + * @api + */ +interface ExchangeDataStrategyInterface +{ + /** + * Data import strategy + * + * @param ImportInterface $import + * @param array $importData + * @return void + */ + public function execute(ImportInterface $import, array $importData): void; +} +``` + +```php +namespace Magento\AsynchronousImportDataExchangingApi\Model; + +/** + * Extension point for adding import request validators via DI configuration + * + * @api + */ +class ImportValidatorChain implements ImportValidatorInterface +{ + ... +} +``` + +```php +namespace Magento\AsynchronousImportDataExchangingApi\Model; + +use Magento\AsynchronousImportDataExchangingApi\Api\Data\ImportInterface; +use Magento\Framework\Validation\ValidationResult; + +/** + * Extension point for adding import request validators + * + * @api + */ +interface ImportValidatorInterface +{ + /** + * Import validation. Extension point for base validation + * + * @param ImportInterface $import + * @return ValidationResult + */ + public function validate(ImportInterface $import): ValidationResult; +} +``` diff --git a/design-documents/asynchronous-import/modularity/import-csv.md b/design-documents/asynchronous-import/modularity/import-csv.md new file mode 100644 index 000000000..4841799c7 --- /dev/null +++ b/design-documents/asynchronous-import/modularity/import-csv.md @@ -0,0 +1,127 @@ +# Start Import based on CSV data (main entry point) + +## Modules + +- `AsynchronousImportCsvApi` +- `AsynchronousImportCsv` + +## Implementation + +### API + +```php +namespace Magento\AsynchronousImportCsvApi\Api; + +use Magento\AsynchronousImportDataConvertingApi\Api\ApplyConvertingRulesException; +use Magento\AsynchronousImportCsvApi\Api\Data\CsvFormatInterface; +use Magento\AsynchronousImportDataConvertingApi\Api\Data\ConvertingRuleInterface; +use Magento\AsynchronousImportDataExchangingApi\Api\Data\ImportInterface; +use Magento\AsynchronousImportDataExchangingApi\Api\ImportDataExchangeException; +use Magento\AsynchronousImportSourceDataRetrievingApi\Api\Data\SourceInterface; +use Magento\AsynchronousImportSourceDataRetrievingApi\Api\SourceDataRetrievingException; +use Magento\Framework\Validation\ValidationException; + +/** + * Start import operation + * + * @api + */ +interface StartImportInterface +{ + /** + * Start import operation + * + * @param SourceInterface $source Describes how to retrieve data from data source + * @param ImportInterface $import Describes how to import data + * @param CsvFormatInterface|null $format Describes how to parse data + * @param ConvertingRuleInterface[] $convertingRules Describes how to change data before import + * @return string + * @throws ValidationException + * @throws SourceDataRetrievingException + * @throws ApplyConvertingRulesException + * @throws ImportDataExchangeException + */ + public function execute( + SourceInterface $source, + ImportInterface $import, + CsvFormatInterface $format = null, + array $convertingRules = [] + ): string; +} +``` + +```php +namespace Magento\AsynchronousImportCsvApi\Api\Data; + +use Magento\Framework\Api\ExtensibleDataInterface; + +/** + * Describes how to parse data + * + * @api + */ +interface CsvFormatInterface extends ExtensibleDataInterface +{ + /** + * Get CSV Escape + * + * @return string|null + */ + public function getEscape(): ?string; + + /** + * Get CSV Enclosure + * + * @return string|null + */ + public function getEnclosure(): ?string; + + /** + * Get CSV Delimiter + * + * @return string|null + */ + public function getDelimiter(): ?string; + + /** + * Get Multiple Value Separator + * + * @return string|null + */ + public function getMultipleValueSeparator(): ?string; + + /** + * Get existing extension attributes object + * + * Used fully qualified namespaces in annotations for proper work of extension interface/class code generation + * + * @return \Magento\AsynchronousImportCsvApi\Api\Data\CsvFormatExtensionInterface|null + */ + public function getExtensionAttributes(): ?CsvFormatExtensionInterface; +} +``` + +### Extension points + +```php +namespace Magento\AsynchronousImportCsvApi\Model; + +use Magento\AsynchronousImportCsvApi\Api\Data\CsvFormatInterface; + +/** + * Extension point for data parsing (based on passed CSV format) + * + * @api + */ +interface DataParserInterface +{ + /** + * Extension point for data parsing (based on passed CSV format) + * + * @param array $data + * @param CsvFormatInterface|null $csvFormat + * @return array + */ + public function execute(array $data, CsvFormatInterface $csvFormat = null): array; +} +``` diff --git a/design-documents/asynchronous-import/modularity/source-data-retrieving.md b/design-documents/asynchronous-import/modularity/source-data-retrieving.md new file mode 100644 index 000000000..791dae9f3 --- /dev/null +++ b/design-documents/asynchronous-import/modularity/source-data-retrieving.md @@ -0,0 +1,153 @@ +# Source data retrieving + +## Info + +- Responsible for retrieving source data; +- Provide ability to easily add new adapters for different data sources; +- Provide ability to use streaming in different adapters (do not load whole data into memory); + +## Modules + +- `AsynchronousImportSourceDataRetrievingApi` +- `AsynchronousImportSourceDataRetrieving` + +## Implementation + +### API + +```php +namespace Magento\AsynchronousImportSourceDataRetrievingApi\Api; + +use Magento\AsynchronousImportSourceDataRetrievingApi\Api\Data\SourceInterface; +use Magento\AsynchronousImportSourceDataRetrievingApi\Api\Data\SourceDataInterface; +use Magento\Framework\Validation\ValidationException; + +/** + * Retrieve source data operation. Uses differect strategies for source data retrieving + * + * @api + */ +interface RetrieveSourceDataInterface +{ + /** + * Retrieve source data operation. Uses differect strategies for source data retrieving + * + * @param SourceInterface $source + * @return SourceDataInterface + * @throws ValidationException + * @throws SourceDataRetrievingException + */ + public function execute(SourceInterface $source): SourceDataInterface; +} +``` + +```php +namespace Magento\AsynchronousImportSourceDataRetrievingApi\Api\Data; + +/** + * Describes how to retrieve data from data source + * + * @api + */ +interface SourceInterface +{ + /** + * Get source type + * + * @return string + */ + public function getSourceType(): string; + + /** + * Get source definition + * + * @return string + */ + public function getSourceDefinition(): string; + + /** + * Get source data format + * + * @return string + */ + public function getSourceDataFormat(): string; +} +``` + +```php +namespace Magento\AsynchronousImportSourceDataRetrievingApi\Api\Data; + +/** + * Represents retrieved source data (result of retrieving operation) + * + * @api + */ +interface SourceDataInterface extends \IteratorAggregate +{ + public const ITERATOR = 'iterator'; +} +``` + +### Extension points + +```php +namespace Magento\AsynchronousImportSourceDataRetrievingApi\Model; + +use Magento\AsynchronousImportSourceDataRetrievingApi\Api\Data\SourceInterface; +use Magento\AsynchronousImportSourceDataRetrievingApi\Api\SourceDataRetrievingException; + +/** + * Extension point for adding source data retrieving algorithms + * Represents concrete strategy + * + * @api + */ +interface RetrieveSourceDataStrategyInterface +{ + /** + * Source data retrieving strategy + * + * @param SourceInterface $source + * @return \Traversable + * @throws SourceDataRetrievingException + */ + public function execute(SourceInterface $source): \Traversable; +} +``` + +```php +namespace Magento\AsynchronousImportSourceDataRetrievingApi\Model; + +/** + * Extension point for adding source validators via DI configuration + * + * @api + */ +class SourceValidatorChain implements SourceValidatorInterface +{ + ... +} +``` + +```php +namespace Magento\AsynchronousImportSourceDataRetrievingApi\Model; + +use Magento\AsynchronousImportSourceDataRetrievingApi\Api\Data\SourceInterface; +use Magento\Framework\Validation\ValidationResult; + +/** + * Extension point for adding source validators + * + * @api + */ +interface SourceValidatorInterface +{ + /** + * Validate source + * + * @param SourceInterface $source + * @return ValidationResult + */ + public function validate(SourceInterface $source): ValidationResult; +} +``` diff --git a/design-documents/asynchronous-import/rest-api.md.md b/design-documents/asynchronous-import/rest-api.md.md new file mode 100644 index 000000000..2ee536fab --- /dev/null +++ b/design-documents/asynchronous-import/rest-api.md.md @@ -0,0 +1,85 @@ +# REST API + +## 1. Source Data Upload Endpoint (optional step) + +The entry point for start uploading Source data to instance storage (Currently supported only local instance) + +**Request:** + +`POST /V1/import/source` + +``` +{ + "source": { + "sourceType": "http", + "sourceDefinition": "http://some.domain/file.csv", + "sourceDataFormat": "CSV" + } +} +``` + +**Response:** + +``` +{ + "filename": "var/import/filename.csv" +} +``` + + +## 2. Start Import Endpoint + +Single entry point for **partial** reading and import of data. + +`POST /V1/import/csv` + +``` +{ + "source": { + "sourceType": "http" + "sourceDefinition": "http://some.domain/file.csv" + "sourceDataFormat": "CSV" + } + "format': { + "escape": "|" + "enclosure": "|" + "delimiter": "|" + "multipleValueSeparator": "|" + "extensionAttributes": [] + } + "convertingRules": [ + { + "identifier": "string_replace" + "applyTo": ["sku", "name"] + "parameters": { + "search": "_" + "replace": "-" + } + } + ] + "import": { + "importType": "advanced_pricing" + "importBehaviour": "add" + "extensionAttributes": [] + } +} +``` + +Where is: +- `source` is **required**. Describes how to retrieve data from Source + - `sourceType` is **required**. Example: `HTTP|HTTPS|local_file|base_64`. Could contains adiditonal information as `encoded string` if needed. + - `sourceDefinition` is **required**. Depends on `sourceType`. Example: `http://some.domain/file.csv|var/import/filename.csv` + - `sourceDataFormat` is **optional**. Needed for preliminary validation before data retrieving. +- `format` is **optional**. Describes how to parse data. +- `convertingRules[]` is **optional**. Describes how to change data before Import process. Additional data could be placed in `parameters`. +- `import` is **required**. Describes how to import data. + - `importType` is **required**. Example: `stock|advanced_pricing|product`. + - `importBehaviour` is **required**. Example: `add|delete` + +**Result:** + +``` +{ + "uuid": "uuid_string" +} +```