Elasticsearch6 implementation. #21458

romainruaud · 2019-02-26T11:15:14Z

Description (*)

Support Of Elasticsearch 6.

As discussed with @maghamed .

What have been done :

Removal of the _all field usage which is not supported anymore.
Writing a new Suggestions data provider since the suggest endpoint of Elasticsearch is deprecated.
Associated Unit Tests
And allow to use ~6.1 version of the PHP client in composer.json

Imho it's enough to deal with the EOL of Elasticsearch 5 and support the 6.

However, the module is quite messy actually (contains many classes and namespaces named after the supported Elasticsearch version, like Elasticsearch5, when Elasticsearch is supposed to be a mixing between generic implementation and legacy 2.x).

The same apply for System configuration fields which are named according to the version of the engine, which is bad imho.

Most probably, it will need a huge refactoring on next stable (2.4) :

Cleanup the mess and re-organize the module structure.
Deprecate the 5.x support
Implement ES 7.x support that will be available soon.

Fixed Issues (if relevant)

Manual testing scenarios (*)

Install version 6 of Elasticsearch
Configure the Engine in the back-office (it uses new configuration fields, like it was done for 5.x support. Imho this is not good, but I implemented the same way).
Run a full reindex and navigate through pages in front, process searches, etc ...

Contribution checklist (*)

Pull request has a meaningful description of its purpose
All commits are accompanied by meaningful commit messages
All new or changed code is covered with unit/integration tests (if applicable)
All automated tests passed successfully (all builds on Travis CI are green)

magento-engcom-team · 2019-02-26T11:15:21Z

Hi @romainruaud. Thank you for your contribution
Here is some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

@magento-engcom-team give me test instance - deploy test instance based on PR changes
@magento-engcom-team give me 2.3-develop instance - deploy vanilla Magento instance

For more details, please, review the Magento Contributor Assistant documentation

magento-cicd2 · 2019-02-26T11:15:23Z

All committers have signed the CLA.

orlangur · 2019-02-26T13:26:09Z

@romainruaud @maghamed current implementation approach does not look as a robust solution to me.

Most probably, it will need a huge refactoring on next stable (2.4) :

It would be pretty hard if not impossible. Much better solution is to implement Elasticsearch 6 support as a new module, possibly disabled by default, old module may be deprecated as a whole when new one is ready.

romainruaud · 2019-02-26T13:55:07Z

@orlangur I agree with you on some points.

It's the whole Elasticsearch "abstraction" and the Search part of Magento framework (if I understood properly, ES will be the only supported engine now that MySQL fulltext search implementation is facing deprecation too ?) that has to be revamped in my opinion :

Search Requests definitions (generic, to be able to query anything)
Search Indices definition (generic, to be able to index anything)
Search Query building with abstraction of all ES query and aggregation DSL types
Properly customizable search relevance
and so on

We can (and we'd enjoy to) help on these topics, since we already built everything listed above in our Open Source implementation of Elasticsearch (which is widely known and used by the community, even documented in the Magento2 devdocs) : https://github.com/Smile-SA/elasticsuite

But this is a huge amount of work to be done and to coordinate.

And for now, Elasticsearch 5 is EOL on 03/11 : https://www.elastic.co/support/eol

So the goal of this PR was to provide support of ES 6 to the community as soon as possible.

Let me know how you wanna process on this PR.

In any cases, I'm not sure there should be a new module "dedicated to Elasticsearch 6", or we will face the same mess on a near future when we will begin to write another module "dedicated to Elasticsearch 7" which is inheriting/extending the other one.

Imho there should be only one module that always handle the latest stable version of Elasticsearch, and that remains compatible with the previous "still maintained" versions with a Deprecation mechanism based on plugins. (So for now, module should fit with ES 6, and handle ES 5 specificity with plugins).
Once a new version of ES is out, we flush all the deprecation plugins, update the default implementation, and build new deprecation plugins. (when ES 7 is out, we provide default implementation of Elasticsearch 7, implement ES 6 specifics as deprecation plugins, and get rid of 5.x completely, because it's EOL). And on and on.

Maybe this part should even be inside Framework directory, or managed on a separate lifecycle.

Product (and more) indexing & querying should be in a separate (most agnostic as possible) modules also.

Let me know,

Best regards

orlangur · 2019-02-26T14:27:40Z

@romainruaud https://github.com/magento/magento2/tree/2.3-develop/app/code/Magento/Tinymce3 is a good example of approach I described.

With all-in-one module it's pretty hard to determine which code parts are responsible for which Elasticsearch version and which versions are actually supported in each particular moment of time.

Elasticsearch6 may be a new standalone module containing everything needed for v6 support. Depending on strategy chosen It may contain plugins for older versions support or such code should be placed in Elasticsearch5 module only.

The same way, when Elasticsearch7 module will be introduced it should provide full support when enabled by itself and may be based on some older code (but not inherit or extend it).

It's enough if only one of Elasticsearch5 / Elasticsearch6 / Elasticsearch7 may be enabled at a time.

romainruaud · 2019-02-26T14:35:18Z

But with such an approach you will end up with several modules actually sharing 95+% of their code, just like a big copy-paste, and will end up having to fix any issue on all of these nearly identical modules.

Why would you treat supported Elasticsearch versions differently than PHP versions (or any other component) ?

Supporting the latest and "old-stable" on a single module (with proper deprecation mechanism) is enough for me, and seems far easier to maintain.

But we are now a bit far from the initial topic.

romainruaud · 2019-02-26T16:27:06Z

Discussed with @maghamed :

I'll rework this PR and provide it as a separated module (which will contain only ES6 related logic).
Later on, we (or more exactly Magento people) will have to decide which support policy to have with ES versions (either support only latest & old-stable, like for PHP or support more versions with as many separated modules as needed).

I let Igor correct me if needed.

Regards

CajuCLC · 2019-02-26T16:40:02Z

@romainruaud I agree with you that it should be only one module that supports what is still being supported.
I understand that having several modules allows end users to use older versions of ES, but that opens those to security issues in the future. And if end users are not updating ES, most likely they are not updating Magento either.

miguelbalparda · 2019-02-26T16:44:41Z

@CajuCLC that's not always true since software versions are maintained and upgraded by hosting companies and Magento updates are up to the end users.

CajuCLC · 2019-02-26T16:54:29Z

@miguelbalparda That goes back to what's the right way to maintain your app. IMHO the best way is to maintain everything update so there is minimal security risk.
Does Magento want to make everyone happy? Then why not support PHP 5.3 for all versions then?
You can't make everyone happy, but you can make sure your system is up to date in terms of security. Most of hosting companies will also support older versions of software for some time even if it's already EOL just so it gives their customers time to update.
It all comes down to security. Does Magento plan to leave support to software versions that can be exploited in the future or force customers to update to new versions to get best in class security?

romainruaud · 2019-02-26T17:22:26Z

OK @maghamed I think I updated my PR before you drop your comment on the issue :)

It's now built as a separate module. Nothing altered in the old Magento\Elasticsearch module and everything specifically needed for ES6 in a new one.

Just one point : I'm not completely aware about what to do to ensure the new module will be properly created on the Magento2 composer private repo.

Feel free to ask anything you want more.

Regards

IvanPletnyov · 2019-02-27T13:27:49Z

@romainruaud Hello. I've created simple product with sku = 'Simple product 1' http://prntscr.com/mqukzb
but in advanced search by sku product isn't founded http://prntscr.com/mqultq .
Is it expected behavior ?

romainruaud · 2019-02-27T13:37:16Z

@IvanPletnyov this use case does not seem to work on other Elasticsearch implementations (2.x or 5.x).

So most probably, it has never worked as expected.

If it's easy to fix I can have a look but unfortunately, the main topic of this PR is not to fix all existing functional bugs on Elasticsearch implementation, but rather to ensure compatibility with current stable version of ES 6.

Regards

IvanPletnyov · 2019-02-27T13:49:18Z

@romainruaud Okay, thanks for the answer.

maghamed

In general, everything looks good.
@romainruaud you did a great job, and that's a great catch regarding ES6 support in Magento 2.3.1 especially taking into account that EOL of Elasticseach 5 is March 11th of 2019!

We will do our best to make this PR a part of Magento 2.3.1 release

maghamed · 2019-02-27T14:39:30Z

app/code/Magento/Elasticsearch6/Model/Config.php

+     * Return true if third party search engine is used
+     *
+     * @return bool
+     * @since 100.1.0


probably this @since 100.1.0 is excessive

maghamed · 2019-02-27T14:40:12Z

app/code/Magento/Elasticsearch6/Model/Config.php

+        $prefix = null
+    ) {
+        parent::__construct($scopeConfig, $clientResolver, $engineResolver, $prefix);
+        $this->engineResolver = $engineResolver ?: ObjectManager::getInstance()->get(EngineResolverInterface::class);


taking into account that it's new module - we can just pass all dependencies as required.
No need to mulate backward compatibility

I just set them as required.

magento-engcom-team · 2019-02-27T14:44:40Z

Hi @maghamed, thank you for the review.
ENGCOM-4389 has been created to process this Pull Request

magento-engcom-team · 2019-02-27T14:44:42Z

@romainruaud thank you for contributing. Please accept Community Contributors team invitation here to gain extended permissions for this repository.

ghost · 2019-03-02T02:20:17Z

Hi @romainruaud, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

darksummenors · 2019-03-08T09:41:04Z

So the ES6 will be supported with version 2.3.2? Thanks romainruaud for the great work! 👍

romainruaud · 2019-03-08T09:43:23Z

@darksummenors I think Magento is working internally to make it reach the 2.3.1 release pipeline but it's out of my scope. Maybe I was too much "just-in-time" for 2.3.1 if they flagged it for the next release.

Probably @maghamed can clarify this point.

Regards

tylers-username · 2019-03-13T13:10:58Z

@romainruaud - thanks for working on this.

darksummenors · 2019-03-28T13:40:21Z

It has been added to 2.3.1 (noticed it release notes)!

tylers-username · 2019-04-02T15:28:13Z

@darksummenors - also 2.2.8

romainruaud added Component: Elasticsearch Partner: Smile Pull Request is created by partner Smile labels Feb 26, 2019

romainruaud requested a review from maghamed February 26, 2019 11:15

magento-engcom-team added Component: AdvancedSearch Release Line: 2.3 partners-contribution Pull Request is created by Magento Partner labels Feb 26, 2019

maghamed self-assigned this Feb 26, 2019

romainruaud force-pushed the feature_es6-compat branch 2 times, most recently from 742fa8c to 4d81048 Compare February 26, 2019 17:14

romainruaud force-pushed the feature_es6-compat branch 2 times, most recently from 78bd12a to 0ed7ebc Compare February 27, 2019 08:01

romainruaud added 2 commits February 27, 2019 13:06

Elasticsearch6 implementation.

0e4ac3b

Allow more recent version of ES client.

57ecd34

romainruaud force-pushed the feature_es6-compat branch from 0ed7ebc to 1332aaf Compare February 27, 2019 12:07

Update composer.json replace.

ba236f9

romainruaud force-pushed the feature_es6-compat branch from 1332aaf to ba236f9 Compare February 27, 2019 12:42

maghamed approved these changes Feb 27, 2019

View reviewed changes

romainruaud added 3 commits February 27, 2019 15:57

Update module dependencies.

bdb032d

Remove excessive since.

b2561d9

Set dependencies as required.

d49a776

maghamed added Award: complex Award: special achievement labels Feb 27, 2019

magento-engcom-team added the Progress: accept label Feb 27, 2019

magento#21458: Fixed static tests

03f3a4a

magento-engcom-team merged commit 03f3a4a into magento:2.3-develop Mar 2, 2019

magento-engcom-team pushed a commit that referenced this pull request Mar 2, 2019

ENGCOM-4389: Elasticsearch6 implementation. #21458

68a2508

magento-engcom-team added this to the Release: 2.3.2 milestone Mar 2, 2019

This was referenced Mar 5, 2019

ElasticSearch indexer breaks on mapper_parsing_exception #21583

Closed

Notice: Undefined index: suggestions #21595

Closed

magento-engcom-team added the Award: category of expertise label Mar 6, 2019

jaideepghosh mentioned this pull request Mar 22, 2019

Search Engine - Elasticsearch throws "mapper_parsing_exception" #21889

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elasticsearch6 implementation. #21458

Elasticsearch6 implementation. #21458

romainruaud commented Feb 26, 2019 •

edited

Loading

magento-engcom-team commented Feb 26, 2019

magento-cicd2 commented Feb 26, 2019 •

edited

Loading

orlangur commented Feb 26, 2019

romainruaud commented Feb 26, 2019

orlangur commented Feb 26, 2019

romainruaud commented Feb 26, 2019

romainruaud commented Feb 26, 2019

CajuCLC commented Feb 26, 2019

miguelbalparda commented Feb 26, 2019

CajuCLC commented Feb 26, 2019

romainruaud commented Feb 26, 2019

IvanPletnyov commented Feb 27, 2019

romainruaud commented Feb 27, 2019

IvanPletnyov commented Feb 27, 2019

maghamed left a comment

maghamed Feb 27, 2019

romainruaud Feb 27, 2019

maghamed Feb 27, 2019

romainruaud Feb 27, 2019

magento-engcom-team commented Feb 27, 2019

magento-engcom-team commented Feb 27, 2019

ghost commented Mar 2, 2019

darksummenors commented Mar 8, 2019

romainruaud commented Mar 8, 2019

tylers-username commented Mar 13, 2019

darksummenors commented Mar 28, 2019

tylers-username commented Apr 2, 2019

Elasticsearch6 implementation. #21458

Elasticsearch6 implementation. #21458

Conversation

romainruaud commented Feb 26, 2019 • edited Loading

Description (*)

Fixed Issues (if relevant)

Manual testing scenarios (*)

Contribution checklist (*)

magento-engcom-team commented Feb 26, 2019

magento-cicd2 commented Feb 26, 2019 • edited Loading

orlangur commented Feb 26, 2019

romainruaud commented Feb 26, 2019

orlangur commented Feb 26, 2019

romainruaud commented Feb 26, 2019

romainruaud commented Feb 26, 2019

CajuCLC commented Feb 26, 2019

miguelbalparda commented Feb 26, 2019

CajuCLC commented Feb 26, 2019

romainruaud commented Feb 26, 2019

IvanPletnyov commented Feb 27, 2019

romainruaud commented Feb 27, 2019

IvanPletnyov commented Feb 27, 2019

maghamed left a comment

Choose a reason for hiding this comment

maghamed Feb 27, 2019

Choose a reason for hiding this comment

romainruaud Feb 27, 2019

Choose a reason for hiding this comment

maghamed Feb 27, 2019

Choose a reason for hiding this comment

romainruaud Feb 27, 2019

Choose a reason for hiding this comment

magento-engcom-team commented Feb 27, 2019

magento-engcom-team commented Feb 27, 2019

ghost commented Mar 2, 2019

darksummenors commented Mar 8, 2019

romainruaud commented Mar 8, 2019

tylers-username commented Mar 13, 2019

darksummenors commented Mar 28, 2019

tylers-username commented Apr 2, 2019

romainruaud commented Feb 26, 2019 •

edited

Loading

magento-cicd2 commented Feb 26, 2019 •

edited

Loading