Elasticsearch 6.X and 5.x implementation

[nanov]

Elasticsearch 5.x db implementation, base operations. Only native Query DSL queries tested ( with query option native ).

[adrai]

Is it possible to save the version in the data i.e. _version property (like couchdb implementation) so no change on ViewModel is needed...
but only if possible

[adrai]

PS. do you think we could add tests for elasticsearch? and then add it to travis too? https://docs.travis-ci.com/user/database-setup/#ElasticSearch

[adrai]

And may you describe (in the readme) what is the difference between elasticsearch and elasticsearch5 implementation?

[nanov]

Elasticsearch gives some error with the "_version" property, one option is to name it other way but I think it is better to use the native Elasticsearch versioning.

This implementation will work also without modifying ViewModel ( as i add the version manually to the ViewModel on get and search ).

Tests are definitely a good idea and planned, hopefully I will get some time over the weekend to implement those.

[nanov]

PS. you are incredibly fast ;)

[adrai]

not always 😜

[nanov]

Well I moved on adapting the whole thing for Elasticsearch 6.X, the biggest diffrence is that from now on Elasticsearch 6 allows only one type per index ( eventually types are going to be removed in version 7 ). I've cleaned up the whole thing ( althoguht there is a lot more to be done ) and tested it on our working system for our purposes.

[nanov]

Now I am having the next dilemma, it would be good to have some specific Elasticsearch properties inside the repository definition ( like the indexes for mongoDb ), as the whole metadata is not stored inside the repository some changes will need to be made.

One solution would be to store the given metadata as a whole, this way each implementation could use the appropriate properties from the metadata when needed.

Another solution would be to take just a field named as the implementation and store it, for example if i use elasticsearch, if a field named "elasticsearch" is present in the viewmodel metadata ( ie. in the collection definition in node-event-denormalizer ) this field will be stored in the repository and be present in the implementation for use.

This is needed as there are some Elasticsearch index and alias settings which are really dependent on the nature and size of data you are storing.

What do you think would be a good approach on this?

[adrai]

Perhaps, extend the commit method on vm and on repository to accept an optional additional argument...

[nanov]

Well the thing is that this would ideally be in the collection definition ( exactly like the indexes property ), because it would affect the way the mapping for the index would be created. I guess this is more denormalizer related than viewmodel, maybe we should continue the discussion there?

[adrai]

If it works to have it in the checkConnection function it's better 😉

[nanov]

It will be eventually used in the checkConnection method, the question is how the user is going to pass the desired definition ?

The definition should be on the collection level for example :

module.exports = cqrsEventDenormalizer.defineCollection({
	name: 'coolEsIndexedCollection',
	context: 'coolDomain',
	defaultPayload: 'payload',
        esSettings:{ 
           settings: { // will be merged with default settings
		number_of_shards: 1, // default
		number_of_replicas: 0 // default
	    },
            mappings : { // will be merged with default (dynamic )mappings in the corresponding type
               properties: { 
                  title:    { "type": "text"  }, 
                  name:     { "type": "text"  }, 
                  age:      { "type": "integer" }, 
               }
            }
        }
});

[nanov]

Is there any particular reason why not to store and pass the meta from the collection to the repository?

[adrai]

No explict reason.

[nanov]

Together with thenativeweb/node-cqrs-eventdenormalizer#56 a full elasticsearch 6.X ( tested with 5.X as well ) is available now. With the right settings the indexing and mapping can be tweaked per repository ( or global ) basis for maximum performance.

[adrai]

As soon as you give the "ok" and add some tests, I will merge both PRs.

[nanov]

I think the first phase is complete. It could definitely use some tweaks and optimizations but those will be done in the next iteration after some more serious in-production use.

[nanov]

Another thing, when you merge, could you give the credit combined also to eCollect, as the PRs are done on company time?

[adrai]

release notes (+tweet) ok?

[nanov]

More than perfect!

[nanov]

I think both PRs are ready for merge.

[adrai]

ok... let's wait 24h in case you find something else, ok?

[adrai]

by the way... why is it named elasticsearch6 and not 5?

[nanov]

For the wait, sure!

For the version :

TL:DR
Basically it turned out to be a 6.x implementation which is compatible with 5.x and not the opposite.

Explanation
Well I have started targeting elasticsearch 5.x, but over the process, with stable elasticsearch 6.x over the corner ( that we already use on one of our environments ), i have decided to make it compatible with both ( biggest change is the removal of mapping types ).

[adrai]

shouldn't this be:

var indexes = (this.repositorySettings && this.repositorySettings.mongodb && this.repositorySettings.mongodb.indexes) ? this.repositorySettings.mongodb.indexes : this.indexes;

[adrai]

comment: elasticsearch6 ;-)

[nanov]

Absolutely, fixed.

[adrai]

@nanov What is your twitter handle?

[nanov]

https://twitter.com/ecollect

Thank you :)

[adrai]

And your personal?

[nanov]

I don't really use my twitter, you can just mention the company, it is totally fine.

[adrai]

https://twitter.com/adrirai/status/894849930635902976