Plugin Wanted: Elasticsearch publisher

[PatrykMatyjasek]

I'd like to get a publisher that can play well with Elasticsearch.

Elasticsearch exposes REST API which allows to create data record via POST call (https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html), or modify it with PUT command (https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html). Basically it should be able to create new records.
Also there are few configuration options like:

• protocol - http or https
• server address - default localhost
• server port - default 9200

All of them should be passed with task manifest configuration. There is no need to put this in global config.
There are should be possible some optional configuration options:

• user - for optional authentication
• password - for optional authentication
• path - endpoint to publish data

[mbbroberg]

Curious to follow up here - anyone known to be working on a publisher for ES?

[PatrykMatyjasek]

Hi @mjbrender I'm working on it.

[ldktta]

+1

[mbbroberg]

Cool to know @PatrykMatyjasek! As soon as you have a testable repo, could you add it to the Catalog (https://github.com/intelsdi-x/snap/blob/master/docs/plugins.yml)?

You can add Fixes #1413 in the PR to bring this all together 👌

[rakah]

I'm new to golang & snap but I was in a rush to see this functionality so I hacked together this. It's missing configuration(uses localhost/snap/test:9200) and tests - not near release ready for sure. It does seem to work, though. @PatrykMatyjasek, hoping this either helps get you going or that you're able to get something more professional out there for this. Looking forward to storing Snap data in ES either way and happy to help any way I can.

[PatrykMatyjasek]

@rakah thanks for your activity. I'll see your repo. Thanks!

[mbbroberg]

Hey @rakah! This is very cool. I'm sharing in our public Slack team to get others to maybe hop in and help you continue development. I hope you keep exploring as you get used to Go & Snap 👍

[mattyjones]

@rakah I forked your repo, I am going to take a look at it and can offer some help if you would like. Do you have specific points besides the points you list above. I can make pr's to your work as I am using it as a base or I can pr directly here and give you credit as well. The choice is yours

[rakah]

Thanks, @mattyjones - I'd be happy to take pr's on the project. This wasn't the best month for me time-wise but it looks like that's cleared up. My last commit for bulk updates should help with metric volume but I did it in a rush.. Variable names may be ugly, but the main thing I didn't like is on line 88 of elasticsearch.go.. http.NewRequest had some problem sending requests with data > 4KB.. Best would be a way to find a way around that but if that doesn't work then there's probably a much better algorithm for figuring out when we're closer to 4KB but not over - maybe just copying the data and then using the previous version when we see it's too big or something.

The ES I'm using snap on right now is just my home lab so I don't have SSL. It looks like that is an option in ES, but the config options I have now and calls to http aren't set up to do that.

Today if I get a chance I'm probably going to take another look at Travis and see if I can get the build pipeline for the thing to build a binary and send it to S3.. Then I want to take a look at other standards snap has for plugins and make sure I'm in line.

If someone really hammers metrics through this thing I could see wanting a connection pool to ES, I haven't had a chance to check practices on how to rig those up yet, though.

Again, I'm new to go so definitely feel free to just generally point out anything that seems against go principals or whatever, I've got thick skin.

I'll try to hop on Slack later and get up to speed on some conversations there..

[katarzyna-z]

Now Elasticsearch publisher is available at https://github.com/intelsdi-x/snap-plugin-publisher-elasticsearch.

@rakah Thanks for your activity but we needed to write Elasticsearch publisher plugin which fits better to our needs. Please take a look at plugin's configuration options, test the plugin and compare with your implementation. If you see something which can be added please open an issue and/or submit a pull request. We will appreciate your contribution.

[mbbroberg]

Hey @mattyjones & @rakah, it does seem my coworkers in Poland beat you to the plugin catalog 😄. I still added both of you to the awesome contribution list I have secretly been gathering.. stay tuned on that one.

I hope to see you two give the one @katarzyna-z made a try. Thanks again for being part of our budding community.

[mbbroberg]

My other takeaway is that it would be helpful to have more of a "claiming" system here for visibly. Docker community has a #dibs functionality (search "dibs" in this guide).

Or maybe it's not a concern and we go with whoever opens a PR to the catalog first. Any opinions from community members here?

[mattyjones]

@mjbrender I have no real thoughts on claiming or credit. It can be sticky as you have someone who was working through something publicly for several weeks/months and then someone else opens a pr that is ready to go through right away. IMHO both should be given credit towards the project but the one who wrote the pr and got it merged is already on record with that code.

I guess IMHO it depends on the specific situation, I always default to giving people credit for even the simplest of things just to foster goodwill as it can be important to either people or companies. FWIW, 50% of my resume is based on open source projects I have contributed to, I always considered open source development my excuse to not attend college. ;)

[rakah]

@mjbrender in this case, #dibs might have been good because I did not get all of the things I wanted in before @PatrykMatyjasek and @katarzyna-z made their PR. In some other case, it might not be good because someone waiting in the wings with some superior version gets locked out.

I read most of the new one and it's very good. I'm definitely going to give it a try. I still wonder if bulk updates & SSL would be good features so once I get a fork set up maybe I'll see everyone over at the project page.

[mattyjones]

@mjbrender @rakah I second SSL. If you can be secure, do it

[mbbroberg]

@rakah @mattyjones SSL for ES, right? If so, let's get an issue on that repo: https://github.com/intelsdi-x/snap-plugin-publisher-elasticsearch

Thanks for your feedback all! I'll look into a claiming system in more detail.