Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using specific pass for each host in redis output #16206

Merged
merged 21 commits into from
Feb 24, 2020

Conversation

rvillablanca
Copy link
Contributor

@rvillablanca rvillablanca commented Feb 9, 2020

What does this PR do?

This PR adds support to redis output for multiple password

Why is it important?

It is useful when you have a multiple redis instances with different credentials.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works

Related issues

@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

1 similar comment
@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

@urso
Copy link

urso commented Feb 10, 2020

Thank you for the PR. I wonder if we want to enforce the : at the beginning, or rather want to make it optional. WDYT?

@rvillablanca
Copy link
Contributor Author

Yeah as I said before, for me it is ok both way but the second one (whithout :) seems the pass could be the username part.

@rvillablanca
Copy link
Contributor Author

So what will we do with the colon?

@urso
Copy link

urso commented Feb 12, 2020

Using redis:// like an URL seems to be quite common practice. Let's keep your implementation treating it as an URL. But let's check for the scheme as well, in case some other scheme like 'rediss' is used.

The redis output supports connecting to redis via TLS as well. The URL scheme would be rediss:// then. Depending on the scheme defined we should enable TLS for the connection. This is what the Elasticsearch output does as well (hidden in the go stdlib, though). If we use https even if TLS is not enabled, then the Elasticsearch output will still attempt to connect via TLS using default settings and the system certificate pool.

You want to give it a try introducing the 'rediss' scheme as well? Or shall follow up later?

@rvillablanca
Copy link
Contributor Author

Sure let's do it

@rvillablanca
Copy link
Contributor Author

@urso at the moment I am validating that a valid tls configuration is required when using rediss scheme, but I know this is not the expected way so I need to know where I can see for that default settings in elasticsearch output using https without tls configuration

@urso
Copy link

urso commented Feb 13, 2020

If TLS is used the Elasticsearch output itself does not much. The output only configures a TLS dialer. The net/http package will use the TLS dialer if the "https" schema is used. If the user did not explicitely configure TLS, then the dialer will use system (and go net) defaults.

The output creates a TCP and TLS dialer first, and assignes those to the http.Transport.

If "https" is used net/http will use the TLS DLS dialer we did set up. In case TLS was not configured, then the crypto/tls package will fallback to its default configuration.

All in all, it is more a matter of configuring and selecting the correct dialer based on the scheme used.

* using specific port instead defaultPort when apply
* default tls config when ussing 'rediss' scheme
@urso
Copy link

urso commented Feb 14, 2020

Please add some integration tests as well. Check out redis_integration_test.go. Thanks.

@rvillablanca
Copy link
Contributor Author

I made all your suggestions, only integration test remain but I need some time, thanks for your mentoring @urso 👌

@rvillablanca
Copy link
Contributor Author

rvillablanca commented Feb 20, 2020

Before writing integration test I'd like to make sure that they are actually passing so, I suppose I can do that by running make integration-tests-environment right ?

@rvillablanca
Copy link
Contributor Author

Running make integration-tests-environment takes a long time but finishes successfully, I wonder if there is a way to run integration tests for redis output only, with the environment in docker.

I tried it by running make start-environment and then go test -tags integration . in redis output directory, but It couldn't connect with dockerized redis (no docker port mapping).

@urso
Copy link

urso commented Feb 22, 2020

Jenkins, test this.

@rvillablanca
Copy link
Contributor Author

I had to add the license header manually because every time I have to run some script related with python I have different problems. I really expect the migration to python 3 is finished.

We are done with this @urso 👍

@rvillablanca
Copy link
Contributor Author

CI is failing and I still having problems running tasks with python, the message is /usr/bin/python2.7: No module named venv when I run make fmt update in the libbeat directory.

If I try to configure python with make python-env I'm receiving the same message. I tried to install virtualenv but it is already installed 🤔

rodrigo@vivobook:~/go/src/github.com/elastic/beats (specific-password-redis)*$ sudo pip2 install virtualenv
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Requirement already satisfied: virtualenv in /usr/lib/python2.7/site-packages (16.1.0)

@rvillablanca
Copy link
Contributor Author

According to this https://www.elastic.co/guide/en/beats/devguide/current/beats-contributing.html migration is done so I will try it again

@urso
Copy link

urso commented Feb 23, 2020

Jenkins, test this.

@urso
Copy link

urso commented Feb 23, 2020

Approved. I started an internal CI run and will merge + prepare backports to release branches tomorrow. I expect the enhancement to become available with 7.7.0.

@urso urso added enhancement review Team:Services (Deprecated) Label for the former Integrations-Services team test-plan Add this PR to be manual test plan v7.7.0 labels Feb 23, 2020
@urso urso merged commit 0d31ea2 into elastic:master Feb 24, 2020
@rvillablanca rvillablanca deleted the specific-password-redis branch February 24, 2020 13:33
urso pushed a commit to urso/beats that referenced this pull request Feb 24, 2020
This PR adds support for redis URL schema when configuring the hosts. Each url can have it's own password, overwriting the outputs password setting. The URL scheme used can enable or disable TLS support as well. Using `redis` we always disable TLS, but when using `rediss` TLS will be enabled.

(cherry picked from commit 0d31ea2)
@urso urso mentioned this pull request Feb 24, 2020
2 tasks
urso pushed a commit that referenced this pull request Feb 24, 2020
This PR adds support for redis URL schema when configuring the hosts. Each url can have it's own password, overwriting the outputs password setting. The URL scheme used can enable or disable TLS support as well. Using `redis` we always disable TLS, but when using `rediss` TLS will be enabled.

(cherry picked from commit 0d31ea2)

Co-authored-by: Rodrigo Villablanca Vásquez <villa061004@gmail.com>
@urso
Copy link

urso commented Feb 24, 2020

PR and backports are merged. Thank you for taking the time to improve the redis output!

@andresrc andresrc added the test-plan-added This PR has been added to the test plan label Mar 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement needs_docs review Team:Services (Deprecated) Label for the former Integrations-Services team test-plan Add this PR to be manual test plan test-plan-added This PR has been added to the test plan v7.7.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Redis output multiple password support.
4 participants