Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support SSL Redis connections #81

Closed
zbarnes757 opened this issue Jul 26, 2023 · 14 comments
Closed

Support SSL Redis connections #81

zbarnes757 opened this issue Jul 26, 2023 · 14 comments

Comments

@zbarnes757
Copy link

Is your feature request related to a problem? Please describe.
We connect to our Redis instances with SSL but this client does not support that. I would like to see the ability to connect via SSL using the same TLS options I pass to the LD client.

Describe the solution you'd like
I believe the issue lies here: https://github.com/launchdarkly/erlang-server-sdk/blob/main/src/ldclient_storage_redis_server.erl#L46-L51

A simple solution would be to propagate the TLS options into eredis. Example:

    Host = ldclient_config:get_value(Tag, redis_host),
    Port = ldclient_config:get_value(Tag, redis_port),
    Database = ldclient_config:get_value(Tag, redis_database),
    Password = ldclient_config:get_value(Tag, redis_password),
    Prefix = ldclient_config:get_value(Tag, redis_prefix),
    HTTPOptions = ldclient_config:get_value(Tag, http_options),
    TLS = maps:get(tls_options, HTTPOptions, []),
    EredisOpts = [
        {host, Host}, {port, Port}, {database, Database}, {password, Password}, {tls, TLS}
    ],
    {ok, Client} = eredis:start_link(EredisOpts),
@kinyoklion
Copy link
Member

Hello @zbarnes757,

Thank you for the issue, and suggestion.

Filed internally as 211045

Thank you,
Ryan

@kinyoklion
Copy link
Member

Hey @zbarnes757,

As long as this approach meets the requirements of your infrastructure, I think it is a place that we can start.

Generally speaking I have some concerns that the certificates and verification appropriate for redis may not also be the configuration most appropriate for other connections. If that turns out to be a problem, then it wouldn't hurt to later allow more specific settings for the redis client. (At that time it could default to the http_options, and then use more specific redis_options if they are available.)

I'm starting to look at this, it will mostly be a case of validating configurations and the behavior of our version of eredis. If possible I would rather not require a new eredis version at the same time. (Though that does need to be done.)

Thanks,
Ryan

@kinyoklion
Copy link
Member

@zbarnes757,

I think this will require at least a bit of extra configuration. The primary problem with directly using the settings is that eredis will attempt to upgrade to tls if TLS options are provided.

This means, at a minimum, a setting to toggle.

The secondary problem I have found is with TLS negotiation. The same set of cipher suites may not be appropriate for general connections as well as redis.

So, I think, I will add some additional redis configuration that allows specific TLS configuration. If the same configuration works for you in both places, then you can easily provide that config to both.

Thank you,
Ryan

@zbarnes757
Copy link
Author

@kinyoklion that makes sense! Probably best to have separate tls configurations.

@kinyoklion
Copy link
Member

@zbarnes757,

Support has been released in version 2.1 of the SDK.

In my individual testing I ended up using a different configuration for redis than that used for HTTPS connections. This is because host name validation and ciphers for the redis configuration I was using were not easily compatible. If you only care about the encryption, versus the authenticity, then you could omit 'verify_peer'.

Additionally the redis configuraton supports both client and server options. This allows for client verification mode of redis.

Thank you,
Ryan

@kinyoklion kinyoklion added the waiting for feedback Indicates LaunchDarkly is waiting for customer feedback before issue is closed due to staleness. label Aug 1, 2023
@kinyoklion
Copy link
Member

Basic example, where TLS is enabled, but it doesn't verify authenticity.

    ldclient:start_instance("LD_SDK_KEY", #{
        feature_store => ldclient_storage_redis,
        redis_password => "some password",
        redis_host => "myurl",
        redis_port => 18749,
        redis_tls => [
            {cacertfile, "/some/path/redis_ca.pem"}
        ],
        http_options => #{
            tls_options => ldclient_config:tls_basic_options()
        }
    }),

If you omit redis_tls, or provide an empty list, then it will not use TLS.

Thank you,
Ryan

@zbarnes757
Copy link
Author

@kinyoklion I was able to get it to work! It took some finagling but I got it.

My setup:

redis_tls_options = [
      cacertfile: '/etc/ssl/certs/ca-certificates.crt',
      verify: :verify_none,
      customize_hostname_check: [match_fun: :public_key.pkix_verify_hostname_match_fun(:https)]
    ]

@zbarnes757
Copy link
Author

the only other thing I would recommend is adding the new option to the specs. My dialyzer is throwing warnings because of it.

@kinyoklion
Copy link
Member

@zbarnes757
That is good news that you have things working.

In regards to the dialyzer, where are you getting errors? The settings should be in the spec, and should allow any tls_option. It is possibly I missed something.

Thanks,
Ryan

@zbarnes757
Copy link
Author

@kinyoklion
Copy link
Member

Ah, thank you, yes. I see that typing is duplicated, I should sort that.

@kinyoklion
Copy link
Member

kinyoklion commented Aug 2, 2023

@zbarnes757 2.1.1 has been released that fixes the specs for the instance typing.

Thanks!
Ryan

@zbarnes757
Copy link
Author

perfect! Works like a charm. Shall I close the issue?

@kinyoklion
Copy link
Member

Ill go ahead and close it. If you encounter any other problems, then let us know.

Thank you,
Ryan

@kinyoklion kinyoklion removed the waiting for feedback Indicates LaunchDarkly is waiting for customer feedback before issue is closed due to staleness. label Aug 3, 2023
kaifee-haque pushed a commit to scorebet/erlang-server-sdk that referenced this issue Nov 13, 2023
* [ch65671] Mass rename from eld to ldclient (launchdarkly#49)

* Alpha => beta (launchdarkly#50)

* Changed package name and description (launchdarkly#51)

* Added event_format to ldclient_user, formatted all events, and added first and last name unit tests (launchdarkly#52)

* [ch76955] Fix dates in tests to be rfc3339 compliant (launchdarkly#53)

* [ch76953] Add Erlang/OTP 23 to circle tests (launchdarkly#54)

* [ch76961] Fix dialyzer warning for missing offline key in options type spec (launchdarkly#55)

* [ch77043] Workaround for fixing shotgun SSE parsing bug (launchdarkly#56)

* [ch82755] Add SSE tests and fix for network timeout on initial connection (launchdarkly#57)

* [ch83275] Handle socket closing in handle_cast (launchdarkly#58)

* Added additional case for socket closing in handle_cast ldclient_event_process_server

* Detect error upon sending events and allow retry

* Split http_request processing into separate function, properly handle error type

* Fixed is_http_error_recoverable return type, added error type to send return type

* [ch85404] Initialize updaters asynchronously, restart updates after crash (launchdarkly#60)

* [ch85404] Fix updater initialization state bugs (launchdarkly#61)

Fixed updater initialization status check, and added cleanup of that status when instance is stopped.
Also added an assertion that confirms expected evaluation result when updater hasn't yet made the initial connection.

* [ch68359] Undo quick fix for shotgun SSE event parsing (launchdarkly#63)

* Upgraded shotgun to 0.5.0, updated CHANGELOG for 1.0.0-beta4 (launchdarkly#62)

* Update version to beta4

* [ch89280] Ignore path in streaming put events (launchdarkly#64)

This fixes a bug that would cause streaming updater to crash with `function_clause` when connected to the relay proxy.

* [ch89932] Fix the events URL path from /api/events/bulk to /bulk (launchdarkly#65)

The old path was legacy and is not supported in relay proxy.

* Clean up generated documentation (launchdarkly#67)

* Added @Private annotation to private modules, added overview.edoc

* Remove version from overview.edoc

* [ch30413] Reload flag data when storage server fails (launchdarkly#68)

* Added flag reload to ets and map init if server fails

* Added storage server state

* Added reload check to ets and map

* Added storage server failure check to poll and stream server process response

* Reload flag data when in-memory storage servers are terminated

* Fixed insert_new bug by moving storage state init above storage init, moved storage state back into update processor state, tests pass

* Removed unnecessary export from ldclient_instance

* [ch55223] Fully parse flags and segments before storing them (launchdarkly#69)

* Parse flags before storage

* Fixed no flags being parsed in update processor

* Fixed 2 unit tests

* Fixed 2 more storage tests

* Added segment parsing, all tests for segments pass

* Fixed 8 unit tests

* Fixed 2 unit tests

* Fixed last failing unit test

* Fixed patch flag and segment parsing, updated unit tests (launchdarkly#70)

* Changed app.launchdarkly to sdk.launchdarkly, changed initial stream reconnect to 1s (launchdarkly#72)

* Cleaned up patch parsing code, segments now pass integration tests (launchdarkly#71)

* Cleaned up patch parsing code

* Pass function instead of atom for patch parsing

* Added unit test for specifying custom user values as custom map instead of as individual keys (launchdarkly#75)

* Added LDD Mode (launchdarkly#76)

* [ch59635] Feature Store: Redis (launchdarkly#74)

* Added eredis dep, changed StorageBackend to FeatureStore, changed put to upsert, added delete

* Mostly done redis prototype, still need to fix config loading and prefixes

* Fixed some build errors, partially passed Tag to redis

* Added prefix to buckets, unit test prototype

* Fixed concatenating prefixes and buckets, changed some eredis return checks

* Fix small test errors

* First 2 redis unit tests pass

* Partially fixed type translation for upsert

* All unit tests pass

* Add remote docker to CircleCI

* Fix spacing in config.yml

* Added circle-tests for circle docker redis

* Remove unnecessary circle docker step

* changed flags to features, added back redis_prefix

* Changed ldclient_settings to ldclient_config

* Changed term_to_binary to jsx:encode

* Fixed redis unit tests by always returning a map instead of term in key value tuple for get

* changed redis_host to redis_url

* Change redis_url back to redis_host

* Change default redis host from localhost to 127.0.0.1

* Added Docker to CONTRIBUTING.md, fixed unit test using old function name

* Removed eredis from app.src, changed FlagStoreServer to FeatureStoreServer, added eredis:stop to terminate

* Fix Dialyzer warning for create_bucket redis type spec

* Added buckets list to redis server state to prevent checking redis to see if a bucket exists

* Added Makefile comment for circle-tests, simplified list prepend

* Simplify bucket_exists

* Fixed unit tests, fixed broken delete stream test

* Added bucket_name func

* Redis Cache (launchdarkly#73)

* Added eredis dep, changed StorageBackend to FeatureStore, changed put to upsert, added delete

* Mostly done redis prototype, still need to fix config loading and prefixes

* Fixed some build errors, partially passed Tag to redis

* Added prefix to buckets, unit test prototype

* Fixed concatenating prefixes and buckets, changed some eredis return checks

* Fix small test errors

* First 2 redis unit tests pass

* Partially fixed type translation for upsert

* All unit tests pass

* Add remote docker to CircleCI

* Fix spacing in config.yml

* Added circle-tests for circle docker redis

* Remove unnecessary circle docker step

* changed flags to features, added back redis_prefix

* Changed ldclient_settings to ldclient_config

* Changed term_to_binary to jsx:encode

* Fixed redis unit tests by always returning a map instead of term in key value tuple for get

* changed redis_host to redis_url

* Change redis_url back to redis_host

* Change default redis host from localhost to 127.0.0.1

* Added Docker to CONTRIBUTING.md, fixed unit test using old function name

* Removed eredis from app.src, changed FlagStoreServer to FeatureStoreServer, added eredis:stop to terminate

* Fix Dialyzer warning for create_bucket redis type spec

* Added buckets list to redis server state to prevent checking redis to see if a bucket exists

* Added Makefile comment for circle-tests, simplified list prepend

* Simplify bucket_exists

* persistent storage caching layer

* Fixed some minor runtime bugs with Tag

* debugging

* Removed debugging prints, cache unit tests pass

* Fixed redis tests

* Added jsx encode/decode to cache, checked for empty map

* Fix unit tests after merge

* Fix lookup_key function spec

* Added unit test for cache ttl expiration

* [ch99672] Add new deleted flag if it doesn't already exist (launchdarkly#77)

* Add new deleted flag if it doesn't already exist

* Add NewVersion to deleted flag

* Changed list to all in eval (launchdarkly#78)

* [ch99670] Add Redis storage check to SDK initialization check (launchdarkly#79)

* Add Redis storage check to sdk initialization check

* Changed reload to false on redis server terminate

* Removed the guides link

* Fixed Redis Parsing (launchdarkly#80)


* Don't parse flags and segments before storing in Redis

* eredis to app.src

* Fix cache and patch

* Changed flag to camel case

* Changed rules and segments to camel case

* Converted clause to camel case

* Rename track_with_metric to track_metric (launchdarkly#81)

* Removing the beta banner (launchdarkly#82)

* Make internal storage files private from generated doc (launchdarkly#83)

* typechecking errors

* make dialyzer happy

* add missing handling for adding alias events

* add tests for alias events

* add second alias method

* remove unused imports and simplify alias function

* add otp 24 to circleci test coverage (launchdarkly#86)

* add opt 24 and elixir tests

* update dependencies to newer versions (launchdarkly#87)

* update deps

* [ch121494] Send header keys as binary for HTTP2 client compatibility (launchdarkly#88)

We were sending header keys as lists (strings) which worked for HTTP1, but failed for the HTTP2 protocol implementation in gun. Changing the keys to binary works for both.

* [ch122091] Force HTTP1 mode (launchdarkly#89)

The shotgun library doesn't support async streaming over HTTP2; force HTTP1 only.

* [ch122248] Add try/catch expressions during evaluation and SSE (launchdarkly#90)

Add try/catch expressions during evaluation and SSE processing to avoid undesired crashes.

* Updating from the deprecated elixir circleci image to a newer one (launchdarkly#91)

* ch123363 Update to not add a default variation of 0 to rules. Pass malformed reason all the way through. (launchdarkly#92)

* ch123363 Update to not add a default variation of 0 to rules. Pass malformed reason all the way through.

* ch123363 Remove CT from eval.

* ch123363 Revert flag change.

* ch123363 Re-work to not change exception behavior.

* ch123363 Cleanup.

* ch123363 Correct error message for case where rule is missing variation or rollout.

* ch123363 Correct case where the falthrough is malformed to support the default value.

* ch64830 Updates to improve the release process for erlang. (launchdarkly#93)

* ch64830 Initial pass at changes to the release process.

* ch64830 Add secrets configuration and update scripts.

* ch64830 Change prepare script to be more concise.

* ch64830 Unspecified image version.

* ch64830 Update prepare script. Add version back to docker image.

* ch64830 Update releaser config version.

* ch64830 Change capitalization gitHubPages.

* ch64830 Add zip.

* ch64830 Put the docs where releaser expects them.

* ch64830 Test publishing.

* ch64830 Test publishing.

* ch64830 Create directory for hex config.

* ch64830 Fix formatting of hex.config.

* ch64830 Add hex plugin to rebar.

* ch64830 Change executable permissions.

* ch64830 Remove testing code.

* ch64830 Add newline to end of file.

* ch108414 Implement support for experiment traffic allocation v2. (launchdarkly#95)

* ch99138 Add ability to use flags from files. (launchdarkly#94)

* sc-120173 Add support for http options. (launchdarkly#96)

Co-authored-by: zurab-darkly <56062336+zurab-darkly@users.noreply.github.com>

* Remove backticks from comments.

* sc-130423 Fix option typings and update dialyzer to allow analyzing t… (launchdarkly#97)

* Local filesystem monitor fix (launchdarkly#52)

* add single instance note

* launchdarkly#1 Initial implementation of the test service. (launchdarkly#99)

Co-authored-by: zurab-darkly <56062336+zurab-darkly@users.noreply.github.com>

* Test data source (launchdarkly#104)

* Handle null items in configuartion and commands. (launchdarkly#105)

* Pin hex plugin version. (launchdarkly#107)

* Rename master to main. (launchdarkly#108)

* [sc-157515] [sc-123638] Fix for use with OTP-25. Update shotgun to resolve issue with rebar3 compatibility with hex. Update certifi to latest version. (launchdarkly#109)

* Update parameter name. (launchdarkly#110)

* [sc-175162] Correctly handle deleted flags and segments. (launchdarkly#111)

* Implement backoff/jitter and temporary/permanent failure for streams. (launchdarkly#112)

* Implement support for server time. (launchdarkly#113)

* [sc-175575] Remove inlining users and alias events. (launchdarkly#114)

* [sc-175574] Implement attribute reference support. (launchdarkly#115)

* [sc-176205] Rename user_keys_capacity -> context_keys_capacity. (launchdarkly#117)

* Account for bookishspork sometimes producing correct dates. (launchdarkly#116)

* Implement basic context support. (launchdarkly#118)

Co-authored-by: Casey Waldren <cwaldren@launchdarkly.com>

* Allow null for track data. (launchdarkly#121)

* [sc-177158] implement context serialization and redaction redaction. (launchdarkly#119)

* [sc-177164] Implement support for creating an ldclient_context:context() from an ldclient_user:user(). (launchdarkly#122)

Co-authored-by: Casey Waldren <cwaldren@launchdarkly.com>

* [sc-177167] Add support for getting the context canonical key. (launchdarkly#120)

Co-authored-by: Casey Waldren <cwaldren@launchdarkly.com>

* Fix spacing from merge.

* [sc-180192] Remove secondary support. (launchdarkly#123)

* Add support for getting context keys, as required for events. (launchdarkly#124)

* [sc-181588] Add the ability to create a context from the JSON representation. (launchdarkly#126)

* [sc-176202] [sc-179455] Basic context flowing through everything. (launchdarkly#125)

Co-authored-by: Casey Waldren <cwaldren@launchdarkly.com>

* [sc-181658] Add support for targeting kind. (launchdarkly#127)

* Report malformed_flag for clauses with invalid attribute references (launchdarkly#128)

* Update contract tests to support V2. (launchdarkly#129)

* feat: Add support for application id and application version configuration. (launchdarkly#130)

In addition to support for application id and version this PR also refactors how headers are handled. It centralizes the definition of the default headers where previously it was duplicated in a few places.

This also adds the generic logic for tags beyond application id and version. Allowing sorting and generation of header strings for any number of tags and values.

* chore: Add support for the polling capability. (launchdarkly#131)

* build(deps): Bump lru version to 2.4.0 (launchdarkly#132)

* Fix docs for 1.6 release. Add docs to CI. (launchdarkly#133)

* fix: Do not attempt to access attribute references into non-map fields. (launchdarkly#136)

* feat: Add support for context targets in segments. (launchdarkly#135)

* feat: Add support for contextKind in rollouts. (launchdarkly#137)

* feat: Support contextTargets for flags. (launchdarkly#139)

* feat: Experimental rollouts always bucket using key. (launchdarkly#138)

* fix: Correct handling of name attribute, default anonymous to false. (launchdarkly#140)

* feat: Add support for nested segments. (launchdarkly#141)

* fix: Fix issues with events for U2C. (launchdarkly#142)

* [sc-187818] fix: Handle circular references in flag prereqs. (#143)

* feat: Update flagbuilder and testdata to support contexts. (#144)

* fix: Implement additional clause testing. Fix exception for invalid regex. (#145)

* chore: Update releaser config for maintenance branches. (#146)

* feat: Support rule building against context kinds. (#147)

* feat: Add user-type capability. (#148)

* fix: Do not attempt to send events for an invalid context. (#149)

---------

Co-authored-by: zurab-darkly <56062336+zurab-darkly@users.noreply.github.com>
Co-authored-by: Ben Woskow <48036130+bwoskow-ld@users.noreply.github.com>
Co-authored-by: Joe Cieslik <5600929+torchhound@users.noreply.github.com>
Co-authored-by: Zurab Davitiani <zurab@launchdarkly.com>
Co-authored-by: LaunchDarklyCI <dev@launchdarkly.com>
Co-authored-by: elliot <elliot@debian.elliot>
Co-authored-by: elliot <https://Apache-HB@github.com>
Co-authored-by: Elliot <35050275+Apache-HB@users.noreply.github.com>
Co-authored-by: Ben Woskow <bwoskow@launchdarkly.com>
Co-authored-by: LaunchDarklyReleaseBot <launchdarklyreleasebot@launchdarkly.com>
Co-authored-by: Ryan Lamb <4955475+kinyoklion@users.noreply.github.com>
Co-authored-by: Matt Glover <mpglover@gmail.com>
Co-authored-by: Molly <molly.jones@launchdarkly.com>
Co-authored-by: Ben Levy <benjaminlevy007@gmail.com>
Co-authored-by: Casey Waldren <cwaldren@launchdarkly.com>
Co-authored-by: Louis Chan <lchan@launchdarkly.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants