S3 Source Connector does not enforce SSL/TLS for non-S3 endpoints

[brianjlai]

Tell us about the problem you're trying to solve

For the Amazon S3 source, the TLS option is not on by default and users can leave it off.

S3 only allows encrypted connections so this does only apply to non-S3 endpoints.

Describe the solution you’d like

We should always assume this value is true in the code. And to reduce confusion, this configuration option should also be removed when setting up a connector.

Describe the alternative you’ve considered or used

A clear and concise description of any alternative solutions or features you've considered or are using today.

Additional context

Add any other context or screenshots about the feature request here.

Are you willing to submit a PR?

[YowanR]

@lazebnyi This is the S3 issue that needs to be tackled asap. Thanks!
cc @YuliiaNahoha

[edgao]

@brianjlai I think we should only enforce this in cloud? Currently this means that if I have OSS and minio running side-by-side on my laptop, I would need to set up HTTPS also. On the DBs side we allow insecure connections in OSS, under the assumption that users are more likely to e.g. put everything inside their firewall or whatever

[brianjlai]

It does feel like something of a risk to assume that all OSS users will adhere to this and we're leaving the door open to a security loophole connection if misconfigured. But if that's the paradigm we're subscribing to for DBs for convenience of OSS that is fine too. But for DBs, how does the connector know to allow insecure connections? Is the connector aware of whether its running on cloud vs. OSS via some sort of input or is the config validated upstream?

[edgao]

yeah, there's definitely some interesting discussion about whether this paradigm is correct. There's always going to be some need for users who don't want to deal with these restrictions - but maybe this should be opt-in for OSS users who know they have a secured environment and can safely ignore them.

there's an env var for whether the container is running as OSS or cloud, but it would need to be ported to Python (cc @tuliren) - https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/bases/base-java/src/main/java/io/airbyte/integrations/base/adaptive/AdaptiveSourceRunner.java

example usage in source-postgres: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSourceRunner.java#L12-L15

Essentially:

• In OSS -> run using the normal spec
• in cloud -> run using the "strict encrypt" version, which is mostly identical, except that it removes the insecure connection options from the spec

[YowanR]

This is a really interesting discussion! Should we consider moving this to Slack so more folks can chime in?