docs: Can I use Envoy as a front-end proxy? #8001

PiotrSikora · 2019-08-22T00:12:33Z

Signed-off-by: Piotr Sikora piotrsikora@google.com

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

PiotrSikora · 2019-08-22T00:12:46Z

cc @mattklein123 @yanavlasov @htuch

docs/root/faq/edge.rst

mattklein123

A few comments to get started. Thanks for excited to see us do this.

/wait

docs/root/faq/edge.rst

htuch

Great. I think we can land something pretty soon and iterate. Thanks @PiotrSikora !

docs/root/faq/edge.rst

stale · 2019-08-30T19:14:00Z

This pull request has been automatically marked as stale because it has not had activity in the last 7 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

mattklein123 · 2019-09-07T03:04:12Z

@PiotrSikora I would really love to land this and iterate given the number of questions we get on this topic. Do you think you will have time to action on the comments? If not I can potentially take this over next week.

yanavlasov

+1 on explaining the need for the use_remote_address setting.

mattklein123 · 2019-10-10T21:51:38Z

@PiotrSikora friendly ping. I would like to land this for 1.12.0 by the end of the month. Will you be able to finish this out? If not I can take it over.

htuch · 2019-10-13T15:09:23Z

docs/root/faq/edge.rst

@@ -0,0 +1,85 @@
+.. _faq_edge:


Should we treat #2763 as well? I feel folks need some guidance on "best practices for secure configuration", and even though we warn about locking down the admin endpoint elsewhere, having this in one place would make sense.

Protecting /admin endpoint is not exclusive to front-end proxies, so I think that it should be documented in a separate entry, and linked from here when it's ready.

I agree, but in the interim can we at least show an admin config that only listens on localhost with a small comment?

Yeah, please do, and at least say a few sentences about this. This isn't one of the places we're aiming for minimalism and zero redundancy; it's a really major security hole if misconfigured and operators should be reminded about this pretty much everywhere that there is discussion of security + Envoy.

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

docs/root/configuration/best_practices/edge.rst

mattklein123 · 2019-10-14T20:18:27Z

docs/root/configuration/best_practices/edge.rst

+          address: 127.0.0.1
+          port_value: 8080
+      http2_protocol_options:
+        max_concurrent_streams: 100


I'm not sure that we want this for upstream. IMO the default is fine and have circuit breakers to limit this on a per-cluster basis as well as the downstream limits.

What is the scope here? Do we want to over edge use cases where Envoy is acting as an egress gateway and speaking to untrusted backends? In this situation, we also need to be defensive on the upstream.

We don't support push, so this provides no protection against the upstream. IIRC it might also limit the number of outgoing streams, but that is already controlled by downstream limits and circuit breakers.

Right. I guess I'd suggest that we do configure all the other defensive limits that we have, or at least provide some nuanced discussion on trust model here. In fact, I think there should be a short section on trust model. By edge, are we really saying "edge ingress"? Because, for "edge egress", we know today that there are some missing features that came up during the HTTP/2 headers CVE handling for example, that are needed when dealing with untrusted upstreams.

Agreed, let's be explicit and also call out a warning against untrusted upstream.

What do you want to see here? It's unclear to me from the comments.

I would delete this line. I don't think it does anything useful.

docs/root/configuration/best_practices/edge.rst

mattklein123 · 2019-10-14T20:21:54Z

docs/root/faq/edge.rst

@@ -0,0 +1,85 @@
+.. _faq_edge:


I agree, but in the interim can we at least show an admin config that only listens on localhost with a small comment?

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

mattklein123

Thanks, LGTM with some small comments. Let's ship and iterate after this.

I know that @alyssawilk has some additional thoughts on recommendations but we can add those later.

/wait

docs/root/faq/configuration/edge.rst

docs/root/configuration/best_practices/edge.rst

mattklein123 · 2019-10-30T23:00:48Z

docs/root/configuration/best_practices/edge.rst

+          address: 127.0.0.1
+          port_value: 8080
+      http2_protocol_options:
+        max_concurrent_streams: 100


I would delete this line. I don't think it does anything useful.

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

PiotrSikora · 2019-10-31T19:18:23Z

@mattklein123 I believe all comments were addressed, in case you still want to ship this in 1.12.0.

htuch · 2019-10-31T19:43:34Z

docs/root/configuration/best_practices/edge.rst

+* :ref:`connection and stream timeouts <faq_configuration_timeouts>`,
+* :ref:`HTTP/2 maximum concurrent streams limit <envoy_api_field_core.Http2ProtocolOptions.max_concurrent_streams>` to 100,
+* :ref:`HTTP/2 initial stream window size limit <envoy_api_field_core.Http2ProtocolOptions.initial_stream_window_size>` to 64 KiB,
+* :ref:`HTTP/2 initial connection window size limit <envoy_api_field_core.Http2ProtocolOptions.initial_connection_window_size>` to 1 MiB.


Can you comment on the relative relationship between connection and stream window default sizes? Is there an idea l ratio?

htuch · 2019-10-31T19:44:17Z

docs/root/configuration/best_practices/edge.rst

+  for details),
+* :ref:`connection and stream timeouts <faq_configuration_timeouts>`,
+* :ref:`HTTP/2 maximum concurrent streams limit <envoy_api_field_core.Http2ProtocolOptions.max_concurrent_streams>` to 100,
+* :ref:`HTTP/2 initial stream window size limit <envoy_api_field_core.Http2ProtocolOptions.initial_stream_window_size>` to 64 KiB,


How much difference does it make if your perimeter on the public Internet vs. Cloud? Do we need to consider bandwidth-delay product?

It doesn't really matter, it's here to disable Envoy's default which is 256 MiB (HTTP/2 default is 65535 B).

Yeah, but the meta point is that this document currently lacks the conveying of this intuition. These docs need to help educate the reader about what is important, why we do things the way we do them, not only to provide some suggested config settings (although the latter is useful as a start).

htuch · 2019-10-31T19:45:31Z

docs/root/configuration/best_practices/edge.rst

+* :ref:`use_remote_address <envoy_api_field_config.filter.network.http_connection_manager.v2.HttpConnectionManager.use_remote_address>`
+  to true (to avoid consuming HTTP headers from external clients, see :ref:`HTTP header sanitizing <config_http_conn_man_header_sanitizing>`
+  for details),
+* :ref:`connection and stream timeouts <faq_configuration_timeouts>`,


What is the guidance here? Is the idea that "anything, as long as it's not effectively infinite" is fine? Which are the timeouts that need to be added, which are optional?

The linked FAQ has a few pages about timeouts, so I didn't see a point in repeating it all that.

I think even a short sentence "see the FAQ at for details on how to configure" is a useful addition. I agree we don't want redundancy and we want a single source of truth, but we can also try and guide the reader.

In general, I think it's instructive to put oneself in the shoes of the average reader. They arrive at this page with the following thought: Envoy timeouts, limits, configuration knobs, WTF ¯\_(ツ)_/¯

Our goal here is not just to tell them truthful and correct things, but to help demystify and have them leave with a feeling that they fully grok the mystery.

mattklein123

This LGTM as a first cut, thanks. I think @htuch comments can be looked as follow ups but will defer to him on further review and merge.

htuch · 2019-10-31T19:57:23Z

If you want to merge for 1.12.0, this is OK. I would appreciate followup on my comments. To me, I think we still have a ton of magic in how to configure Envoy for the edge, but this doc is a useful beach head and starting point.

mattklein123 · 2019-10-31T20:07:13Z

To me, I think we still have a ton of magic in how to configure Envoy for the edge, but this doc is a useful beach head and starting point.

I agree completely, but I would rather ship and iterate on this as this certainly puts us in a much better place. @htuch will defer to you to approve/merge if you want.

mattklein123 · 2019-10-31T21:49:09Z

Let's merge this to get this into the 1.12.0 docs and we can iterate from here.

Description: use_proxy_proto is part of the listener.FilterChain (not HttpConnectionManager) Risk Level: None Testing: Manually tested with envoyproxy/envoy-alpine:v1.12.1 (Docker image) Docs Changes: see description Release Notes: docs: corrected use of use_proxy_proto in edge-proxy best-practice example Fix for #8001 Signed-off-by: Dominik <dominik-k@mailbox.org>

docs: Can I use Envoy as a front-end proxy?

672bf99

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

glerchundi mentioned this pull request Aug 22, 2019

config: sane defaults for envoy projectcontour/contour#1375

Closed

snowp reviewed Aug 22, 2019

View reviewed changes

docs/root/faq/edge.rst Outdated Show resolved Hide resolved

mattklein123 assigned mattklein123, htuch and alyssawilk Aug 22, 2019

mattklein123 requested changes Aug 22, 2019

View reviewed changes

repokitteh-read-only bot added the waiting label Aug 22, 2019

htuch reviewed Aug 23, 2019

View reviewed changes

docs/root/faq/edge.rst Outdated Show resolved Hide resolved

docs/root/faq/edge.rst Outdated Show resolved Hide resolved

docs/root/faq/edge.rst Outdated Show resolved Hide resolved

docs/root/faq/edge.rst Outdated Show resolved Hide resolved

ilackarms mentioned this pull request Aug 26, 2019

best practices for gateway-proxy helm chart kgateway-dev/kgateway#1084

Closed

stale bot added the stale stalebot believes this issue/PR has not been touched recently label Aug 30, 2019

mattklein123 added the no stalebot Disables stalebot from closing an issue label Aug 30, 2019

stale bot removed the stale stalebot believes this issue/PR has not been touched recently label Aug 30, 2019

mattklein123 mentioned this pull request Sep 2, 2019

Large size of http2_protocol_options initial_stream_window_size causes send/read timeout on ALB/nginx upstream on large responses and slow clients #8114

Closed

yanavlasov requested changes Sep 9, 2019

View reviewed changes

mattklein123 mentioned this pull request Oct 10, 2019

FAQ for default settings for safety, benchmarking, etc. #5348

Open

htuch reviewed Oct 13, 2019

View reviewed changes

PiotrSikora added 7 commits October 13, 2019 21:57

Merge remote-tracking branch 'origin/master' into setec/edge

66e6445

review: move from FAQ to Best Practices.

26c74c6

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

review: use KiB, MiB, GiB.

b11be8b

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

review: refresh_interval: 0.25s.

5a7592f

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

review: add envoy.overload_actions.shrink_heap.

79f42f5

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

review: add idle_timeout: 840s.

258e048

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

review: explain use_remote_address: true.

10c2a90

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

repokitteh-read-only bot removed the waiting label Oct 14, 2019

mattklein123 reviewed Oct 14, 2019

View reviewed changes

repokitteh-read-only bot removed the waiting label Oct 30, 2019

PiotrSikora added 6 commits October 30, 2019 01:39

review: fix format.

a2bfc74

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

review: add timeouts.

a77b4c2

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

review: configure admin endpoint.

f6c1265

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

review: actually expose edge proxy to the Internet.

8c5e80f

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

review: git add docs/root/faq/configuration/edge.rst.

8e87f2f

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

review: add FAQ to the overview.

f95179c

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

mattklein123 requested changes Oct 30, 2019

View reviewed changes

repokitteh-read-only bot added waiting and removed waiting labels Oct 30, 2019

PiotrSikora added 3 commits October 31, 2019 00:59

review: drop "the".

4494914

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

review: more timeouts.

f375512

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

reivew: remove max_concurrent_streams from upstream config.

c1e65ed

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

PiotrSikora force-pushed the faq-edge branch from 8955ba2 to c1e65ed Compare October 31, 2019 01:01

mattklein123 added the waiting label Oct 31, 2019

review: add note about the PROXY protocol.

20bd32d

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

repokitteh-read-only bot removed the waiting label Oct 31, 2019

htuch suggested changes Oct 31, 2019

View reviewed changes

mattklein123 approved these changes Oct 31, 2019

View reviewed changes

htuch approved these changes Oct 31, 2019

View reviewed changes

mattklein123 merged commit 6a5e23c into envoyproxy:master Oct 31, 2019

This was referenced Nov 20, 2019

docs/best_practices/edge: fix use_proxy_proto location #9079

Closed

docs/best_practices/edge: fix use_proxy_proto location #9082

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: Can I use Envoy as a front-end proxy? #8001

docs: Can I use Envoy as a front-end proxy? #8001

PiotrSikora commented Aug 22, 2019

PiotrSikora commented Aug 22, 2019

mattklein123 left a comment

htuch left a comment

stale bot commented Aug 30, 2019

mattklein123 commented Sep 7, 2019

yanavlasov left a comment

mattklein123 commented Oct 10, 2019

htuch Oct 13, 2019

PiotrSikora Oct 14, 2019

mattklein123 Oct 14, 2019

htuch Oct 14, 2019

mattklein123 Oct 14, 2019

htuch Oct 15, 2019

mattklein123 Oct 15, 2019

htuch Oct 16, 2019

mattklein123 Oct 16, 2019

PiotrSikora Oct 30, 2019

mattklein123 Oct 30, 2019

mattklein123 Oct 14, 2019

mattklein123 left a comment

mattklein123 Oct 30, 2019

PiotrSikora commented Oct 31, 2019

htuch Oct 31, 2019

htuch Oct 31, 2019

PiotrSikora Oct 31, 2019

htuch Oct 31, 2019

htuch Oct 31, 2019

PiotrSikora Oct 31, 2019

htuch Oct 31, 2019

mattklein123 left a comment

htuch commented Oct 31, 2019

mattklein123 commented Oct 31, 2019

mattklein123 commented Oct 31, 2019

docs: Can I use Envoy as a front-end proxy? #8001

docs: Can I use Envoy as a front-end proxy? #8001

Conversation

PiotrSikora commented Aug 22, 2019

PiotrSikora commented Aug 22, 2019

mattklein123 left a comment

Choose a reason for hiding this comment

htuch left a comment

Choose a reason for hiding this comment

stale bot commented Aug 30, 2019

mattklein123 commented Sep 7, 2019

yanavlasov left a comment

Choose a reason for hiding this comment

mattklein123 commented Oct 10, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattklein123 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PiotrSikora commented Oct 31, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattklein123 left a comment

Choose a reason for hiding this comment

htuch commented Oct 31, 2019

mattklein123 commented Oct 31, 2019

mattklein123 commented Oct 31, 2019