Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Can I use Envoy as a front-end proxy? #8001

Merged
merged 22 commits into from
Oct 31, 2019
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
672bf99
docs: Can I use Envoy as a front-end proxy?
PiotrSikora Aug 12, 2019
66e6445
Merge remote-tracking branch 'origin/master' into setec/edge
PiotrSikora Oct 13, 2019
26c74c6
review: move from FAQ to Best Practices.
PiotrSikora Oct 14, 2019
b11be8b
review: use KiB, MiB, GiB.
PiotrSikora Oct 14, 2019
5a7592f
review: refresh_interval: 0.25s.
PiotrSikora Oct 14, 2019
79f42f5
review: add envoy.overload_actions.shrink_heap.
PiotrSikora Oct 14, 2019
258e048
review: add idle_timeout: 840s.
PiotrSikora Oct 14, 2019
10c2a90
review: explain use_remote_address: true.
PiotrSikora Oct 14, 2019
8189e16
Merge remote-tracking branch 'origin/master' into PiotrSikora/faq-edge
PiotrSikora Oct 30, 2019
2aa0f9b
review: add FAQ.
PiotrSikora Oct 30, 2019
93e8c68
review: drop useless praise.
PiotrSikora Oct 30, 2019
4dfac86
review: use bullet points.
PiotrSikora Oct 30, 2019
a2bfc74
review: fix format.
PiotrSikora Oct 30, 2019
a77b4c2
review: add timeouts.
PiotrSikora Oct 30, 2019
f6c1265
review: configure admin endpoint.
PiotrSikora Oct 30, 2019
8c5e80f
review: actually expose edge proxy to the Internet.
PiotrSikora Oct 30, 2019
8e87f2f
review: git add docs/root/faq/configuration/edge.rst.
PiotrSikora Oct 30, 2019
f95179c
review: add FAQ to the overview.
PiotrSikora Oct 30, 2019
4494914
review: drop "the".
PiotrSikora Oct 31, 2019
f375512
review: more timeouts.
PiotrSikora Oct 31, 2019
c1e65ed
reivew: remove max_concurrent_streams from upstream config.
PiotrSikora Oct 31, 2019
20bd32d
review: add note about the PROXY protocol.
PiotrSikora Oct 31, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 106 additions & 4 deletions docs/root/configuration/best_practices/edge.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,107 @@
Configuring Envoy for the edge
==============================
.. _best_practices_edge:

This document describes best practices for configuring Envoy for the edge. It is currently a
work-in-progress
Configuring Envoy as an edge proxy
==================================

Envoy is a production-ready edge proxy, however, the default settings are tailored
for the service mesh use case, and some values need to be adjusted when using Envoy
as an edge proxy.

TCP proxies should configure:

* restrict access to the admin endpoint,
* :ref:`overload_manager <config_overload_manager>`,
* :ref:`listener buffer limits <envoy_api_field_Listener.per_connection_buffer_limit_bytes>` to 32 KiB,
* :ref:`cluster buffer limits <envoy_api_field_Cluster.per_connection_buffer_limit_bytes>` to 32 KiB.

HTTP proxies should additionally configure:

* :ref:`use_remote_address <envoy_api_field_config.filter.network.http_connection_manager.v2.HttpConnectionManager.use_remote_address>`
to true (to avoid consuming HTTP headers from external clients, see :ref:`HTTP header sanitizing <config_http_conn_man_header_sanitizing>`
for details),
* :ref:`connection and stream timeouts <faq_configuration_timeouts>`,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the guidance here? Is the idea that "anything, as long as it's not effectively infinite" is fine? Which are the timeouts that need to be added, which are optional?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The linked FAQ has a few pages about timeouts, so I didn't see a point in repeating it all that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think even a short sentence "see the FAQ at for details on how to configure" is a useful addition. I agree we don't want redundancy and we want a single source of truth, but we can also try and guide the reader.

In general, I think it's instructive to put oneself in the shoes of the average reader. They arrive at this page with the following thought: Envoy timeouts, limits, configuration knobs, WTF ¯\_(ツ)_/¯

Our goal here is not just to tell them truthful and correct things, but to help demystify and have them leave with a feeling that they fully grok the mystery.

* :ref:`HTTP/2 maximum concurrent streams limit <envoy_api_field_core.Http2ProtocolOptions.max_concurrent_streams>` to 100,
* :ref:`HTTP/2 initial stream window size limit <envoy_api_field_core.Http2ProtocolOptions.initial_stream_window_size>` to 64 KiB,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How much difference does it make if your perimeter on the public Internet vs. Cloud? Do we need to consider bandwidth-delay product?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't really matter, it's here to disable Envoy's default which is 256 MiB (HTTP/2 default is 65535 B).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, but the meta point is that this document currently lacks the conveying of this intuition. These docs need to help educate the reader about what is important, why we do things the way we do them, not only to provide some suggested config settings (although the latter is useful as a start).

* :ref:`HTTP/2 initial connection window size limit <envoy_api_field_core.Http2ProtocolOptions.initial_connection_window_size>` to 1 MiB.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you comment on the relative relationship between connection and stream window default sizes? Is there an idea l ratio?


The following is a YAML example of the above recommendation.

.. code-block:: yaml

overload_manager:
refresh_interval: 0.25s
resource_monitors:
- name: "envoy.resource_monitors.fixed_heap"
config:
# TODO: Tune for your system.
max_heap_size_bytes: 2147483648 # 2 GiB
actions:
- name: "envoy.overload_actions.shrink_heap"
triggers:
- name: "envoy.resource_monitors.fixed_heap"
threshold:
value: 0.95
- name: "envoy.overload_actions.stop_accepting_requests"
triggers:
- name: "envoy.resource_monitors.fixed_heap"
threshold:
value: 0.98

admin:
access_log_path: "/var/log/envoy_admin.log"
address:
socket_address:
address: 127.0.0.1
port_value: 9090

static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 443
listener_filters:
- name: "envoy.listener.tls_inspector"
typed_config: {}
per_connection_buffer_limit_bytes: 32768 # 32 KiB
filter_chains:
- filter_chain_match:
server_names: ["example.com", "www.example.com"]
tls_context:
common_tls_context:
tls_certificates:
- certificate_chain: { filename: "example_com_cert.pem" }
private_key: { filename: "example_com_key.pem" }
filters:
- name: envoy.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
stat_prefix: ingress_http
use_remote_address: true
common_http_protocol_options:
idle_timeout: 3600s # 1 hour
http2_protocol_options:
max_concurrent_streams: 100
initial_stream_window_size: 65536 # 64 KiB
initial_connection_window_size: 1048576 # 1 MiB
stream_idle_timeout: 300s # 5 mins
request_timeout: 300s # 5 mins
route_config:
virtual_hosts:
- name: default
domains: "*"
routes:
- match: { prefix: "/" }
route: { cluster: service_foo }
clusters:
name: service_foo
connect_timeout: 15s
per_connection_buffer_limit_bytes: 32768 # 32 KiB
hosts:
socket_address:
address: 127.0.0.1
port_value: 8080
http2_protocol_options:
max_concurrent_streams: 100
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure that we want this for upstream. IMO the default is fine and have circuit breakers to limit this on a per-cluster basis as well as the downstream limits.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the scope here? Do we want to over edge use cases where Envoy is acting as an egress gateway and speaking to untrusted backends? In this situation, we also need to be defensive on the upstream.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't support push, so this provides no protection against the upstream. IIRC it might also limit the number of outgoing streams, but that is already controlled by downstream limits and circuit breakers.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. I guess I'd suggest that we do configure all the other defensive limits that we have, or at least provide some nuanced discussion on trust model here. In fact, I think there should be a short section on trust model. By edge, are we really saying "edge ingress"? Because, for "edge egress", we know today that there are some missing features that came up during the HTTP/2 headers CVE handling for example, that are needed when dealing with untrusted upstreams.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, let's be explicit and also call out a warning against untrusted upstream.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you want to see here? It's unclear to me from the comments.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would delete this line. I don't think it does anything useful.

initial_stream_window_size: 65536 # 64 KiB
initial_connection_window_size: 1048576 # 1 MiB
7 changes: 7 additions & 0 deletions docs/root/faq/configuration/edge.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
.. _faq_edge:

How do I configure Envoy as an edge proxy?
==========================================

Refer to the :ref:`configuring Envoy as an edge proxy <best_practices_edge>`
for an example of the edge proxy configuration.
1 change: 1 addition & 0 deletions docs/root/faq/overview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ Configuration
.. toctree::
:maxdepth: 2

configuration/edge
configuration/sni
configuration/zone_aware_routing
configuration/zipkin_tracing
Expand Down