Add v3 xDS support (#140)

* Add v3 xDS support

This commit adds support for the v3 xDS transport as well as support for
storing and serving v3 resources out of the resource cache.

Note that resource versions and xDS transport version are decoupled to
provide a migration path to v3. End users can decide to generate v3
resources and serve them over v2 transport during migration, or continue
to generate v2 resources and serve them over v3 transport.

For implementation simplicity, it is not possible to generate a
combination of v2 and v3 resources in the control plane, which would
require either up or down conversion.

Signed-off-by: Michael Puncel <mpuncel@squareup.com>

* PR comments: move type URLs to their own static class. Fix broken javadoc link. Use method overloading instead of different names for v2 vs v3 node hash

Signed-off-by: Michael Puncel <mpuncel@squareup.com>

* PR comments: use package names to differentiate v2 and v3 cache types

Signed-off-by: Michael Puncel <mpuncel@squareup.com>

* put tests in v2 and v3 packages

Signed-off-by: Michael Puncel <mpuncel@squareup.com>

* rename v2 onStreamRequest callback, remove default implementation

Signed-off-by: Michael Puncel <mpuncel@squareup.com>

* rename onStreamRequest on DiscoveryServerCallbacks

Signed-off-by: Michael Puncel <mpuncel@squareup.com>

* DRY most of the DiscoveryServerCallbacks implementations in integration tests

Signed-off-by: Michael Puncel <mpuncel@squareup.com>

* make v3 transport integration tests also use v3 resource version

Note that there are not tests that cover a different transport version
from resource version in the interest of not doubling the number of
integration tests.

Signed-off-by: Michael Puncel <mpuncel@squareup.com>

* update README and add v2 -> v3 guide

Signed-off-by: Michael Puncel <mpuncel@squareup.com>

README.md

@@ -7,6 +7,12 @@ in [data-plane-api](https://github.com/envoyproxy/data-plane-api). It started li
 [go-control-plane](https://github.com/envoyproxy/go-control-plane), but building an idiomatic Java implementation is
 prioritized over exact interface parity with the Go implementation.
 
+Both v2 and v3 resources as well as transport versions are supported. Migrating
+to v3 is recommended as Envoy will drop v2 support at EOY 2020 (see
+[API_VERSIONING.md](https://github.com/envoyproxy/envoy/blob/4c6206865061591155d18b55972b4d626e1703dd/api/API_VERSIONING.md))
+
+See the (v2-to-v3 migration guide)[V2_TO_V3_GUIDE.md] for an exmplanation of migration paths.
+
 ### Requirements
 
 1. Java 8+

V2_TO_V3_GUIDE.md

@@ -0,0 +1,41 @@
+# Migrating from xDS v2 to v3
+
+To faciliate migrating from the v2 xDS APIs to v3, this repo supports both the
+v2 and v3 gRPC transports, with each transport supporting type URL rewriting of
+DiscoveryResponse to whatever version the client requests with
+api_resource_version.
+
+The migration requires care - for example, using v3-only fields too soon or trying to use
+deprecated v2 fields too late can cause Envoy to reject or improperly apply config.
+
+### Recommended Sequence
+
+This section assumes you have sufficient control over Envoy sidecar versions that you do
+not need to run v2 and v3 simultaneously for a long migration period.
+
+1. Make sure your oldest Envoy client supports final v2 message versions.
+2.
+ 1. Ensure your control plane is not using any deprecated v2 fields.
+ Deprecated v2 fields will cause errors when they are translated to v3
+ (because deprecated v2 fields are dropped in v3).
+ 2. Configure a V3DiscoveryServer alongside the V2DiscoveryServer in your
+ control plane. You can (and should) use the same (v2) Cache implementation
+ in both servers.
+3. Deploy all Envoy clients to switch to both the v3 transport_api_version and
+ resource_api_version in each respective xDS configs. As this happens, the V3DiscoveryServer
+ will be translating your v2 resources to v3 automatically, and the V2DiscoveryServer will
+ stop being used.
+4.
+ 1. Rewrite your control plane code to use v3 resources, which means using
+ V3SimpleCache (if you use SimpleCache). You may now start using v3-only
+ message fields if you choose.
+ 2. Drop the V2DiscoveryServer.
+
+### Alternative
+
+Another possible path to the one above is to switch to generating v3 in the
+control plane first (e.g. by using V3SimpleCache) and then deploying Envoy
+clients to use v3 transport and resource versions.
+
+This approach requires care to not use new V3-only fields until the client side
+upgrade is complete (or at least understand the consequences of doing so).

cache/src/main/java/io/envoyproxy/controlplane/cache/Cache.java

@@ -1,6 +1,5 @@
 package io.envoyproxy.controlplane.cache;
 
-import io.envoyproxy.envoy.api.v2.core.Node;
 import java.util.Collection;
 import javax.annotation.concurrent.ThreadSafe;
 
@@ -11,13 +10,13 @@
 public interface Cache<T> extends ConfigWatcher {
 
  /**
- * Returns all known {@link Node} groups.
+ * Returns all known groups.
  *
  */
  Collection<T> groups();
 
  /**
- * Returns the current {@link StatusInfo} for the given {@link Node} group.
+ * Returns the current {@link StatusInfo} for the given group.
  *
  * @param group the node group whose status is being fetched
  */

cache/src/main/java/io/envoyproxy/controlplane/cache/ConfigWatcher.java

@@ -1,6 +1,5 @@
 package io.envoyproxy.controlplane.cache;
 
-import io.envoyproxy.envoy.api.v2.DiscoveryRequest;
 import java.util.Set;
 import java.util.function.Consumer;
 import javax.annotation.concurrent.ThreadSafe;
@@ -25,7 +24,7 @@ public interface ConfigWatcher {
  */
  Watch createWatch(
  boolean ads,
- DiscoveryRequest request,
+ XdsRequest request,
  Set<String> knownResourceNames,
  Consumer<Response> responseConsumer,
  boolean hasClusterChanged);

cache/src/main/java/io/envoyproxy/controlplane/cache/NodeGroup.java

@@ -15,4 +15,11 @@ public interface NodeGroup<T> {
  * @param node identifier for the envoy instance that is requesting config
  */
  T hash(Node node);
+
+ /**
+ * Returns a consistent identifier of the given {@link io.envoyproxy.envoy.config.core.v3.Node}.
+ *
+ * @param node identifier for the envoy instance that is requesting config
+ */
+ T hash(io.envoyproxy.envoy.config.core.v3.Node node);
 }

cache/src/main/java/io/envoyproxy/controlplane/cache/Resources.java

@@ -1,6 +1,13 @@
 package io.envoyproxy.controlplane.cache;
 
 import static com.google.common.base.Strings.isNullOrEmpty;
+import static io.envoyproxy.controlplane.cache.Resources.ApiVersion.V2;
+import static io.envoyproxy.controlplane.cache.Resources.ApiVersion.V3;
+import static io.envoyproxy.controlplane.cache.Resources.ResourceType.CLUSTER;
+import static io.envoyproxy.controlplane.cache.Resources.ResourceType.ENDPOINT;
+import static io.envoyproxy.controlplane.cache.Resources.ResourceType.LISTENER;
+import static io.envoyproxy.controlplane.cache.Resources.ResourceType.ROUTE;
+import static io.envoyproxy.controlplane.cache.Resources.ResourceType.SECRET;
 import static io.envoyproxy.envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager.RouteSpecifierCase.RDS;
 
 import com.google.common.base.Preconditions;
@@ -30,33 +37,113 @@
 
 public class Resources {
 
+ /**
+ * Version-agnostic representation of a resource. This is useful when the version qualifier
+ * isn't needed.
+ */
+ public enum ResourceType {
+ CLUSTER,
+ ENDPOINT,
+ LISTENER,
+ ROUTE,
+ SECRET
+ }
+
+ public enum ApiVersion {
+ V2,
+ V3
+ }
+
  private static final Logger LOGGER = LoggerFactory.getLogger(Resources.class);
 
  static final String FILTER_ENVOY_ROUTER = "envoy.router";
  static final String FILTER_HTTP_CONNECTION_MANAGER = "envoy.http_connection_manager";
 
- private static final String TYPE_URL_PREFIX = "type.googleapis.com/envoy.api.v2.";
-
- public static final String CLUSTER_TYPE_URL = TYPE_URL_PREFIX + "Cluster";
- public static final String ENDPOINT_TYPE_URL = TYPE_URL_PREFIX + "ClusterLoadAssignment";
- public static final String LISTENER_TYPE_URL = TYPE_URL_PREFIX + "Listener";
- public static final String ROUTE_TYPE_URL = TYPE_URL_PREFIX + "RouteConfiguration";
- public static final String SECRET_TYPE_URL = TYPE_URL_PREFIX + "auth.Secret";
-
- public static final List<String> TYPE_URLS = ImmutableList.of(
- CLUSTER_TYPE_URL,
- ENDPOINT_TYPE_URL,
- LISTENER_TYPE_URL,
- ROUTE_TYPE_URL,
- SECRET_TYPE_URL);
-
- public static final Map<String, Class<? extends Message>> RESOURCE_TYPE_BY_URL = ImmutableMap.of(
- CLUSTER_TYPE_URL, Cluster.class,
- ENDPOINT_TYPE_URL, ClusterLoadAssignment.class,
- LISTENER_TYPE_URL, Listener.class,
- ROUTE_TYPE_URL, RouteConfiguration.class,
- SECRET_TYPE_URL, Secret.class
- );
+ public static class V2 {
+ private static final String TYPE_URL_PREFIX = "type.googleapis.com/envoy.api.v2.";
+ public static final String SECRET_TYPE_URL = TYPE_URL_PREFIX + "auth.Secret";
+ public static final String ROUTE_TYPE_URL = TYPE_URL_PREFIX + "RouteConfiguration";
+ public static final String LISTENER_TYPE_URL = TYPE_URL_PREFIX + "Listener";
+ public static final String ENDPOINT_TYPE_URL = TYPE_URL_PREFIX + "ClusterLoadAssignment";
+ public static final String CLUSTER_TYPE_URL = TYPE_URL_PREFIX + "Cluster";
+
+ public static final List<String> TYPE_URLS = ImmutableList.of(
+ CLUSTER_TYPE_URL,
+ ENDPOINT_TYPE_URL,
+ LISTENER_TYPE_URL,
+ ROUTE_TYPE_URL,
+ SECRET_TYPE_URL);
+ }
+
+ public static class V3 {
+
+ public static final String CLUSTER_TYPE_URL = "type.googleapis.com/envoy.config.cluster.v3"
+ + ".Cluster";
+ public static final String ENDPOINT_TYPE_URL = "type.googleapis.com/envoy.config.endpoint.v3"
+ + ".ClusterLoadAssignment";
+ public static final String LISTENER_TYPE_URL = "type.googleapis.com/envoy.config.listener.v3"
+ + ".Listener";
+ public static final String ROUTE_TYPE_URL = "type.googleapis.com/envoy.config.route.v3"
+ + ".RouteConfiguration";
+ public static final String SECRET_TYPE_URL = "type.googleapis.com/envoy.extensions"
+ + ".transport_sockets.tls.v3.Secret";
+
+ public static final List<String> TYPE_URLS = ImmutableList.of(
+ CLUSTER_TYPE_URL,
+ ENDPOINT_TYPE_URL,
+ LISTENER_TYPE_URL,
+ ROUTE_TYPE_URL,
+ SECRET_TYPE_URL);
+ }
+
+ public static final List<ResourceType> RESOURCE_TYPES_IN_ORDER = ImmutableList.of(
+ CLUSTER,
+ ENDPOINT,
+ LISTENER,
+ ROUTE,
+ SECRET);
+
+ public static final Map<String, String> V3_TYPE_URLS_TO_V2 = ImmutableMap.of(
+ Resources.V3.CLUSTER_TYPE_URL, Resources.V2.CLUSTER_TYPE_URL,
+ Resources.V3.ENDPOINT_TYPE_URL, Resources.V2.ENDPOINT_TYPE_URL,
+ Resources.V3.LISTENER_TYPE_URL, Resources.V2.LISTENER_TYPE_URL,
+ Resources.V3.ROUTE_TYPE_URL, Resources.V2.ROUTE_TYPE_URL,
+ Resources.V3.SECRET_TYPE_URL, Resources.V2.SECRET_TYPE_URL);
+
+ public static final Map<String, String> V2_TYPE_URLS_TO_V3 = ImmutableMap.of(
+ Resources.V2.CLUSTER_TYPE_URL, Resources.V3.CLUSTER_TYPE_URL,
+ Resources.V2.ENDPOINT_TYPE_URL, Resources.V3.ENDPOINT_TYPE_URL,
+ Resources.V2.LISTENER_TYPE_URL, Resources.V3.LISTENER_TYPE_URL,
+ Resources.V2.ROUTE_TYPE_URL, Resources.V3.ROUTE_TYPE_URL,
+ Resources.V2.SECRET_TYPE_URL, Resources.V3.SECRET_TYPE_URL);
+
+ public static final Map<String, ResourceType> TYPE_URLS_TO_RESOURCE_TYPE =
+ new ImmutableMap.Builder<String, ResourceType>()
+ .put(Resources.V3.CLUSTER_TYPE_URL, CLUSTER)
+ .put(Resources.V2.CLUSTER_TYPE_URL, CLUSTER)
+ .put(Resources.V3.ENDPOINT_TYPE_URL, ENDPOINT)
+ .put(Resources.V2.ENDPOINT_TYPE_URL, ENDPOINT)
+ .put(Resources.V3.LISTENER_TYPE_URL, LISTENER)
+ .put(Resources.V2.LISTENER_TYPE_URL, LISTENER)
+ .put(Resources.V3.ROUTE_TYPE_URL, ROUTE)
+ .put(Resources.V2.ROUTE_TYPE_URL, ROUTE)
+ .put(Resources.V3.SECRET_TYPE_URL, SECRET)
+ .put(Resources.V2.SECRET_TYPE_URL, SECRET)
+ .build();
+
+ public static final Map<String, Class<? extends Message>> RESOURCE_TYPE_BY_URL =
+ new ImmutableMap.Builder<String, Class<? extends Message>>()
+ .put(Resources.V2.CLUSTER_TYPE_URL, Cluster.class)
+ .put(Resources.V2.ENDPOINT_TYPE_URL, ClusterLoadAssignment.class)
+ .put(Resources.V2.LISTENER_TYPE_URL, Listener.class)
+ .put(Resources.V2.ROUTE_TYPE_URL, RouteConfiguration.class)
+ .put(Resources.V2.SECRET_TYPE_URL, Secret.class)
+ .put(Resources.V3.CLUSTER_TYPE_URL, io.envoyproxy.envoy.config.cluster.v3.Cluster.class)
+ .put(Resources.V3.ENDPOINT_TYPE_URL, io.envoyproxy.envoy.config.endpoint.v3.ClusterLoadAssignment.class)
+ .put(Resources.V3.LISTENER_TYPE_URL, io.envoyproxy.envoy.config.listener.v3.Listener.class)
+ .put(Resources.V3.ROUTE_TYPE_URL, io.envoyproxy.envoy.config.route.v3.RouteConfiguration.class)
+ .put(Resources.V3.SECRET_TYPE_URL, io.envoyproxy.envoy.extensions.transport_sockets.tls.v3.Secret.class)
+ .build();
 
  /**
  * Returns the name of the given resource message.
@@ -84,6 +171,26 @@ public static String getResourceName(Message resource) {
  return ((Secret) resource).getName();
  }
 
+ if (resource instanceof io.envoyproxy.envoy.config.cluster.v3.Cluster) {
+ return ((io.envoyproxy.envoy.config.cluster.v3.Cluster) resource).getName();
+ }
+
+ if (resource instanceof io.envoyproxy.envoy.config.endpoint.v3.ClusterLoadAssignment) {
+ return ((io.envoyproxy.envoy.config.endpoint.v3.ClusterLoadAssignment) resource).getClusterName();
+ }
+
+ if (resource instanceof io.envoyproxy.envoy.config.listener.v3.Listener) {
+ return ((io.envoyproxy.envoy.config.listener.v3.Listener) resource).getName();
+ }
+
+ if (resource instanceof io.envoyproxy.envoy.config.route.v3.RouteConfiguration) {
+ return ((io.envoyproxy.envoy.config.route.v3.RouteConfiguration) resource).getName();
+ }
+
+ if (resource instanceof io.envoyproxy.envoy.extensions.transport_sockets.tls.v3.Secret) {
+ return ((io.envoyproxy.envoy.extensions.transport_sockets.tls.v3.Secret) resource).getName();
+ }
+
  return "";
  }
 
@@ -133,6 +240,17 @@ public static Set<String> getResourceReferences(Collection<? extends Message> re
  refs.add(c.getName());
  }
  }
+ } else if (r instanceof io.envoyproxy.envoy.config.cluster.v3.Cluster) {
+ io.envoyproxy.envoy.config.cluster.v3.Cluster c = (io.envoyproxy.envoy.config.cluster.v3.Cluster) r;
+
+ // For EDS clusters, use the cluster name or the service name override.
+ if (c.getType() == io.envoyproxy.envoy.config.cluster.v3.Cluster.DiscoveryType.EDS) {
+ if (!isNullOrEmpty(c.getEdsClusterConfig().getServiceName())) {
+ refs.add(c.getEdsClusterConfig().getServiceName());
+ } else {
+ refs.add(c.getName());
+ }
+ }
  } else if (r instanceof Listener) {
  Listener l = (Listener) r;
 
@@ -144,17 +262,56 @@ public static Set<String> getResourceReferences(Collection<? extends Message> re
  }
 
  try {
- HttpConnectionManager.Builder config = HttpConnectionManager.newBuilder();
-
- // TODO: Filter#getConfig() is deprecated, migrate to use Filter#getTypedConfig().
- structAsMessage(filter.getConfig(), config);
+ HttpConnectionManager config;
+
+ if (filter.hasTypedConfig()) {
+ config = filter.getTypedConfig().unpack(HttpConnectionManager.class);
+ } else {
+ HttpConnectionManager.Builder builder = HttpConnectionManager.newBuilder();
+ structAsMessage(filter.getConfig(), builder);
+ config = builder.build();
+ }
 
  if (config.getRouteSpecifierCase() == RDS && !isNullOrEmpty(config.getRds().getRouteConfigName())) {
  refs.add(config.getRds().getRouteConfigName());
  }
  } catch (InvalidProtocolBufferException e) {
  LOGGER.error(
- "Failed to convert HTTP connection manager config struct into protobuf message for listener {}",
+ "Failed to convert v2 HTTP connection manager config struct into protobuf "
+ + "message for listener {}",
+ getResourceName(l),
+ e);
+ }
+ }
+ }
+ } else if (r instanceof io.envoyproxy.envoy.config.listener.v3.Listener) {
+
+ io.envoyproxy.envoy.config.listener.v3.Listener l =
+ (io.envoyproxy.envoy.config.listener.v3.Listener) r;
+
+ // Extract the route configuration names from the HTTP connection manager.
+ for (io.envoyproxy.envoy.config.listener.v3.FilterChain chain : l.getFilterChainsList()) {
+ for (io.envoyproxy.envoy.config.listener.v3.Filter filter : chain.getFiltersList()) {
+ if (!filter.getName().equals(FILTER_HTTP_CONNECTION_MANAGER)) {
+ continue;
+ }
+
+ try {
+ io.envoyproxy.envoy.extensions.filters.network
+ .http_connection_manager.v3.HttpConnectionManager config = filter
+ .getTypedConfig().unpack(
+ io.envoyproxy.envoy.extensions.filters.network
+ .http_connection_manager.v3.HttpConnectionManager.class);
+
+ if (config.getRouteSpecifierCase() == io.envoyproxy.envoy.extensions.filters.network
+ .http_connection_manager.v3.HttpConnectionManager.RouteSpecifierCase.RDS
+ && !isNullOrEmpty(config.getRds().getRouteConfigName())) {
+ refs.add(config.getRds().getRouteConfigName());
+ }
+ } catch (InvalidProtocolBufferException e) {
+ LOGGER.error(
+ "Failed to convert v3 HTTP connection manager config struct into protobuf "
+ + "message for listener {}",
  getResourceName(l),
  e);
  }
@@ -166,6 +323,19 @@ public static Set<String> getResourceReferences(Collection<? extends Message> re
  return refs.build();
  }
 
+ /**
+ * Returns the API version (v2 or v3) for a given type URL.
+ */
+ public static ApiVersion getResourceApiVersion(String typeUrl) {
+ if (Resources.V2.TYPE_URLS.contains(typeUrl)) {
+ return V2;
+ } else if (Resources.V3.TYPE_URLS.contains(typeUrl)) {
+ return V3;
+ }
+
+ throw new RuntimeException(String.format("Unsupported API version for type URL %s", typeUrl));
+ }
+
  private static void structAsMessage(Struct struct, Message.Builder messageBuilder)
  throws InvalidProtocolBufferException {