Merge branch 'main' into docs/capigw-v0.5-peered-services-support

.changelog/14679.txt

@@ -1,3 +1,3 @@
-```release-note:improvement
-dns: **(Enterprise Only)** All enterprise locality labels are now optional in DNS lookups. For example, service lookups support the following format: <tag>.]<service>.service[.<namespace>.ns][.<partition>.ap][.<datacenter>.dc]<domain>`. 
-```
+```release-note:improvement
+dns: **(Enterprise Only)** All enterprise locality labels are now optional in DNS lookups. For example, service lookups support the following format: `[<tag>.]<service>.service[.<namespace>.ns][.<partition>.ap][.<datacenter>.dc]<domain>`.
+```

.changelog/15370.txt

@@ -0,0 +1,3 @@
+```release-note:improvement
+auto-config: Relax the validation on auto-config JWT authorization to allow non-whitespace, non-quote characters in node names.
+```

.changelog/_3557.txt

@@ -1,3 +1,3 @@
 ```release-note:improvement
-dns/peering: **(Enterprise Only)** Support addresses in the formats <servicename>.virtual.<namespace>.ns.<partition>.ap.<peername>.peer.consul and <servicename>.virtual.<partition>.ap.<peername>.peer.consul. This longer form address that allows specifying `.peer` would need to be used for tproxy DNS requests made within non-default partitions for imported services.
-```
+dns/peering: **(Enterprise Only)** Support addresses in the formats `<servicename>.virtual.<namespace>.ns.<partition>.ap.<peername>.peer.consul` and `<servicename>.virtual.<partition>.ap.<peername>.peer.consul`. This longer form address that allows specifying `.peer` would need to be used for tproxy DNS requests made within non-default partitions for imported services.
+```

.circleci/config.yml

@@ -1053,6 +1053,8 @@ workflows:
                 - stable-website
                 - /^docs\/.*/
                 - /^ui\/.*/
+                - /^backport\/docs\/.*/
+                - /^backport\/ui\/.*/
       - check-generated-protobuf: *filter-ignore-non-go-branches
       - check-generated-deep-copy: *filter-ignore-non-go-branches
       - lint-enums: *filter-ignore-non-go-branches
@@ -1163,6 +1165,7 @@ workflows:
               only:
                 - main
                 - /^ui\/.*/
+                - /^backport\/ui\/.*/
       - workspace-tests:
           requires:
             - frontend-cache

CHANGELOG.md

@@ -1,3 +1,105 @@
+## 1.14.0 (November 15, 2022)
+
+BREAKING CHANGES:
+
+* config: Add new `ports.grpc_tls` configuration option.
+Introduce a new port to better separate TLS config from the existing `ports.grpc` config.
+The new `ports.grpc_tls` only supports TLS encrypted communication.
+The existing `ports.grpc` now only supports plain-text communication. [[GH-15339](https://github.com/hashicorp/consul/issues/15339)]
+* config: update 1.14 config defaults: Enable `peering` and `connect` by default. [[GH-15302](https://github.com/hashicorp/consul/issues/15302)]
+* config: update 1.14 config defaults: Set gRPC TLS port default value to 8503 [[GH-15302](https://github.com/hashicorp/consul/issues/15302)]
+* connect: Removes support for Envoy 1.20 [[GH-15093](https://github.com/hashicorp/consul/issues/15093)]
+* peering: Rename `PeerName` to `Peer` on prepared queries and exported services. [[GH-14854](https://github.com/hashicorp/consul/issues/14854)]
+* xds: Convert service mesh failover to use Envoy's aggregate clusters. This
+changes the names of some [Envoy dynamic HTTP metrics](https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats#dynamic-http-statistics). [[GH-14178](https://github.com/hashicorp/consul/issues/14178)]
+
+SECURITY:
+
+* Ensure that data imported from peers is filtered by ACLs at the UI Nodes/Services endpoints [CVE-2022-3920](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-3920) [[GH-15356](https://github.com/hashicorp/consul/issues/15356)]
+
+FEATURES:
+
+* DNS-proxy support via gRPC request. [[GH-14811](https://github.com/hashicorp/consul/issues/14811)]
+* cli: Add -node-name flag to redirect-traffic command to support running in environments without client agents. [[GH-14933](https://github.com/hashicorp/consul/issues/14933)]
+* cli: Add `-consul-dns-port` flag to the `consul connect redirect-traffic` command to allow forwarding DNS traffic to a specific Consul DNS port. [[GH-15050](https://github.com/hashicorp/consul/issues/15050)]
+* connect: Add Envoy connection balancing configuration fields. [[GH-14616](https://github.com/hashicorp/consul/issues/14616)]
+* grpc: Added metrics for external gRPC server. Added `server_type=internal|external` label to gRPC metrics. [[GH-14922](https://github.com/hashicorp/consul/issues/14922)]
+* http: Add new `get-or-empty` operation to the txn api. Refer to the [API docs](https://www.consul.io/api-docs/txn#kv-operations) for more information. [[GH-14474](https://github.com/hashicorp/consul/issues/14474)]
+* peering: Add mesh gateway local mode support for cluster peering. [[GH-14817](https://github.com/hashicorp/consul/issues/14817)]
+* peering: Add support for stale queries for trust bundle lookups [[GH-14724](https://github.com/hashicorp/consul/issues/14724)]
+* peering: Add support to failover to services running on cluster peers. [[GH-14396](https://github.com/hashicorp/consul/issues/14396)]
+* peering: Add support to redirect to services running on cluster peers with service resolvers. [[GH-14445](https://github.com/hashicorp/consul/issues/14445)]
+* peering: Ensure un-exported services get deleted even if the un-export happens while cluster peering replication is down. [[GH-14797](https://github.com/hashicorp/consul/issues/14797)]
+* peering: add support for routine peering control-plane traffic through mesh gateways [[GH-14981](https://github.com/hashicorp/consul/issues/14981)]
+* sdk: Configure `iptables` to forward DNS traffic to a specific DNS port. [[GH-15050](https://github.com/hashicorp/consul/issues/15050)]
+* telemetry: emit memberlist size metrics and broadcast queue depth metric. [[GH-14873](https://github.com/hashicorp/consul/issues/14873)]
+* ui: Added support for central config merging [[GH-14604](https://github.com/hashicorp/consul/issues/14604)]
+* ui: Create peerings detail page [[GH-14947](https://github.com/hashicorp/consul/issues/14947)]
+* ui: Detect a TokenSecretID cookie and passthrough to localStorage [[GH-14495](https://github.com/hashicorp/consul/issues/14495)]
+* ui: Display notice banner on nodes index page if synthetic nodes are being filtered. [[GH-14971](https://github.com/hashicorp/consul/issues/14971)]
+* ui: Filter agentless (synthetic) nodes from the nodes list page. [[GH-14970](https://github.com/hashicorp/consul/issues/14970)]
+* ui: Filter out node health checks on agentless service instances [[GH-14986](https://github.com/hashicorp/consul/issues/14986)]
+* ui: Remove node meta on service instances when using agentless and consolidate external-source labels on service instances page if they all match. [[GH-14921](https://github.com/hashicorp/consul/issues/14921)]
+* ui: Removed reference to node name on service instance page when using agentless [[GH-14903](https://github.com/hashicorp/consul/issues/14903)]
+* ui: Use withCredentials for all HTTP API requests [[GH-14343](https://github.com/hashicorp/consul/issues/14343)]
+* xds: servers will limit the number of concurrent xDS streams they can handle to balance the load across all servers [[GH-14397](https://github.com/hashicorp/consul/issues/14397)]
+
+IMPROVEMENTS:
+
+* peering: Add peering datacenter and partition to initial handshake. [[GH-14889](https://github.com/hashicorp/consul/issues/14889)]
+* xds: Added a rate limiter to the delivery of proxy config updates, to prevent updates to "global" resources such as wildcard intentions from overwhelming servers (see: `xds.update_max_per_second` config field) [[GH-14960](https://github.com/hashicorp/consul/issues/14960)]
+* xds: Removed a bottleneck in Envoy config generation, enabling a higher number of dataplanes per server [[GH-14934](https://github.com/hashicorp/consul/issues/14934)]
+* agent/hcp: add initial HashiCorp Cloud Platform integration [[GH-14723](https://github.com/hashicorp/consul/issues/14723)]
+* agent: Added configuration option cloud.scada_address. [[GH-14936](https://github.com/hashicorp/consul/issues/14936)]
+* api: Add filtering support to Catalog's List Services (v1/catalog/services) [[GH-11742](https://github.com/hashicorp/consul/issues/11742)]
+* api: Increase max number of operations inside a transaction for requests to /v1/txn (128) [[GH-14599](https://github.com/hashicorp/consul/issues/14599)]
+* auto-config: Relax the validation on auto-config JWT authorization to allow non-whitespace, non-quote characters in node names. [[GH-15370](https://github.com/hashicorp/consul/issues/15370)]
+* config-entry: Validate that service-resolver `Failover`s and `Redirect`s only
+specify `Partition` and `Namespace` on Consul Enterprise. This prevents scenarios
+where OSS Consul would save service-resolvers that require Consul Enterprise. [[GH-14162](https://github.com/hashicorp/consul/issues/14162)]
+* connect: Add Envoy 1.24.0 to support matrix [[GH-15093](https://github.com/hashicorp/consul/issues/15093)]
+* connect: Bump Envoy 1.20 to 1.20.7, 1.21 to 1.21.5 and 1.22 to 1.22.5 [[GH-14831](https://github.com/hashicorp/consul/issues/14831)]
+* connect: service-router destinations have gained a `RetryOn` field for specifying the conditions when Envoy should retry requests beyond specific status codes and generic connection failure which already exists. [[GH-12890](https://github.com/hashicorp/consul/issues/12890)]
+* dns/peering: **(Enterprise Only)** Support addresses in the formats `<servicename>.virtual.<namespace>.ns.<partition>.ap.<peername>.peer.consul` and `<servicename>.virtual.<partition>.ap.<peername>.peer.consul`. This longer form address that allows specifying `.peer` would need to be used for tproxy DNS requests made within non-default partitions for imported services.
+* dns: **(Enterprise Only)** All enterprise locality labels are now optional in DNS lookups. For example, service lookups support the following format: `[<tag>.]<service>.service[.<namespace>.ns][.<partition>.ap][.<datacenter>.dc]<domain>`. [[GH-14679](https://github.com/hashicorp/consul/issues/14679)]
+* integ test: fix flakiness due to test condition from retry app endoint [[GH-15233](https://github.com/hashicorp/consul/issues/15233)]
+* metrics: Service RPC calls less than 1ms are now emitted as a decimal number. [[GH-12905](https://github.com/hashicorp/consul/issues/12905)]
+* peering: adds an internally managed server certificate for automatic TLS between servers in peer clusters. [[GH-14556](https://github.com/hashicorp/consul/issues/14556)]
+* peering: require TLS for peering connections using server cert signed by Connect CA [[GH-14796](https://github.com/hashicorp/consul/issues/14796)]
+* peering: return information about the health of the peering when the leader is queried to read a peering. [[GH-14747](https://github.com/hashicorp/consul/issues/14747)]
+* raft: Allow nonVoter to initiate an election to avoid having an election infinite loop when a Voter is converted to NonVoter [[GH-14897](https://github.com/hashicorp/consul/issues/14897)]
+* raft: Cap maximum grpc wait time when heartbeating to heartbeatTimeout/2 [[GH-14897](https://github.com/hashicorp/consul/issues/14897)]
+* raft: Fix a race condition where the snapshot file is closed without being opened [[GH-14897](https://github.com/hashicorp/consul/issues/14897)]
+* telemetry: Added a `consul.xds.server.streamStart` metric to measure time taken to first generate xDS resources for an xDS stream. [[GH-14957](https://github.com/hashicorp/consul/issues/14957)]
+* ui: Improve guidance around topology visualisation [[GH-14527](https://github.com/hashicorp/consul/issues/14527)]
+* xds: Set `max_ejection_percent` on Envoy's outlier detection to 100% for peered services. [[GH-14373](https://github.com/hashicorp/consul/issues/14373)]
+
+BUG FIXES:
+
+* checks: Do not set interval as timeout value [[GH-14619](https://github.com/hashicorp/consul/issues/14619)]
+* checks: If set, use proxy address for automatically added sidecar check instead of service address. [[GH-14433](https://github.com/hashicorp/consul/issues/14433)]
+* cli: Fix Consul kv CLI 'GET' flags 'keys' and 'recurse' to be set together [[GH-13493](https://github.com/hashicorp/consul/issues/13493)]
+* connect: Fix issue where mesh-gateway settings were not properly inherited from configuration entries. [[GH-15186](https://github.com/hashicorp/consul/issues/15186)]
+* connect: fixed bug where endpoint updates for new xDS clusters could block for 15s before being sent to Envoy. [[GH-15083](https://github.com/hashicorp/consul/issues/15083)]
+* connect: strip port from DNS SANs for ingress gateway leaf certificate to avoid an invalid hostname error when using the Vault provider. [[GH-15320](https://github.com/hashicorp/consul/issues/15320)]
+* debug: fixed bug that caused consul debug CLI to error on ACL-disabled clusters [[GH-15155](https://github.com/hashicorp/consul/issues/15155)]
+* deps: update go-memdb, fixing goroutine leak [[GH-15010](https://github.com/hashicorp/consul/issues/15010)] [[GH-15068](https://github.com/hashicorp/consul/issues/15068)]
+* grpc: Merge proxy-defaults and service-defaults in GetEnvoyBootstrapParams response. [[GH-14869](https://github.com/hashicorp/consul/issues/14869)]
+* metrics: Add duplicate metrics that have only a single "consul_" prefix for all existing metrics with double ("consul_consul_") prefix, with the intent to standardize on single prefixes. [[GH-14475](https://github.com/hashicorp/consul/issues/14475)]
+* namespace: **(Enterprise Only)** Fixed a bug where a client may incorrectly log that namespaces were not enabled in the local datacenter
+* peering: Fix a bug that resulted in /v1/agent/metrics returning an error. [[GH-15178](https://github.com/hashicorp/consul/issues/15178)]
+* peering: fix nil pointer in calling handleUpdateService [[GH-15160](https://github.com/hashicorp/consul/issues/15160)]
+* peering: fix the error of wan address isn't taken by the peering token. [[GH-15065](https://github.com/hashicorp/consul/issues/15065)]
+* peering: when wan address is set, peering stream should use the wan address. [[GH-15108](https://github.com/hashicorp/consul/issues/15108)]
+* proxycfg(mesh-gateway): Fix issue where deregistered services are not removed from mesh-gateway clusters. [[GH-15272](https://github.com/hashicorp/consul/issues/15272)]
+* server: fix goroutine/memory leaks in the xDS subsystem (these were present regardless of whether or not xDS was in-use) [[GH-14916](https://github.com/hashicorp/consul/issues/14916)]
+* server: fixes the error trying to source proxy configuration for http checks, in case of proxies using consul-dataplane. [[GH-14924](https://github.com/hashicorp/consul/issues/14924)]
+* xds: Central service configuration (proxy-defaults and service-defaults) is now correctly applied to Consul Dataplane proxies [[GH-14962](https://github.com/hashicorp/consul/issues/14962)]
+
+NOTES:
+
+* deps: Upgrade to use Go 1.19.2 [[GH-15090](https://github.com/hashicorp/consul/issues/15090)]
+
 ## 1.13.3 (October 19, 2022)
 
 FEATURES:

agent/config/builder.go

@@ -1258,6 +1258,10 @@ func (b *builder) validate(rt RuntimeConfig) error {
 		b.warn("Node name %q will not be discoverable "+
 			"via DNS due to invalid characters. Valid characters include "+
 			"all alpha-numerics and dashes.", rt.NodeName)
+	case consul.InvalidNodeName.MatchString(rt.NodeName):
+		// todo(kyhavlov): Add stronger validation here for node names.
+		b.warn("Found invalid characters in node name %q - whitespace and quotes "+
+			"(', \", `) cannot be used with auto-config.", rt.NodeName)
 	case len(rt.NodeName) > dns.MaxLabelLength:
 		b.warn("Node name %q will not be discoverable "+
 			"via DNS due to it being too long. Valid lengths are between "+

agent/consul/auto_config_endpoint.go

@@ -57,6 +57,7 @@ type jwtAuthorizer struct {
 // This includes an extra single-quote character not specified in the grammar for safety in case it is later added.
 // https://github.com/hashicorp/go-bexpr/blob/v0.1.11/grammar/grammar.peg#L188-L191
 var invalidSegmentName = regexp.MustCompile("[`'\"\\s]+")
+var InvalidNodeName = invalidSegmentName
 
 func (a *jwtAuthorizer) Authorize(req *pbautoconf.AutoConfigRequest) (AutoConfigOptions, error) {
 	// perform basic JWT Authorization
@@ -70,7 +71,7 @@ func (a *jwtAuthorizer) Authorize(req *pbautoconf.AutoConfigRequest) (AutoConfig
 	// This is not the cleanest way to prevent this behavior. Ideally, the bexpr would allow us to
 	// inject a variable on the RHS for comparison as well, but it would be a complex change to implement
 	// that would likely break backwards-compatibility in certain circumstances.
-	if dns.InvalidNameRe.MatchString(req.Node) {
+	if InvalidNodeName.MatchString(req.Node) {
 		return AutoConfigOptions{}, fmt.Errorf("Invalid request field. %v = `%v`", "node", req.Node)
 	}
 	if invalidSegmentName.MatchString(req.Segment) {

api/go.mod

@@ -6,7 +6,7 @@ replace github.com/hashicorp/consul/sdk => ../sdk
 
 require (
 	github.com/google/go-cmp v0.5.7
-	github.com/hashicorp/consul/sdk v0.11.0
+	github.com/hashicorp/consul/sdk v0.12.0
 	github.com/hashicorp/go-cleanhttp v0.5.1
 	github.com/hashicorp/go-hclog v0.12.0
 	github.com/hashicorp/go-rootcerts v1.0.2

go.mod

@@ -31,9 +31,9 @@ require (
 	github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4
 	github.com/hashicorp/consul-awsauth v0.0.0-20220713182709-05ac1c5c2706
 	github.com/hashicorp/consul-net-rpc v0.0.0-20220307172752-3602954411b4
-	github.com/hashicorp/consul/api v1.15.3
-	github.com/hashicorp/consul/proto-public v0.1.0
-	github.com/hashicorp/consul/sdk v0.11.0
+	github.com/hashicorp/consul/api v1.16.0
+	github.com/hashicorp/consul/proto-public v0.2.0
+	github.com/hashicorp/consul/sdk v0.12.0
 	github.com/hashicorp/go-bexpr v0.1.2
 	github.com/hashicorp/go-checkpoint v0.5.0
 	github.com/hashicorp/go-cleanhttp v0.5.1

test/integration/connect/envoy/case-cross-peer-control-plane-mgw/alpha/verify.bats

@@ -38,12 +38,6 @@ load helpers
   assert_upstream_has_endpoints_in_status 127.0.0.1:19003 server.primary.peering HEALTHY 1
 }
 
-# Re-peering the clusters is a way to have alpha dial out through its own gateway
-# since we know it is configured with endpoints for primary from the first time they peered.
-@test "re-peer the two clusters together" {
-  create_peering primary alpha
-}
-
-@test "alpha servers made connection to primary servers via alpha gateway" {
+@test "dialer servers in alpha made connection to primary servers via alpha gateway" {
   assert_envoy_metric_at_least 127.0.0.1:19003 "cluster.server.primary.peering.*cx_total" 1
-}
+}

test/integration/consul-container/go.mod

@@ -6,8 +6,8 @@ require (
 	github.com/docker/docker v20.10.11+incompatible
 	github.com/docker/go-connections v0.4.0
 	github.com/hashicorp/consul v1.13.3
-	github.com/hashicorp/consul/api v1.15.3
-	github.com/hashicorp/consul/sdk v0.11.0
+	github.com/hashicorp/consul/api v1.16.0
+	github.com/hashicorp/consul/sdk v0.12.0
 	github.com/hashicorp/serf v0.10.1
 	github.com/itchyny/gojq v0.12.9
 	github.com/pkg/errors v0.9.1
@@ -67,7 +67,7 @@ require (
 	github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4 // indirect
 	github.com/hashicorp/consul-awsauth v0.0.0-20220713182709-05ac1c5c2706 // indirect
 	github.com/hashicorp/consul-net-rpc v0.0.0-20220307172752-3602954411b4 // indirect
-	github.com/hashicorp/consul/proto-public v0.1.0 // indirect
+	github.com/hashicorp/consul/proto-public v0.2.0 // indirect
 	github.com/hashicorp/errwrap v1.1.0 // indirect
 	github.com/hashicorp/go-bexpr v0.1.2 // indirect
 	github.com/hashicorp/go-cleanhttp v0.5.2 // indirect