IMPROVEMENTS:
- Helm:
- Add a
global.extraLabels
stanza to allow setting global Kubernetes labels for all components deployed by theconsul-k8s
Helm chart. [GH-1778]
- Add a
IMPROVEMENTS:
- Helm:
- CNI: Add
connectInject.cni.namespace
stanza which allows the CNI plugin resources to be deployed in a namespace other than the namespace that Consul is installed. [GH-1756]
- CNI: Add
BUG FIXES:
- Helm:
IMPROVEMENTS:
- Control Plane
- Bump Dockerfile base image for RedHat UBI
consul-k8s-control-plane
image toubi-minimal:9.1
. [GH-1725]
- Bump Dockerfile base image for RedHat UBI
- Helm
- Add fields
localConnectTimeoutMs
andlocalRequestTimeoutMs
to theServiceDefaults
CRD. [GH-1647]
- Add fields
BUG FIXES:
- Helm:
- Disable PodSecurityPolicies templating for
gossip-encryption-autogenerate
andpartition-init
whenglobal.enablePodSecurityPolicies
isfalse
. [GH-1693]
- Disable PodSecurityPolicies templating for
BUG FIXES:
- Control Plane
- Add discover binary to control-plane image [GH-1749]
- Helm:
- Don't pass in a CA file to the API Gateway controller when
externalServers.useSystemRoots
istrue
. [GH-1743]
- Don't pass in a CA file to the API Gateway controller when
BREAKING CHANGES:
- Admin Partitions (Consul Enterprise only): Remove the partition service. When configuring Admin Partitions, the expose-servers service should be used instead.
- Consul Dataplane:
- Consul client agents are no longer deployed by default, and Consul service mesh no longer uses Consul clients to operate. This change affects several main areas listed below. [GH-1552]
- A new component
consul-dataplane
is now injected as a sidecar-proxy instead of plain Envoy.consul-dataplane
manages the Envoy proxy process and proxies xDS requests from Envoy to Consul servers. - All services on the service mesh are now registered directly with the central catalog in Consul servers.
- All service-mesh consul-k8s components are configured to talk directly to Consul servers.
- Mesh, ingress, and terminating gateways are now registered centrally by the endpoints controller, similar to how service-mesh services are registered.
- CLI:
- Change default behavior of
consul-k8s install
to perform the installation when no answer is provided to the prompt. [GH-1673]
- Change default behavior of
- Helm:
-
Kubernetes-1.25 is now supported with the caveat that
global.enablePodSecurityPolicies
is not supported since PodSecurityPolicies have been removed in favor of PodSecurityStandards in Kubernetes-1.25. Full support for PodSecurityStandards will be added in a follow-on commit. [GH-1726] -
Support simplified default deployment values to allow for easier quick starts and testing:
- Set
connectInject.replicas
to 1 [GH-1702] - Set
meshGateway.affinity
to null andmeshGateway.replicas
to 1 [GH-1702] - Set
ingressGateways.defaults.affinity
to null andingressGateways.defaults.replicas
to 1 [GH-1702] - Set
terminatingGateways.defaults.affinity
to null andterminatingGateways.defaults.replicas
to 1 [GH-1702] - Set
server.replicas
to1
. Formerly, this defaulted to3
. [GH-1551]
- Set
-
client.enabled
now defaults tofalse
. Setting it totrue
will deploy client agents, however, none of the consul-k8s components will use clients for their operation. -
global.imageEnvoy
is no longer used for sidecar proxies, as well as mesh, terminating, and ingress gateways. -
externalServers.grpcPort
default is now8502
instead of8503
. -
meshGateway.service.enabled
value is removed. Mesh gateways now will always have a Kubernetes service as this is required to register them as a service with Consul. -
meshGateway.initCopyConsulContainer
,ingressGateways.initCopyConsulContainer
,terminatingGateways.initCopyConsulContainer
values are removed. -
connectInject.enabled
now defaults totrue
. [GH-1551] -
syncCatalog.consulNamespaces.mirroringK8S
now defaults totrue
. [GH-1601] -
connectInject.consulNamespaces.mirroringK8S
now defaults totrue
. [GH-1601] -
Remove
controller
section from the values file as the controller has now been merged into the connect-inject deployment. [GH-1697] -
Remove
global.consulSidecarContainer
from values file as there is no longer a consul sidecar. [GH-1635] -
Consul snapshot-agent now runs as a sidecar with Consul servers. [GH-1620]
This results in the following changes to Helm values:
- Move
client.snapshotAgent
values toserver.snapshotAgent
, with the exception of the following values:client.snaphostAgent.replicas
client.snaphostAgent.serviceAccount
- Remove
global.secretsBackend.vault.consulSnapshotAgentRole
value. You should now use theglobal.secretsBackend.vault.consulServerRole
for access to any Vault secrets.
- Move
-
Change
dns.enabled
anddns.enableRedirection
to default to the value ofconnectInject.transparentProxy.defaultEnabled
. Previously,dns.enabled
defaulted to the value ofglobal.enabled
anddns.enableRedirection
defaulted to the value tofalse
. [GH-1688] -
Remove
global.imageEnvoy
and replace withglobal.imageConsulDataplane
for running the sidecar proxy. -
Add
apiGateway.imageEnvoy
as for configuring the version of Envoy that the API Gateway uses. [GH-1698]
-
- Peering:
- Rename
PeerName
toPeer
in ExportedServices CRD. [GH-1596] - Remove support for customizing the server addresses in peering token generation. Instead, mesh gateways should be used to establish peering connections if the server pods are not directly reachable. [GH-1610]
- Require
global.tls.enabled
when peering is enabled. [GH-1610] - Require
meshGateway.enabled
when peering is enabled. [GH-1683]
- Rename
FEATURES:
- CLI:
- Consul Dataplane:
- Ingress Gateway
- Add support for MaxConnections, MaxConcurrentRequests, and MaxPendingRequests to Ingress Gateway CRD. [GH-1691]
- Peering:
- Support peering over mesh gateways.
- Add support for
PeerThroughMeshGateways
in Mesh CRD. [GH-1478]
- Add support for
- Support peering over mesh gateways.
IMPROVEMENTS:
- CLI
consul-k8s status
command will only show status of servers if they are expected to be present in the Kubernetes cluster. [GH-1603]- Update demo charts and CLI command to not presume tproxy when using HCP preset. Also, use the most recent version of hashicups. [GH-1657]
- Update minimum go version for project to 1.19 [GH-1633]
- Enable
consul-k8s uninstall
to delete custom resources when uninstalling Consul. This is done by default. [GH-1623]
- Control Plane
- Update minimum go version for project to 1.19 [GH-1633]
- Remove unneeded
agent:read
ACL permissions from mesh gateway policy. [GH-1255] - Support updating health checks on consul clients during an upgrade to agentless. [GH-1690]
- Remove unused curl from docker images [1624]
- Bump Dockerfile base image for RedHat UBI
consul-k8s-control-plane
image toubi-minimal:9.1
. [[GH-1725][hashicorp#1725]]
- Helm:
- Remove deprecated annotation
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
in theserver-service
template. [GH-1619] - Support
minAvailable
on connect injectorPodDisruptionBudget
. [GH-1557] - Add
tolerations
andnodeSelector
to Server ACL init jobs andnodeSelector
to Webhook cert manager. [GH-1581] - API Gateway: Add
tolerations
toapiGateway.managedGatewayClass
andapiGateway.controller
[GH-1650] - API Gateway: Create PodSecurityPolicy for controller when
global.enablePodSecurityPolicies=true
. [GH-1656] - API Gateway: Create PodSecurityPolicy and allow controller to bind it to ServiceAccounts that it creates for Gateway Deployments when
global.enablePodSecurityPolicies=true
. [GH-1672] - Deploy
expose-servers
service only when Admin Partitions(ENT) is enabled. [GH-1683] - Use a distroless image for
consul-dataplane
. [GH-1676] - The Envoy version is now 1.24.0 for
consul-dataplane
. [GH-1676] - Allow addition of extra labels to Connect Inject pods. [GH-1678]
- Add fields
localConnectTimeoutMs
andlocalRequestTimeoutMs
to theServiceDefaults
CRD. [GH-1647] - API Gateway: Enable API Gateways to directly connect to Consul servers when running in the agentless configuration. [GH-1694]
- Add
connectInject.consulNode.meta
to allow users to provide custom metadata to append to the NodeMeta [GH-1707] - Add
externalServers.skipServerWatch
which prevents consul-dataplane from consuming the server update stream. This is useful for situations where Consul servers are behind a load balancer. [GH-1686] - API Gateway: Allow controller to read MeshServices for use as a route backend. [GH-1574]
- API Gateway: Add support for using dynamic server discovery strings when running without agents. [GH-1732]
- Remove deprecated annotation
BUG FIXES:
- CLI
- Allow optional environment variables for use in the cloud preset to the CLI for cluster bootstrapping. [GH-1608]
- Configure
-tls-server-name
whenglobal.cloud.enabled=true
so that it matches the server certificate created via HCP [GH-1591] - Do not query clients in the status command since clients no longer exist. [GH-1573]
- Peering
- Add
peering:read
permissions to mesh gateway token to fix peering connections through the mesh gateways. [GH-1685]
- Add
- Helm:
- Disable PodSecurityPolicies in all templates when
global.enablePodSecurityPolicies
isfalse
. [GH-1693]
- Disable PodSecurityPolicies in all templates when
BREAKING CHANGES:
- Peering:
- Rename
PeerName
toPeer
in ExportedServices CRD. [GH-1596]
- Rename
FEATURES:
- Ingress Gateway
- Add support for MaxConnections, MaxConcurrentRequests, and MaxPendingRequests to Ingress Gateway CRD. [GH-1691]
IMPROVEMENTS:
- Helm:
- Add
tolerations
andnodeSelector
to Server ACL init jobs andnodeSelector
to Webhook cert manager. [GH-1581] - API Gateway: Allow controller to read MeshServices for use as a route backend. [GH-1574]
- API Gateway: Add
tolerations
toapiGateway.managedGatewayClass
andapiGateway.controller
[GH-1650] - API Gateway: Create PodSecurityPolicy for controller when
global.enablePodSecurityPolicies=true
. [GH-1656] - API Gateway: Create PodSecurityPolicy and allow controller to bind it to ServiceAccounts that it creates for Gateway Deployments when
global.enablePodSecurityPolicies=true
. [GH-1672]
- Add
FEATURES:
- CLI:
- Add support for tab autocompletion [GH-1437]
- Consul CNI Plugin
- Support for OpenShift and Multus CNI plugin [GH-1527]
BUG FIXES:
- Control plane
- Helm:
IMPROVEMENTS:
- Helm:
- API Gateway: Set primary datacenter flag when deploying controller into secondary datacenter with federation enabled [GH-1511]
- API Gateway: Allow controller to create and update Secrets for storing Consul CA cert alongside gateway Deployments [GH-1542]
- New parameter
EnforcingConsecutive5xx
which supports a configurable percent chance of automatic ejection of a host when a consecutive number of 5xx response codes are received [GH-1484]
- Control-plane:
- Support escaped commas in service tag annotations for pods which use
consul.hashicorp.com/connect-service-tags
orconsul.hashicorp.com/service-tags
. [GH-1532]
- Support escaped commas in service tag annotations for pods which use
FEATURES:
- MaxInboundConnections in service-defaults CRD
- Add support for MaxInboundConnections on the Service Defaults CRD. [GH-1437]
- Consul CNI Plugin
- CNI Plugin for Consul-k8s [GH-1465]
- Kubernetes 1.24 Support
- Add support for Kubernetes 1.24 where ServiceAccounts no longer have long-term JWT tokens. [GH-1431]
- Upgrade kubeVersion in helm chart to support Kubernetes 1.21+.
- Cluster Peering:
BREAKING CHANGES:
-
Kubernetes 1.24 Support
- Users deploying multiple services to the same Pod (multiport) on Kubernetes 1.24 must also deploy a Kubernetes Secret for each ServiceAccount associated with the Consul service. The name of the Secret must match the ServiceAccount name and be of type
kubernetes.io/service-account-token
[GH-1431] - Kubernetes 1.19 and 1.20 are no longer supported.
Example:
apiVersion: v1 kind: Secret metadata: name: svc1 annotations: kubernetes.io/service-account.name: svc1 type: kubernetes.io/service-account-token --- apiVersion: v1 kind: Secret metadata: name: svc2 annotations: kubernetes.io/service-account.name: svc2 type: kubernetes.io/service-account-token
- Users deploying multiple services to the same Pod (multiport) on Kubernetes 1.24 must also deploy a Kubernetes Secret for each ServiceAccount associated with the Consul service. The name of the Secret must match the ServiceAccount name and be of type
-
Control Plane
- Rename flag
server-address
totoken-server-address
in theinject-connect
subcommand to avoid overloading the context of theserver-address
flag. [GH-1426]
- Rename flag
IMPROVEMENTS:
- CLI:
- Display clusters by their short names rather than FQDNs for the
proxy read
command. [GH-1412] - Display a message when
proxy list
returns no results. [GH-1412] - Display a warning when a user passes a field and table filter combination to
proxy read
where the given field is not present in any of the output tables. [GH-1412] - Extend the timeout for
consul-k8s proxy read
to establish a connection from 5s to 10s. [GH-1442] - Expand the set of Envoy Listener Filters that may be parsed and output to the Listeners table. [GH-1442]
- Display clusters by their short names rather than FQDNs for the
- Helm:
- The default Envoy proxy image is now
envoyproxy/envoy:v1.23.1
. [GH-1473]
- The default Envoy proxy image is now
BUG FIXES:
- Helm
- API Gateway: Configure ACL auth for controller correctly when deployed in secondary datacenter with federation enabled [GH-1462]
- CLI
BUG FIXES:
- Helm
- Update the version of the
imageK8S
invalues.yaml
to the latest control-plane image. [GH-1355]
- Update the version of the
FEATURES:
- Transparent Proxy Egress
- Add support for Destinations on the Service Defaults CRD. [GH-1352]
- CLI:
- [Experimental] Cluster Peering:
- Add support for ACLs and TLS. [GH-1343] [GH-1366]
- Add support for Load Balancers or external addresses in front of Consul servers for peering stream.
- Support new expose-servers Kubernetes Service deployed by Helm chart to expose the Consul servers, and using the service address in the peering token. [GH-1378]
- Support non-default partitions by using
externalServers.hosts
as the server addresses in the peering token. [GH-1384] - Support arbitrary addresses as the server addresses in the peering token via
global.peering.tokenGeneration.source="static"
andglobal.peering.tokenGeneration.static=["sample-server-address:8502"]
. [GH-1392]
- Generate new peering token only on user-triggered events. [GH-1399]
IMPROVEMENTS:
- Helm
- Bump default Envoy version to 1.22.4. [GH-1413]
- Added support for Consul API Gateway to read ReferenceGrant custom resources. This will require either installing Consul API Gateway CRDs from the upcoming v0.4.0 release with
kubectl apply --kustomize "github.com/hashicorp/consul-api-gateway/config/crd?ref=v0.4.0"
or manually installing the ReferenceGrant CRD from the Gateway API v0.5 Experimental Channel when settingapiGateway.enabled=true
[GH-1299]
BUG FIXES:
- Helm
- Fix permissions in client-daemonset and server-statefulset when using extra-config volumes to prevent errors on OpenShift. [GH-1307]
IMPROVEMENTS:
- Control Plane
- Update alpine to 3.16 in the Docker image. [GH-1372]
FEATURES:
- [Experimental] Cluster Peering:
- Add support for secret watchers on the Peering Acceptor and Peering Dialer controllers. [GH-1284]
- Add support for version annotation on the Peering Acceptor and Peering Dialer controllers. [GH-1302]
- Add validation webhooks for the Peering Acceptor and Peering Dialer CRDs. [GH-1310]
- Add Conditions to the status of the Peering Acceptor and Peering Dialer CRDs. [GH-1335]
IMPROVEMENTS:
- Control Plane
- Added annotations
consul.hashicorp.com/prometheus-ca-file
,consul.hashicorp.com/prometheus-ca-path
,consul.hashicorp.com/prometheus-cert-file
, andconsul.hashicorp.com/prometheus-key-file
for configuring TLS scraping on Prometheus metrics endpoints for Envoy sidecars. To enable, set the cert and key file annotations along with one of the ca file/path annotations. [GH-1303] - Added annotations
consul.hashicorp.com/consul-sidecar-user-volume
andconsul.hashicorp.com/consul-sidecar-user-volume-mount
for attaching Volumes and VolumeMounts to the Envoy sidecar. Both should be JSON objects. [GH-1315] - Update minimum go version for project to 1.18 [GH-1292]
- Added annotations
- Helm
- CLI
- Update minimum go version for project to 1.18 [GH-1292]
BUG FIXES:
- Helm
- When using Openshift do not set securityContext in gossip-encryption-autogenerate job. [GH-1308]
- Control Plane
FEATURES:
-
[Experimental] Cluster Peering: Support Consul cluster peering, which allows service connectivity between two independent clusters. [GH-1273]
Enabling peering will deploy the peering controllers and PeeringAcceptor and PeeringDialer CRDs. The new CRDs are used to establish a peering connection between two clusters.
See the Cluster Peering on Kubernetes for full instructions.
Requirements:
- Consul 1.13+
global.peering.enabled=true
andconnectInject.enabled=true
must be set to enable peering.- Mesh gateways are required for service to service communication across peers, i.e
meshGateway.enabled=true
.
IMPROVEMENTS:
- Helm
- Enable the configuring of snapshot intervals in the client snapshot agent via
client.snapshotAgent.interval
. [GH-1235] - Enable configuring the pod topologySpreadConstraints for mesh, terminating, and ingress gateways. [GH-1257]
- Present Consul server CA chain when using Vault secrets backend. [GH-1251]
- API Gateway: Enable configuring of the new High Availability feature (requires Consul API Gateway v0.3.0+). [GH-1261]
- Enable the configuration of Envoy proxy concurrency via
connectInject.sidecarProxy.concurrency
which can be overridden at the pod level via the annotationconsul.hashicorp.com/consul-envoy-proxy-concurrency
. This PR also sets the default concurrency for envoy proxies to2
. [GH-1277] - Update Mesh CRD with Mesh HTTP Config. [GH-1282]
- Enable the configuring of snapshot intervals in the client snapshot agent via
- Control Plane
BUG FIXES:
- Helm
BREAKING CHANGES:
- Helm
- Using the Vault integration requires Consul 1.12.0+. [GH-1213], [GH-1218]
- The default Envoy proxy image is now
envoyproxy/envoy:v1.22.0
which is no longer alpine based. The default trust store location is no longer/etc/ssl/cert.pem
, please use/etc/ssl/certs/ca-certificates.crt
when configuring Terminating Gateway configuration entries for non-alpine based Envoy images. See [docs].
IMPROVEMENTS:
- Helm
- Enable the ability to
configure global.consulAPITimeout
to configure how long requests to the Consul API will wait to resolve before canceling. The default value is 5 seconds. [GH-1178]
- Enable the ability to
BUG FIXES:
- Security
- Bump golang.org/x/crypto and golang.org/x/text dependencies to address CVE-2022-27291 and CVE-2021-38561 respectively on both CLI and Control Plane. There's no known exposure within Consul on Kubernetes as the dependencies are not invoked. [GH-1189]
- Control Plane
- Endpoints Controller queuing up service registrations/deregistrations when request to agent on a terminated pod does not time out. This could result in pods not being registered and service instances not being deregistered. [GH-1178]
- Helm
- Update client-daemonset to include ca-cert volumeMount only when tls is enabled. [GH-1194]
- Update create-federation-secret-job to look up the automatically generated gossip encryption key by the right name when global.name is unset or set to something other than consul. [GH-1196]
- Add Admin Partitions support to Sync Catalog (Consul Enterprise only). [GH-1180]
- Correct webhook-cert-manager-clusterrole to utilize the web-cert-manager podsecuritypolicy rather than connect-injectors when
global.enablePodSecurityPolicies
is true. [GH-1202] - Enable Consul auto-reload-config only when Vault is enabled. [GH-1213]
- Revert TLS config to be compatible with Consul 1.11. [GH-1218]
BREAKING CHANGES:
- Helm
- Requires Consul 1.12.0+ as the Server statefulsets are now provisioned with Consul
-auto-reload-config
flag which monitors changes to specific Consul configuration properties and reloads itself when changes are detected. [GH-1135] - API Gateway: Re-use connectInject.consulNamespaces instead of requiring that apiGateway.consulNamespaces have the same value when ACLs are enabled. [GH-1169]
- Requires Consul 1.12.0+ as the Server statefulsets are now provisioned with Consul
FEATURES:
- Control Plane
- Add a
"consul.hashicorp.com/kubernetes-service"
annotation for pods to specify which Kubernetes service they want to use for registration when multiple services target the same pod. [GH-1150]
- Add a
BUG FIXES:
- CLI
- Fix issue where clusters not in the same namespace as their deployment name could not be upgraded. [GH-1115]
- Fix issue where the CLI was looking for secrets in namespaces other than the namespace targeted by the release. [GH-1156]
- Fix issue where the federation secret was not being found in certain configurations. [GH-1154]
- Control Plane
- Fix issue where upgrading a deployment from non-service mesh to service mesh would cause Pods to hang in init. [GH-1136]
- Helm
- Respect client nodeSelector, tolerations, and priorityClass when scheduling
create-federation-secret
Job. [GH-1108]
- Respect client nodeSelector, tolerations, and priorityClass when scheduling
IMPROVEMENTS:
- Control Plane
- Support new annotation for mounting connect-inject volume to other containers. [GH-1111]
- Helm
BREAKING CHANGES:
- Helm
- Minimum Kubernetes version supported is 1.19 and now matches what is stated in the
README.md
file. [GH-1049]
- Minimum Kubernetes version supported is 1.19 and now matches what is stated in the
- ACLs
- Support Terminating Gateway obtaining an ACL token using a k8s auth method. [GH-1102]
- Note: If you have updated a token with a new policy for a terminating gateway, this will not apply any more as ACL tokens will be ephemeral and are issued to the terminating gateways when the pod is created and destroyed when the pod is stopped. To achieve the same ACL permissions, you will need to assign the policy to the role for the terminating gateway, rather than the token.
- Support Mesh Gateway obtaining an ACL token using a k8s auth method. [GH-1102]
- Note: This is a breaking change if you are using a mesh gateway with mesh federation. To properly configure mesh federation with mesh gateways, you will need to configure the
global.federation.k8sAuthMethodHost
in secondary datacenters to point to the address of the Kubernetes API server of the secondary datacenter. This address must be reachable from the Consul servers in the primary datacenter.
- Note: This is a breaking change if you are using a mesh gateway with mesh federation. To properly configure mesh federation with mesh gateways, you will need to configure the
- General Note on old ACL Tokens: As of this release, ACL tokens no longer need to be stored as Kubernetes secrets. They will transparently be provisioned by the Kubernetes Auth Method when client and component pods are provisioned and will also be destroyed when client and component pods are destroyed. Old ACL tokens, however, will still exist as Kubernetes secrets and in Consul and will need to be identified and manually deleted.
- Support Terminating Gateway obtaining an ACL token using a k8s auth method. [GH-1102]
FEATURES:
- ACLs: Enable issuing ACL tokens via Consul login with a Kubernetes Auth Method and replace the need for storing ACL tokens as Kubernetes secrets.
- Support CRD controller obtaining an ACL token via using a k8s auth method. [GH-995]
- Support Connect Inject obtaining an ACL token via using a k8s auth method. [GH-1076]
- Support Sync Catalog obtaining an ACL token via using a k8s auth method. [GH-1081], [GHT-1077]
- Support API Gateway controller obtaining an ACL token via using a k8s auth method. [GH-1083]
- Support Snapshot Agent obtaining an ACL token via using a k8s auth method. [GH-1084]
- Support Mesh Gateway obtaining an ACL token via using a k8s auth method. [GH-1085]
- Support Ingress Gateway obtaining an ACL token via using a k8s auth method. [GH-1118]
- Support Terminating Gateway obtaining an ACL token via using a k8s auth method. [GH-1102]
- Support Consul Client obtaining an ACL token via using a k8s auth method. [GH-1093]
- Support issuing global ACL tokens via k8s auth method. [GH-1075]
IMPROVEMENTS:
- Control Plane
- Upgrade Docker image Alpine version from 3.14 to 3.15. [GH-1058
- Helm
- Vault
- Enable snapshot agent configuration to be retrieved from vault. [GH-1113]
- CLI
- Enable users to set up secondary clusters with existing federation secrets. [GH-1126]
BUG FIXES:
- Helm
- CLI
- Fix
install
andupgrade
commands for Windows. [GH-1139]
- Fix
BUG FIXES:
- Helm
- Support Envoy 1.20.2. [GH-1051]
FEATURES:
- Support WAN federation via Mesh Gateways with Vault as the secrets backend. [GH-1016,GH-1025,GH-1029,GH-1038]
- Note: To use WAN federation with ACLs and Vault, you will need to create a KV secret in Vault that will serve as the replication token with
a random UUID:
vault kv put secret/consul/replication key="$(uuidgen)"
. - You will need to then provide this secret to both the primary
and the secondary datacenters with
global.acls.replicationToken
values and allow theglobal.secretsBackend.vault.manageSystemACLsRole
Vault role to read it. In the primary datacenter, the Helm chart will create the replication token in Consul using the UUID as the secret ID of the token.
- Note: To use WAN federation with ACLs and Vault, you will need to create a KV secret in Vault that will serve as the replication token with
a random UUID:
- Connect: Support workaround for pods with multiple ports, by registering a Consul service and injecting an Envoy sidecar and init container per port. [GH-1012]
- Transparent proxying, metrics, and metrics merging are not supported for multi-port pods.
- Multi-port pods should specify annotations in the format, such that the service names and port names correspond with each other in the specified order, i.e.
web
service is listening on8080
,web-admin
service is listening on9090
.consul.hashicorp.com/connect-service': 'web,web-admin
consul.hashicorp.com/connect-service-port': '8080,9090
IMPROVEMENTS:
- Helm
- Vault: Allow passing arbitrary annotations to the vault agent. [GH-1015]
- Vault: Add support for customized IP and DNS SANs for server cert in Vault. [GH-1020]
- Vault: Add support for Enterprise License to be configured in Vault. [GH-1032]
- API Gateway: Allow Kubernetes namespace to Consul enterprise namespace mapping for deployed gateways and mesh services. [GH-1024]
BUG FIXES:
- API Gateway
- Fix issue where if the API gateway controller pods restarted, gateway pods would become disconnected from the secret discovery service. [GH-1007]
- Fix issue where the API gateway controller could not update existing Deployments or Services. [GH-1014]
- Fix issue where the API gateway controller lacked sufficient permissions to bind routes when ACLs were enabled. [GH-1018]
BREAKING CHANGES:
- Helm
- Rename fields of IngressGateway CRD to fix incorrect names (
gatewayTLSConfig
=>tls
,gatewayServiceTLSConfig
=>tls
,gatewayTLSSDSConfig
=>sds
). [GH-1017]
- Rename fields of IngressGateway CRD to fix incorrect names (
BREAKING CHANGES:
- Helm
FEATURES:
- Helm
- Support Envoy 1.20.1. [GH-958]
- Support Consul 1.11.2. [GH-976]
- Support Consul API Gateway Controller deployment through the Helm chart and provision an ACL token to for API Gateway via server-acl-init [GH-925]
IMPROVEMENTS:
- Helm
- Allow customization of
terminationGracePeriodSeconds
on the ingress gateways. [GH-947] - Support
ui.dashboardURLTemplates.service
value for setting dashboard URL templates. [GH-937] - Allow using dash-separated names for config entries when using
kubectl
. [GH-965] - Support Pod Security Policies with Vault integration. [GH-985]
- Rename Consul resources to remove resource kind suffixes from the resource names to standardize resource names across the Helm chart. [GH-993]
- Append
-client
to the Consul Daemonset name to standardize resource names across the Helm chart. [GH-1000]
- Allow customization of
- CLI
- Show a diff when upgrading a Consul installation on Kubernetes [GH-934]
- Control Plane
- Support the value
$POD_NAME
for the annotationconsul.hashicorp.com/service-meta-*
that will now be interpolated and set to the pod's name in the service's metadata. [GH-982] - Allow managing Consul sidecar resources via annotations. [GH-956]
- Support using a backslash to escape commas in
consul.hashicorp.com/service-tags
annotation. [GH-983] - Avoid making unnecessary calls to Consul in the endpoints controller to improve application startup time when Consul is down. [GH-779]
- Support the value
BUG FIXES:
- Helm
- Add
PodDisruptionBudget
Kind when checking for existing versions so thathelm template
can generate the right version. [GH-923]
- Add
- Control Plane
- Admin Partitions (Consul Enterprise only): Attach anonymous-policy to the anonymous token from non-default partitions to support DNS queries when the default partition is on a VM. [GH-966]
FEATURES:
- Helm
- CLI
- BETA Add
upgrade
command to modify Consul installation on Kubernetes. [GH-898]
- BETA Add
IMPROVEMENTS:
- Control Plane
BREAKING CHANGES:
- Control Plane
FEATURES:
-
Vault as a Secrets Backend: Add support for Vault as a secrets backend for Gossip Encryption, Server TLS certs and Service Mesh TLS certificates, removing the existing usage of Kubernetes Secrets for the respective secrets. [GH-904]
See the Consul Kubernetes and Vault documentation for full install instructions.
Requirements:
- Consul 1.11+
- Vault 1.9+ and Vault-K8s 0.14+ must be installed with the Vault Agent Injector enabled (
injector.enabled=true
) into the Kubernetes cluster that Consul is installed into. global.tls.enableAutoEncryption=true
is required for TLS support.- If TLS is enabled in Vault,
global.secretsBackend.vault.ca
must be provided and should reference a Kube secret which holds a copy of the Vault CA cert. - Add boolean metric to merged metrics response
consul_merged_service_metrics_success
to indicate if service metrics were scraped successfully. [GH-551]
-
Helm
- Rename
PartitionExports
CRD toExportedServices
. [GH-902]
- Rename
IMPROVEMENTS:
- CLI
- Pre-check in the
install
command to verify the correct license secret exists when using an enterprise Consul image. [GH-875]
- Pre-check in the
- Control Plane
- Helm Chart
BUG FIXES:
- Control Plane:
- Helm Chart
- Admin Partitions (Consul Enterprise only): Do not mount Consul CA certs to partition-init job if
externalServers.useSystemRoots
istrue
. [GH-885]
- Admin Partitions (Consul Enterprise only): Do not mount Consul CA certs to partition-init job if
BREAKING CHANGES:
-
Previously UI metrics would be enabled when
global.metrics=false
andui.metrics.enabled=-
. If you are no longer seeing UI metrics, setglobal.metrics=true
orui.metrics.enabled=true
. [GH-841] -
The
enterpriseLicense
section of the values file has been migrated from being under theserver
stanza to being under theglobal
stanza. Migrating the contents ofserver.enterpriseLicense
toglobal.enterpriseLicense
will ensure the license job works. [GH-856] -
Consul streaming is re-enabled by default. Streaming is broken when using multi-DC federation and Consul versions 1.10.0, 1.10.1, 1.10.2. If you are using those versions and multi-DC federation, you must upgrade to Consul >= 1.10.3 or set:
client: extraConfig: | {"use_streaming_backend": false}
[GH-851]
FEATURES:
- Helm Chart
- Add support for Consul services to utilize Consul DNS for service discovery. Set
dns.enableRedirection
to allow services to use Consul DNS via the Consul DNS Service. [GH-833]
- Add support for Consul services to utilize Consul DNS for service discovery. Set
- Control Plane
- Connect: Allow services using Connect to utilize Consul DNS to perform service discovery. [GH-833]
IMPROVEMENTS:
- Control Plane
- TLS: Support PKCS1 and PKCS8 private keys for Consul certificate authority. [GH-843]
- Connect: Log a warning when ACLs are enabled and the default service account is used. [GH-842]
- Update Service Router, Service Splitter and Ingress Gateway CRD with support for RequestHeaders and ResponseHeaders. [GH-863]
- Update Ingress Gateway CRD with partition support for the IngressService and TLS Config. [GH-863]
- CLI
- Delete jobs, cluster roles, and cluster role bindings on
uninstall
. [GH-820]
- Delete jobs, cluster roles, and cluster role bindings on
- Helm Chart
- Add
component
labels to all resources. [GH-840] - Update Consul version to 1.10.4. [GH-861]
- Update Service Router, Service Splitter and Ingress Gateway CRD with support for RequestHeaders and ResponseHeaders. [GH-863]
- Update Ingress Gateway CRD with partition support for the IngressService and TLS Config. [GH-863]
- Re-enable streaming for Consul clients. [GH-851]
- Add
BUG FIXES:
- Control Plane
- ACLs: Fix issue where if one or more servers fail to have their ACL tokens set on the initial run of server-acl-init then on subsequent re-runs of server-acl-init the tokens are never set. [GH-825]
- ACLs: Fix issue where if the number of Consul servers is increased, the new servers are never provisioned an ACL token. [GH-677]
- Fix issue where after a
helm upgrade
, users would seex509: certificate signed by unknown authority.
errors when modifying config entry resources. [GH-837]
- Helm Chart
- (Consul Enterprise only) Error on Helm install if a reserved name is used for the admin partition name or a Consul destination namespace for connect or catalog sync. [GH-846]
- Truncate Persistent Volume Claim names when namespace names are too long. [GH-799]
- Fix issue where UI metrics would be enabled when
global.metrics=false
andui.metrics.enabled=-
. [GH-841] - Populate the federation secret with the generated Gossip key when
global.gossipEncryption.autoGenerate
is set to true. [GH-854]
BREAKING CHANGES:
- Helm Chart
- The
kube-system
andlocal-path-storage
namespaces are now excluded from connect injection by default on Kubernetes versions >= 1.21. If you wish to enable injection on those namespaces, setconnectInject.namespaceSelector
tonull
. [GH-726]
- The
IMPROVEMENTS:
- Helm Chart
- Automatic retry for
gossip-encryption-autogenerate-job
on failure [GH-789] kube-system
andlocal-path-storage
namespaces are now excluded from connect injection by default on Kubernetes versions >= 1.21. This prevents deadlock issues whenkube-system
components go down and allows Kind to work without changing the failure policy of the mutating webhook. [GH-726]- Add support for services across Admin Partitions to communicate using mesh gateways. [GH-807]
- Documentation for the installation can be found here.
- Add support for PartitionExports CRD to enable cross-partition networking. [GH-802]
- Automatic retry for
- CLI
- Add
status
command. [GH-768] - Add
-verbose
,-v
flag to theconsul-k8s install
command, which outputs all logs emitted from the installation. By default, verbose is set tofalse
to hide logs that show resources are not ready. [GH-810] - Set
prometheus.enabled
to true and enable all metrics for Consul K8s when installing via thedemo
preset. [GH-809] - Set
controller.enabled
totrue
when installing via thedemo
preset. [GH818] - Set
global.gossipEncryption.autoGenerate
totrue
andglobal.tls.enableAutoEncrypt
totrue
when installing via thesecure
preset. [GH818]
- Add
- Control Plane
- Add support for partition-exports config entry as a Custom Resource Definition to help manage cross-partition networking. [GH-802]
FEATURES:
- Control Plane
- Helm Chart
IMPROVEMENTS:
- Control Plane
- Helm Chart
- Enable adding extra containers to server and client Pods. [GH-749]
- ACL support for Admin Partitions. (Consul Enterprise only)
BETA [GH-766]
- This feature now enabled ACL support for Admin Partitions. The server-acl-init job now creates a Partition token. This token can be used to bootstrap new partitions as well as manage ACLs in the non-default partitions.
- Partition to partition networking is disabled if ACLs are enabled.
- Documentation for the installation can be found here.
- CLI
BUG FIXES:
- Helm
- Fix consul-k8s image version in values file. [GH-732]
FEATURES:
- CLI
- The
consul-k8s
CLI enables users to deploy and operate Consul on Kubernetes.- Support
consul-k8s install
command. [GH-713]
- Support
- The
- Helm Chart
- Add support for Admin Partitions. (Consul Enterprise only)
ALPHA [GH-729]
-
This feature allows Consul to be deployed across multiple Kubernetes clusters while sharing a single set of Consul servers. The services on each cluster can be independently managed. This feature is an alpha feature. It requires:
- a flat pod and node network in order for inter-partition networking to work.
- TLS to be enabled.
- Consul Namespaces enabled.
Transparent Proxy is unsupported for cross partition communication.
-
- Add support for Admin Partitions. (Consul Enterprise only)
ALPHA [GH-729]
To enable Admin Partitions on the server cluster use the following config.
global:
enableConsulNamespaces: true
tls:
enabled: true
image: hashicorp/consul-enterprise:1.11.0-ent-alpha
adminPartitions:
enabled: true
server:
exposeGossipAndRPCPorts: true
enterpriseLicense:
secretName: license
secretKey: key
connectInject:
enabled: true
transparentProxy:
defaultEnabled: false
consulNamespaces:
mirroringK8S: true
controller:
enabled: true
Identify the LoadBalancer External IP of the partition-service
kubectl get svc consul-consul-partition-service -o json | jq -r '.status.loadBalancer.ingress[0].ip'
Migrate the TLS CA credentials from the server cluster to the workload clusters
kubectl get secret consul-consul-ca-key --context "server-context" -o yaml | kubectl apply --context "workload-context" -f -
kubectl get secret consul-consul-ca-cert --context "server-context" -o yaml | kubectl apply --context "workload-context" -f -
Configure the workload cluster using the following config.
global:
enabled: false
enableConsulNamespaces: true
image: hashicorp/consul-enterprise:1.11.0-ent-alpha
adminPartitions:
enabled: true
name: "alpha" # Name of Admin Partition
tls:
enabled: true
caCert:
secretName: consul-consul-ca-cert
secretKey: tls.crt
caKey:
secretName: consul-consul-ca-key
secretKey: tls.key
server:
enterpriseLicense:
secretName: license
secretKey: key
externalServers:
enabled: true
hosts: [ "loadbalancer IP" ] # external IP of partition service LB
tlsServerName: server.dc1.consul
client:
enabled: true
exposeGossipPorts: true
join: [ "loadbalancer IP" ] # external IP of partition service LB
connectInject:
enabled: true
consulNamespaces:
mirroringK8S: true
controller:
enabled: true
This should lead to the workload cluster having only Consul agents that connect with the Consul server. Services in this cluster behave like independent services. They can be configured to communicate with services in other partitions by configuring the upstream configuration on the individual services.
- Control Plane
- Add support for Admin Partitions. (Consul Enterprise only) **
ALPHA** [GH-729]
- Add Partition-Init job that runs in Kubernetes clusters that do not have servers running to provision Admin Partitions.
- Update endpoints-controller, config-entry controller and config entries to add partition config to them.
- Add support for Admin Partitions. (Consul Enterprise only) **
ALPHA** [GH-729]
IMPROVEMENTS:
- Helm Chart
- Add ability to specify port for ui service. [GH-604]
- Use
policy/v1
for Consul serverPodDisruptionBudget
if supported. [GH-606] - Add readiness, liveness and startup probes to the connect inject deployment. [GH-626][GH-701]
- Add support for setting container security contexts on client and server Pods. [GH-620]
- Update Envoy image to 1.18.4 [GH-699]
- Add configuration for webhook-cert-manager tolerations [GH-712]
- Update default Consul version to 1.10.2 [GH-718]
- Control Plane
- Add health endpoint to the connect inject webhook that will be healthy when webhook certs are present and not empty. [GH-626]
- Catalog Sync: Fix issue registering NodePort services with wrong IPs when a node has multiple IP addresses. [GH-619]
- Allow registering the same service in multiple namespaces. [GH-697]
BUG FIXES:
- Helm Chart
BREAKING CHANGES:
- The consul-k8s repository has been merged with consul-helm and now contains the
consul-k8s-control-plane
binary (previously namedconsul-k8s
) and the Helm chart to deploy Consul on Kubernetes. The docker image previously namedhashicorp/consul-k8s
has been renamed tohashicorp/consul-k8s-control-plane
. The binary and Helm chart will be released together with the same version. NOTE: If you install Consul through the Helm chart and are not customizing theglobal.imageK8S
value then this will not be a breaking change. [GH-589]- Helm chart v0.33.0+ will support the corresponding
consul-k8s-control-plane
image with the same version only. For example Helm chart 0.33.0 will only be supported to work with the default valueglobal.imageK8S
:hashicorp/consul-k8s-control-plane:0.33.0
. - The control-plane binary has been renamed from
consul-k8s
toconsul-k8s-control-plane
and is now invoked asconsul-k8s-control-plane
in the Helm chart. The first version of this newly renamed binary will be 0.33.0. - The Go module
github.com/hashicorp/consul-k8s
has been named togithub.com/hashicorp/consul-k8s/control-plane
. - The Helm chart is located under
consul-k8s/charts/consul
. - The control-plane source code is located under
consul-k8s/control-plane
.
- Helm chart v0.33.0+ will support the corresponding
- Minimum Kubernetes versions supported is 1.17+ and now matches what is stated in the
README.md
file. [GH-1053]
IMPROVEMENTS:
- Control Plane
- Helm Chart
- Substitute
HOST_IP/POD_IP/HOSTNAME
variables inserver.extraConfig
andclient.extraConfig
so they are passed in to server/client config already evaluated at runtime. [GH-1042] - Set failurePolicy to Fail for connectInject mutating webhook so that pods fail to schedule when the webhook is offline. This can be controlled via
connectInject.failurePolicy
. [GH-1024] - Allow setting global.logLevel and global.logJSON and propogate this to all consul-k8s commands. [GH-980]
- Allow setting
connectInject.replicas
to control number of replicas of webhook injector. [GH-1029] - Add the ability to manually specify a k8s secret containing server-cert via the value
server.serverCert.secretName
. [GH-1024] - Allow setting
ui.pathType
for providers that do not support the default pathType "Prefix". [GH-1012] - Allow setting
client.nodeMeta
to specify arbitrary key-value pairs to associate with the node. [GH-728]
- Substitute
BUG FIXES:
- Control Plane
- Connect: Use
AdmissionregistrationV1
instead ofAdmissionregistrationV1beta1
API as it was deprecated in k8s 1.16. [GH-558] - Connect: Fix bug where environment variables
<NAME>_CONNECT_SERVICE_HOST
and<NAME>_CONNECT_SERVICE_PORT
weren't being set when the upstream annotation was used. [GH-549] - Connect: Fix a bug with leaving around ACL tokens after a service has been deregistered. Note that this will not clean up existing leftover ACL tokens. [GH-540][GH-599]
- CRDs: Fix ProxyDefaults and ServiceDefaults resources not syncing with Consul < 1.10.0 [GH-1023]
- Connect: Skip service registration for duplicate services only on Kubernetes. [GH-581]
- Connect: redirect-traffic command passes ACL token when ACLs are enabled. [GH-576]
- Connect: Use
FEATURES:
-
Connect: Support Transparent Proxy. [GH-481] This feature enables users to use KubeDNS to reach other services within the Consul Service Mesh, as well as enforces the inbound and outbound traffic to go through the Envoy proxy.
Using transparent proxy for your service mesh applications means:
- Proxy service registrations will set
mode
totransparent
in the proxy configuration so that Consul can configure the Envoy proxy to have an inbound and outbound listener. - Both proxy and service registrations will include the cluster IP and service port of the Kubernetes service as tagged addresses so that Consul can configure Envoy to route traffic based on that IP and port.
- The
consul-connect-inject-init
container will runconsul connect redirect-traffic
command, which will apply rules (via iptables) to redirect inbound and outbound traffic to the proxy. To run this command theconsul-connect-inject-init
requires running as root with capabilityNET_ADMIN
.
This feature includes the following changes:
- Add new
-enable-transparent-proxy
flag to theinject-connect
command. Whentrue
, transparent proxy will be used for all services on the Consul Service Mesh within a Kubernetes cluster. This flag defaults totrue
. - Add new
consul.hashicorp.com/transparent-proxy
pod annotation to allow enabling and disabling transparent proxy for individual services.
- Proxy service registrations will set
-
CRDs: Add CRD for MeshConfigEntry. Supported in Consul 1.10+ [GH-513]
-
Connect: Overwrite Kubernetes HTTP readiness and/or liveness probes to point to Envoy proxy when transparent proxy is enabled. [GH-517]
-
Connect: Allow exclusion of inbound ports, outbound ports and CIDRs, and additional user IDs when Transparent Proxy is enabled. [GH-506]
The following annotations are supported:
consul.hashicorp.com/transparent-proxy-exclude-inbound-ports
- Comma-separated list of inbound ports to exclude.consul.hashicorp.com/transparent-proxy-exclude-outbound-ports
- Comma-separated list of outbound ports to exclude.consul.hashicorp.com/transparent-proxy-exclude-outbound-cidrs
- Comma-separated list of IPs or CIDRs to exclude.consul.hashicorp.com/transparent-proxy-exclude-uids
- Comma-separated list of Linux user IDs to exclude.
-
Connect: Add the ability to set default tproxy mode at namespace level via label. [GH-501]
- Setting the annotation
consul.hashicorp.com/transparent-proxy
totrue/false
will define whether tproxy is enabled/disabled for the pod. - Setting the label
consul.hashicorp.com/transparent-proxy
totrue/false
on a namespace will define the default behavior for pods in that namespace, which do not also have the annotation set. - The default tproxy behavior will be defined by the value of
-enable-transparent-proxy
flag to theconsul-k8s inject-connect
command. It can be overridden in a namespace by the the label on the namespace or for a pod using the annotation on the pod.
- Setting the annotation
-
Connect: support upgrades for services deployed before endpoints controller to upgrade to a version of consul-k8s with endpoints controller. [GH-509]
-
Connect: A new command
consul-k8s connect-init
has been added. It replaces the existing init-container logic for ACL login and Envoy bootstrapping and introduces a polling wait for service registration, seeEndpoints Controller
for more information. [GH-446], [GH-452], [GH-459] -
Connect: A new controller
Endpoints Controller
has been added which is responsible for managing service endpoints and service registration. When a Kubernetes service references a deployed connect-injected pod, the endpoints controller will be responsible for managing the lifecycle of the connect-injected deployment. [GH-455], [GH-467], [GH-470], [GH-475]- This includes:
- service registration and deregistration, formerly managed by the
consul-connect-inject-init
. - monitoring health checks, formerly managed by
healthchecks-controller
. - re-registering services in the events of consul agent failures, formerly managed by
consul-sidecar
.
- service registration and deregistration, formerly managed by the
- The endpoints controller replaces the health checks controller while preserving existing functionality. [GH-472]
- The endpoints controller replaces the cleanup controller while preserving existing functionality. [GH-476], [GH-454]
- Merged metrics configuration support is now partially managed by the endpoints controller. [GH-469]
- This includes:
IMPROVEMENTS:
- Connect: skip service registration when a service with the same name but in a different Kubernetes namespace is found and Consul namespaces are not enabled. [GH-527]
- Connect: Leader election support for connect-inject deployment. [GH-479]
- Connect: the
consul-connect-inject-init
container has been split into two init containers. [GH-441] Connect: Connect webhook no longer generates its own certificates and relies on them being provided as files on the disk. [GH-454]] - CRDs: Update
ServiceDefaults
withMode
,TransparentProxy
,DialedDirectly
andUpstreamConfigs
fields. Note:Mode
andTransparentProxy
should not be set using this CRD but via annotations. [GH-502], [GH-485], [GH-533] - CRDs: Update
ProxyDefaults
withMode
,DialedDirectly
andTransparentProxy
fields. Note:Mode
andTransparentProxy
should not be set using the CRD but via annotations. [GH-505], [GH-485], [GH-533] - CRDs: update the CRD versions from v1beta1 to v1. [GH-464]
- Delete secrets created by webhook-cert-manager when the deployment is deleted. [GH-530]
BUG FIXES:
- CRDs: Update the type of connectTimeout and TTL in ServiceResolver and ServiceRouter from time.Duration to metav1.Duration. This allows a user to set these values as a duration string on the resource. Existing resources that had set a specific integer duration will continue to function with a duration with 'n' nanoseconds, 'n' being the set value.
- CRDs: Fix a bug where the
config
field inProxyDefaults
CR failed syncing to Consul becauseapiextensions.k8s.io/v1
requires CRD spec to have structured schema. [GH-495] - CRDs: make
lastSyncedTime
a pointer to prevent setting last synced time Reconcile errors. [GH-466]
BREAKING CHANGES:
-
Connect: Add a security context to the init copy container and the envoy sidecar and ensure they do not run as root. If a pod container shares the same
runAsUser
(5995) as Envoy an error is returned. [GH-493] -
Connect: Kubernetes Services are required for all Consul Service Mesh applications. The Kubernetes service name will be used as the service name to register with Consul unless the annotation
consul.hashicorp.com/connect-service
is provided to the deployment/pod to override this. If using ACLs, the ServiceAccountName must match the service name used with Consul.Note: if you're already using a Kubernetes service, no changes required.
Example Service:
--- apiVersion: v1 kind: Service metadata: name: sample-app spec: selector: app: sample-app ports: - port: 80 targetPort: 9090 --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: sample-app name: sample-app spec: replicas: 1 selector: matchLabels: app: sample-app template: metadata: annotations: 'consul.hashicorp.com/connect-inject': 'true' labels: app: sample-app spec: containers: - name: sample-app image: sample-app:0.1.0 ports: - containerPort: 9090
-
Connect:
consul.hashicorp.com/connect-sync-period
annotation is no longer supported. This annotation used to configure the sync period of theconsul-sidecar
(akalifecycle-sidecar
). Since we no longer inject theconsul-sidecar
to keep services registered in Consul, this annotation has been removed. [GH-467] -
Connect: transparent proxy feature enabled by default. This may break existing deployments. Please see details of the feature.
IMPROVEMENTS:
- Connect: Overwrite Kubernetes HTTP readiness and/or liveness probes to point to Envoy proxy when transparent proxy is enabled. [GH-517]
- Connect: Don't set security context for the Envoy proxy when on OpenShift and transparent proxy is disabled. [GH-521]
- Connect:
consul-connect-inject-init
run withprivileged: true
when transparent proxy is enabled. [GH-524]
BUG FIXES:
- Connect: Process every Address in an Endpoints object before returning an error. This ensures an address that isn't reconciled successfully doesn't prevent the remaining addresses from getting reconciled. [GH-519]
BREAKING CHANGES:
- Connect: Add a security context to the init copy container and the envoy sidecar and ensure they
do not run as root. If a pod container shares the same
runAsUser
(5995) as Envoy an error is returned on scheduling. [GH-493]
IMPROVEMENTS:
-
CRDs: Update ServiceDefaults with Mode, TransparentProxy and UpstreamConfigs fields. Note: Mode and TransparentProxy should not be set using this CRD but via annotations. [GH-502], [GH-485]
-
CRDs: Update ProxyDefaults with Mode and TransparentProxy fields. Note: Mode and TransparentProxy should not be set using the CRD but via annotations. [GH-505], [GH-485]
-
CRDs: Add CRD for MeshConfigEntry. Supported in Consul 1.10+ [GH-513]
-
Connect: No longer set multiple tagged addresses in Consul when k8s service has multiple ports and Transparent Proxy is enabled. [GH-511]
-
Connect: Allow exclusion of inbound ports, outbound ports and CIDRs, and additional user IDs when Transparent Proxy is enabled. [GH-506]
The following annotations are supported:
consul.hashicorp.com/transparent-proxy-exclude-inbound-ports
- Comma-separated list of inbound ports to exclude.consul.hashicorp.com/transparent-proxy-exclude-outbound-ports
- Comma-separated list of outbound ports to exclude.consul.hashicorp.com/transparent-proxy-exclude-outbound-cidrs
- Comma-separated list of IPs or CIDRs to exclude.consul.hashicorp.com/transparent-proxy-exclude-uids
- Comma-separated list of Linux user IDs to exclude.
-
Connect: Add the ability to set default tproxy mode at namespace level via label. [GH-501]
- Setting the annotation
consul.hashicorp.com/transparent-proxy
totrue/false
will define whether tproxy is enabled/disabled for the pod. - Setting the label
consul.hashicorp.com/transparent-proxy
totrue/false
on a namespace will define the default behavior for pods in that namespace, which do not also have the annotation set. - The default tproxy behavior will be defined by the value of
-enable-transparent-proxy
flag to theconsul-k8s inject-connect
command. It can be overridden in a namespace by the the label on the namespace or for a pod using the annotation on the pod.
- Setting the annotation
-
Connect: support upgrades for services deployed before endpoints controller to upgrade to a version of consul-k8s with endpoints controller. [GH-509]
-
Connect: add additional logging to the endpoints controller and connect-init command to help the user debug if pods arent starting right away. [GH-514]
BUG FIXES:
- Connect: Use
runAsNonRoot: false
for connect-init's container when tproxy is enabled. [GH-493] - CRDs: Fix a bug where the
config
field inProxyDefaults
CR was not synced to Consul becauseapiextensions.k8s.io/v1
requires CRD spec to have structured schema. [GH-495] - Connect: Fix a bug where health status in Consul is updated incorrectly due to stale pod information in cache. [GH-503]
BREAKING CHANGES:
-
Connect: Kubernetes Services are now required for all Consul Service Mesh applications. The Kubernetes service name will be used as the service name to register with Consul unless the annotation
consul.hashicorp.com/connect-service
is provided to the deployment/pod to override this. If using ACLs, the ServiceAccountName must match the service name used with Consul.Note: if you're already using a Kubernetes service, no changes are required.
Example Service:
--- apiVersion: v1 kind: Service metadata: name: sample-app spec: selector: app: sample-app ports: - port: 80 targetPort: 9090 --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: sample-app name: sample-app spec: replicas: 1 selector: matchLabels: app: sample-app template: metadata: annotations: 'consul.hashicorp.com/connect-inject': 'true' labels: app: sample-app spec: containers: - name: sample-app image: sample-app:0.1.0 ports: - containerPort: 9090
-
Connect:
consul.hashicorp.com/connect-sync-period
annotation is no longer supported. This annotation was used to configure the sync period of theconsul-sidecar
(akalifecycle-sidecar
). Since we no longer inject theconsul-sidecar
to keep services registered in Consul, this annotation is now meaningless. [GH-467] -
Connect: transparent proxy feature is enabled by default. This may break existing deployments. Please see details of the feature below.
FEATURES:
-
Connect: Support Transparent Proxy. [GH-481] This feature enables users to use KubeDNS to reach other services within the Consul Service Mesh, as well as enforces the inbound and outbound traffic to go through the Envoy proxy. Using transparent proxy for your service mesh applications means:
- Proxy service registrations will set
mode
totransparent
in the proxy configuration so that Consul can configure the Envoy proxy to have an inbound and outbound listener. - Both proxy and service registrations will include the cluster IP and service port of the Kubernetes service as tagged addresses so that Consul can configure Envoy to route traffic based on that IP and port.
- The
consul-connect-inject-init
container will runconsul connect redirect-traffic
command, which will apply rules (via iptables) to redirect inbound and outbound traffic to the proxy. To run this command theconsul-connect-inject-init
requires running as root with capabilityNET_ADMIN
.
Note: this feature is currently in beta.
This feature includes the following changes:
- Add new
-enable-transparent-proxy
flag to theinject-connect
command. Whentrue
, transparent proxy will be used for all services on the Consul Service Mesh within a Kubernetes cluster. This flag defaults totrue
. - Add new
consul.hashicorp.com/transparent-proxy
pod annotation to allow enabling and disabling transparent proxy for individual services.
- Proxy service registrations will set
IMPROVEMENTS:
- CRDs: update the CRD versions from v1beta1 to v1. [GH-464]
- Connect: the
consul-connect-inject-init
container has been split into two init containers. [GH-441] - Connect: A new internal command
consul-k8s connect-init
has been added. It replaces the existing init container logic for ACL login and Envoy bootstrapping and introduces a polling wait for service registration, seeEndpoints Controller
for more information. [GH-446], [GH-452], [GH-459] - Connect: A new controller
Endpoints Controller
has been added which is responsible for managing service endpoints and service registration. When a Kubernetes service referencing a connect-injected pod is deployed, the endpoints controller will be responsible for managing the lifecycle of the connect-injected deployment. [GH-455], [GH-467], [GH-470], [GH-475]-
This includes:
- service registration and deregistration, formerly managed by the
consul-connect-inject-init
. - monitoring health checks, formerly managed by
healthchecks-controller
. - re-registering services in the events of consul agent failures, formerly managed by
consul-sidecar
.
- service registration and deregistration, formerly managed by the
-
The endpoints controller replaces the health checks controller while preserving existing functionality. [GH-472]
-
The endpoints controller replaces the cleanup controller while preserving existing functionality. [GH-476], [GH-454]
-
Merged metrics configuration support is now partially managed by the endpoints controller. [GH-469]
-
- Connect: Leader election support for connect webhook and controller deployment. [GH-479]
- Connect: Connect webhook no longer generates its own certificates and relies on them being provided as files on the disk. [GH-454]]
- Connect: Connect pods and their Envoy sidecars no longer have a preStop hook as service deregistration is managed by the endpoints controller. [GH-467]
BUG FIXES:
- CRDs: make
lastSyncedTime
a pointer to prevent setting last synced time Reconcile errors. [GH-466]
FEATURES:
-
Metrics: add metrics configuration to inject-connect and metrics-merging capability to consul-sidecar. When metrics and metrics merging are enabled, the consul-sidecar will expose an endpoint that merges the app and proxy metrics.
The flags
-merged-metrics-port
,-service-metrics-port
and-service-metrics-path
can be used to configure the merged metrics server, and the application service metrics endpoint on the consul sidecar.The flags
-default-enable-metrics
,-default-enable-metrics-merging
,-default-merged-metrics-port
,-default-prometheus-scrape-port
and-default-prometheus-scrape-path
configure the inject-connect command.
IMPROVEMENTS:
- CRDs: add field Last Synced Time to CRD status and add printer column on CRD to display time since when the resource was last successfully synced with Consul. [GH-448]
BUG FIXES:
- CRDs: fix incorrect validation for
ServiceResolver
. [GH-456]
BREAKING CHANGES:
-
Connect: the
lifecycle-sidecar
command has been renamed toconsul-sidecar
. [GH-428] -
Connect: the
consul-connect-lifecycle-sidecar
container name has been changed toconsul-sidecar
and theconsul-connect-envoy-sidecar
container name has been changed toenvoy-sidecar
. [GH-428] -
Connect: the
-default-protocol
and-enable-central-config
flags are no longer supported. Theconsul.hashicorp.com/connect-service-protocol
annotation on Connect pods is also no longer supported. [GH-418]Current deployments that have the annotation should remove it, otherwise they will get an error if a pod from that deployment is rescheduled.
Removing the annotation will not change their protocol since the config entry was already written to Consul. If you wish to change the protocol you must migrate the config entry to be managed by a
ServiceDefaults
resource. See Upgrade to CRDs for more information.To set the protocol for new services, you must use the
ServiceDefaults
resource, e.g.apiVersion: consul.hashicorp.com/v1alpha1 kind: ServiceDefaults metadata: name: my-service-name spec: protocol: "http"
-
Connect: pods using an upstream that references a datacenter, e.g.
consul.hashicorp.com/connect-service-upstreams: service:8080:dc2
will error during injection if Consul does not have aproxy-defaults
config entry with a mesh gateway mode set tolocal
orremote
. [GH-421]In practice, this would have already been causing issues since without that config setting, traffic wouldn't have been routed through mesh gateways and so would not be actually making it to the other service.
FEATURES:
- CRDs: support annotation
consul.hashicorp.com/migrate-entry
on custom resources that will allow an existing config entry to be migrated onto a Kubernetes custom resource. [GH-419] - Connect: add new cleanup controller that runs in the connect-inject deployment. This
controller cleans up Consul service instances that remain registered despite their
pods being deleted. This could happen if the pod's
preStop
hook failed to execute for some reason. [GH-433]
IMPROVEMENTS:
- CRDs: give a more descriptive error when a config entry already exists in Consul. [GH-420]
- Set
User-Agent: consul-k8s/<version>
header on calls to Consul where<version>
is the current version ofconsul-k8s
. [GH-434]
BUG FIXES:
- CRDs: Fix issue where a
ServiceIntentions
resource could be continually resynced with Consul because Consul's internal representation had a different order for an array than the Kubernetes resource. [GH-416] - CRDs: (Consul Enterprise only) default the
namespace
fields on resources where Consul performs namespace defaulting to prevent constant re-syncing. [GH-413]
IMPROVEMENTS:
- ACLs: give better error if policy that consul-k8s tries to update was created manually by user. [GH-412]
FEATURES:
- TLS: add
tls-init
command that is responsible for creating and updating Server TLS certificates. [GH-410]
BUG FIXES:
- Connect: on termination of a connect injected pod the lifecycle-sidecar sometimes re-registered the application resulting in stale service entries for applications which no longer existed. [GH-409]
BREAKING CHANGES:
- Connect: the flags
-envoy-image
and-consul-image
for commandinject-connect
are now required. [GH-405]
FEATURES:
- CRDs: add new CRD
IngressGateway
for configuring Consul's ingress-gateway config entry. [GH-407] - CRDs: add new CRD
TerminatingGateway
for configuring Consul's terminating-gateway config entry. [GH-408]
IMPROVEMENTS:
- Connect: Add
-log-level
flag toinject-connect
command. [GH-400] - Connect: Ensure
consul-connect-lifecycle-sidecar
container shuts down gracefully upon receivingSIGTERM
. [GH-389] - Connect: (Consul Enterprise only) give more descriptive error message if using Consul namespaces with a Consul installation that doesn't support namespaces. [GH-399]
FEATURES:
- Connect: Support Kubernetes health probe synchronization with Consul for connect injected pods. [GH-363]
- Adds a new controller to the connect-inject webhook which is responsible for synchronizing Kubernetes pod health checks with Consul service instance health checks. A Consul health check is registered for each connect-injected pod which mirrors the pod's Readiness status to Consul. This modifies connect routing to only pods which have passing Kubernetes health checks. See breaking changes for more information.
- Adds a new label to connect-injected pods which mirrors the
consul.hashicorp.com/connect-inject-status
annotation. - (Consul Enterprise only) Adds a new annotation to connect-injected pods when namespaces are enabled:
consul.hashicorp.com/consul-namespace
. [GH-376]
BREAKING CHANGES:
- Connect: With the addition of the connect-inject health checks controller any connect services which have failing Kubernetes readiness probes will no longer be routable through connect until their Kubernetes health probes are passing. Previously, if any connect services were failing their Kubernetes readiness checks they were still routable through connect. Users should verify that their connect services are passing Kubernetes readiness probes prior to using health checks synchronization.
DEPRECATIONS:
-
create-inject-token
in the server-acl-init command has been un-deprecated.-create-inject-auth-method
has been deprecated and replaced by-create-inject-token
.-create-inject-namespace-token
in the server-acl-init command has been deprecated. Please use-create-inject-token
and-enable-namespaces
flags to achieve the same functionality. [GH-368]
IMPROVEMENTS:
-
Connect: support passing extra arguments to the envoy binary. [GH-378]
Arguments can be passed in 2 ways:
- via a flag to the consul-k8s inject-connect command,
e.g.
consul-k8s inject-connect -envoy-extra-args="--log-level debug --disable-hot-restart"
- via pod annotations,
e.g.
consul.hashicorp.com/envoy-extra-args: "--log-level debug --disable-hot-restart"
- via a flag to the consul-k8s inject-connect command,
e.g.
-
CRDs:
BUG FIXES:
- Federation: (Consul Enterprise only) ensure replication ACL token can replicate policies and tokens in Consul namespaces other than
default
. [GH-364] - CRDs: (Consul Enterprise only) validate custom resources can only set namespace fields if Consul namespaces are enabled. [GH-375]
- CRDs: Ensure ACL token is global so that secondary DCs can manage custom resources. Without this fix, controllers running in secondary datacenters would get ACL errors. [GH-369]
- CRDs: (Consul Enterprise only) Do not attempt to create a
*
namespace when service intentions specify*
asdestination.namespace
. [GH-382] - CRDs: (Consul Enterprise only) Fix namespace support for ServiceIntentions CRD. [GH-362]
- CRDs: Rename field namespaces -> namespace in ServiceResolver CRD. [GH-365]
FEATURES:
-
Add beta support for new commands
consul-k8s controller
andconsul-k8s webhook-cert-manager
. [GH-353]controller
will start a Kubernetes controller that acts on Consul Custom Resource Definitions. The currently supported CRDs are:ProxyDefaults
- https://www.consul.io/docs/agent/config-entries/proxy-defaultsServiceDefaults
- https://www.consul.io/docs/agent/config-entries/service-defaultsServiceSplitter
- https://www.consul.io/docs/agent/config-entries/service-splitterServiceRouter
- https://www.consul.io/docs/agent/config-entries/service-routerServiceResolver
- https://www.consul.io/docs/agent/config-entries/service-resolverServiceIntentions
(requires Consul >= 1.9.0) - https://www.consul.io/docs/agent/config-entries/service-intentions
See https://www.consul.io/docs/k8s/crds for more information on the CRD schemas. Requires Consul >= 1.8.4.
webhook-cert-manager
manages certificates for Kubernetes webhooks. It will refresh expiring certificates and update corresponding secrets and mutating webhook configurations.
BREAKING CHANGES:
-
Connect: No longer set
--max-obj-name-len
flag when executingenvoy
. This flag was deprecated in Envoy 1.11.0 and had no effect from then onwards. With Envoy >= 1.15.0 setting this flag will result in an error, hence why we're removing it. [GH-350]If you are running any Envoy version >= 1.11.0 this change will have no effect. If you are running an Envoy version < 1.11.0 then you must upgrade Envoy to a newer version. This can be done by setting the
global.imageEnvoy
key in the Consul Helm chart.
IMPROVEMENTS:
- Add an ability to configure the synthetic Consul node name where catalog sync registers services. [GH-312]
- Sync: Add
-consul-node-name
flag to thesync-catalog
command to configure the Consul node name for syncing services to Consul. - ACLs: Add
-sync-consul-node-name
flag to the server-acl-init command so that it can create correct policy for the sync catalog.
- Sync: Add
BUG FIXES:
- Connect: use the first secret of type
kubernetes.io/service-account-token
when creating/updating auth method. [GH-350]
BUG FIXES:
- Connect: Reduce downtime caused by an alias health check of the sidecar proxy not being healthy for up to 1 minute
when a Connect-enabled service is restarted. Note that this fix reverts the behavior of Consul Connect to the behavior
it had before consul-k8s
v0.16.0
and Consulv1.8.x
, where Consul can route to potentially unhealthy instances of a service because we don't respect Kubernetes readiness/liveness checks yet. Please follow GH-155 for updates on that feature. [GH-305]
IMPROVEMENTS:
- Connect: Add resource request and limit flags for the injected init and lifecycle sidecar containers. These flags replace the hardcoded values previously included. As part of this change, the default value for the lifecycle sidecar container memory limit has increased from
25Mi
to50Mi
. [GH-298], [GH-300]
BUG FIXES:
- Connect: Respect allow/deny list flags when namespaces are disabled. [GH-296]
BREAKING CHANGES:
- ACLs: Always update Kubernetes auth method created by the
server-acl-init
job. Previously, we would only update the auth method if Consul namespaces are enabled. With this change, we always update it to make sure that any configuration changes or updates to theconnect-injector-authmethod-svc-account
are propagated [GH-282]. - Connect: Connect pods have had the following resource settings changed:
consul-connect-inject-init
now has its memory limit set to150M
up from25M
andconsul-connect-lifecycle-sidecar
has its CPU request and limit set to20m
up from10m
. [GH-291]
IMPROVEMENTS:
- Extracted Consul's HTTP flags into our own package so we no longer depend on the internal Consul golang module. [GH-259]
BUG FIXES:
- Connect: Update resource settings to fix out of memory errors and CPU usage at 100% of limit. [GH-283, consul-helm GH-515]
- Connect: Creating a pod with a different service account name than its Consul service name will now result in an error when ACLs are enabled. Previously this would not result in an error, but the pod would not be able to send or receive traffic because its ACL token would be for a different service name. [GH-237]
FEATURES:
- ACLs:
server-acl-init
now supports creating tokens for ingress and terminating gateways [GH-264].- Add
-ingress-gateway-name
flag that takes the name of an ingress gateway that needs an acl token. May be specified multiple times. [Enterprise Only] If using Consul namespaces and registering the gateway outside of the default namespace, specify the value in the form<GatewayName>.<ConsulNamespace>
. - Add
-terminating-gateway-name
flag that takes the name of a terminating gateway that needs an acl token. May be specified multiple times. [Enterprise Only] If using Consul namespaces and registering the gateway outside of the default namespace, specify the value in the form<GatewayName>.<ConsulNamespace>
.
- Add
- Connect: Add support for configuring resource settings for memory and cpu limits/requests for sidecar proxies. [GH-267]
BREAKING CHANGES:
- Gateways:
service-address
command will now return hostnames if that is the address of the Kubernetes LB. Previously it would resolve the hostname to 1 IP. The-resolve-hostnames
flag was added to preserve the IP resolution behavior. [GH-271]
IMPROVEMENTS:
- Sync: Add
-sync-lb-services-endpoints
flag to optionally sync load balancer endpoint IPs instead of load balancer ingress IP or hostname to Consul [GH-257]. - Connect: Add pod name to the consul connect metadata for connect injected pods. [GH-231]
BUG FIXES:
- Connect:
- Fix bug where preStop hook was malformed. This caused Consul ACL tokens to never be deleted for connect services. [GH-265]
- Fix bug where environment variable for upstream was not populated when using a different datacenter resulted. [GH-246]
- Fix bug where the Connect health-check was defined with a service name instead of a service ID. This check was passing in consul version before 1.8, but will now fail with versions 1.8 and higher. [GH-272]
BREAKING CHANGES:
- The
service-address
command now resolves load balancer hostnames to the first IP. Previously it would use the hostname directly. This is a stop-gap measure because Consul currently only supports IP addresses for mesh gateways. [GH-260]
FEATURES:
- Add new
create-federation-secret
command that will create a Kubernetes secret containing data needed for secondary datacenters to federate. This command should be run only in the primary datacenter. [GH-253]
BREAKING CHANGES:
- ACLs: Remove
-expected-replicas
,-release-name
, and-server-label-selector
flags in favor of the new required-server-address
flag [GH-238].
FEATURES:
- ACLs: The
server-acl-init
command can now run against Consul servers running outside of k8s [GH-243]:- Add
-bootstrap-token-file
flag to provide your own bootstrap token. If set, the command will skip ACL bootstrapping. -server-address
flag can also take a cloud auto-join string to discover server addresses.- Add
-inject-auth-method-host
flag to allow configuring the location of the Kubernetes API server for the Kubernetes auth method. This is useful because during the login workflow Consul servers are talking to the Kubernetes API to verify the service account token. When Consul servers are external to the Kubernetes cluster, we no longer know the address of the Kubernetes API server that is accessible from the external Consul servers.
- Add
IMPROVEMENTS:
- ACLs: Add
-server-address
and-server-port
flags so that we don't need to discover server pod IPs and ports through the Kubernetes API [GH-238].
BUG FIXES:
- Connect: Fix upstream annotation parsing when multiple prepared queries are separated by spaces [GH-224]
- ACLs: Fix bug with
acl-init -token-sink-file
where running the command twice would fail [GH-248]
FEATURES:
-
ACLs: Support new flag
server-acl-init -create-acl-replication-token
that creates an ACL token with permissions to perform ACL replication. [GH-210] -
ACLs: Support ACL replication from another datacenter. If
-acl-replication-token-file
is set, theserver-acl-init
command will skip ACL bootstrapping and instead will use the token in that file to create policies and tokens. This enables theserver-acl-init
command to be run in secondary datacenters. [GH-226] -
ACLs: Support new flag
acl-init -token-sink-file
that will write the token to the specified file. [GH-232] -
Commands: Add new command
service-address
that writes the address of the specified Kubernetes service to a file. If the service is of typeLoadBalancer
, the command will wait until the external address of the load balancer has been assigned. If the service is of typeClusterIP
it will write the cluster IP. Services of typeNodePort
orExternalName
will result in an error. [GH-234 and GH-235]Example usage:
consul-k8s service-address \ -k8s-namespace=default \ -name=consul-mesh-gateway \ -output-file=address.txt
-
Commands: Add new
get-consul-client-ca
command that retrieves Consul clients' CA when auto-encrypt is enabled and writes it to a file [GH-211].
IMPROVEMENTS:
- ACLs: The following ACL tokens have been changed to local tokens rather than
global tokens because they only need to be valid in their local datacenter:
client
,enterprise-license
,snapshot-agent
. In addition, if Consul Enterprise namespaces are not enabled, thecatalog-sync
token will be local. [GH-226] - ACLs: If running with
-create-acl-replication-token=true
and-create-inject-auth-method=true
, the anonymous policy will be configured to allow read access to all nodes and services. This is required for cross-datacenter Consul Connect requests to work. [GH-230]. - ACLs: The policy for the anonymous token has been renamed from
dns-policy
toanonymous-token-policy
since it is used for more than DNS now (see above). [GH-230].
BUG FIXES:
- Sync: Fix a race condition where sync would delete services at initial startup [GH-208]
DEPRECATIONS:
- ACLs: The flag
-init-type=sync
for the commandacl-init
has been deprecated. Only the flag-init-type=client
is supported. Previously, setting-init-type=sync
had no effect so this is not a breaking change. [GH-232] - Connect: deprecate the
-consul-ca-cert
flag in favor of-ca-file
[GH-217]
BREAKING CHANGES:
-
Connect Injector
- Previously the injector would inject sidecars into pods in all namespaces. New flags
-allow-k8s-namespace
and-deny-k8s-namespace
have been added. If no-allow-k8s-namespace
flag is specified, the injector will not inject sidecars into pods in any namespace. To maintain the previous behavior, set-allow-k8s-namespace='*'
.
- Previously the injector would inject sidecars into pods in all namespaces. New flags
-
Catalog Sync
kube-system
andkube-public
namespaces are now synced from unless-deny-k8s-namespace=kube-system -deny-k8s-namespace=kube-public
are passed to thesync-catalog
command.- Previously, multiple sync processes could be run in the same Kubernetes cluster with different source Kubernetes namespaces and the same
-consul-k8s-tag
. This is no longer possible. The sync processes will now delete one-another's registrations. To continue running multiple sync processes, each process must be passed a different-consul-k8s-tag
flag. - Previously, catalog sync would delete services tagged with
-consul-k8s-tag
(defaults tok8s
) that were registered out-of-band, i.e. not by the sync process itself. It would delete services regardless of which node they were registered on. Now the sync process will only delete those services not registered by itself if they are on thek8s-sync
node (the synthetic node created by the catalog sync process).
-
Connect and Mesh Gateways: Consul 1.7+ now requires that we pass
-envoy-version
flag if using a version other than the default (1.13.0) so that it can generate correct bootstrap configuration. This is not yet supported in the Helm chart and consul-k8s, and as such, we require Envoy version 1.13.0.
IMPROVEMENTS:
- Support Consul namespaces [Enterprise feature] in all consul-k8s components [GH-197]
- Create allow and deny lists of k8s namespaces for catalog sync and Connect inject
- Connect Inject
- Changes default Consul Docker image (
-consul-image
) toconsul:1.7.1
- Changes default Envoy Docker image (
-envoy-image
) toenvoyproxy/envoy-alpine:v1.13.0
- Changes default Consul Docker image (
BUG FIXES:
- Bootstrap ACLs: Allow users to update their Connect ACL binding rule definition on upgrade
- Bootstrap ACLs: Fixes mesh gateway ACL policies to have the correct permissions
- Sync: Fixes a hot loop bug when getting an error from Consul when retrieving service information [GH-204]
DEPRECATIONS:
connect-inject
flag-create-inject-token
is deprecated in favor of new flag-create-inject-auth-method
NOTES:
-
Bootstrap ACLs: Previously, ACL policies were not updated after creation. Now, if namespaces are enabled, they are updated every time the ACL bootstrapper is run so that any namespace config changes can be adjusted. This change is only an issue if you are updating ACL policies after creation.
-
Connect: Adds additional parsing of the upstream annotation to support namespaces. The format of the annotation becomes:
service_name.optional_namespace:port:optional_datacenter
The
service_name.namespace
is only parsed if namespaces are enabled. If they are not enabled and someone has added a.namespace
, the upstream will not work correctly, as is the case when someone has put in an incorrect service name, port or datacenter. If namespaces are enabled and the.namespace
is not defined, Consul will automatically fallback to assuming the service is in the same namespace as the service defining the upstream.
Improvements:
Notes:
-
Build: Our darwin releases for this version and up will be signed and notarized according to Apple's requirements. Prior to this release, MacOS 10.15+ users attempting to run our software may see the error: "'consul-k8s' cannot be opened because the developer cannot be verified." This error affected all MacOS 10.15+ users who downloaded our software directly via web browsers, and was caused by changes to Apple's third-party software requirements.
MacOS 10.15+ users should plan to upgrade to 0.11.0+.
-
Build: ARM release binaries: Starting with 0.11.0,
consul-k8s
will ship three separate versions of ARM builds. The previous ARM binaries of Consul could potentially crash due to the way the Go runtime manages internal pointers to its Go routine management constructs and how it keeps track of them especially during signal handling (golang/go#32912). From 0.11.0 forward, it is recommended to use:consul-k8s_{version}_linux_armelv5.zip for all 32-bit armel systems consul-k8s_{version}_linux_armhfv6.zip for all armhf systems with v6+ architecture consul-k8s_{version}_linux_arm64.zip for all v8 64-bit architectures
-
Build: The
freebsd_arm
variant has been removed.
Bug Fixes:
- Connect: Fix bug where the new lifecycle sidecar didn't have permissions to read the ACL token file. [GH-182]
Bug Fixes:
-
Connect: Fix critical bug where Connect-registered services instances would be deregistered when the Consul client on the same node was restarted. This fix adds a new sidecar that ensures the service instance is always registered. [GH-161]
-
Connect: Fix bug where UI links between sidecar and service didn't work because the wrong service ID was being used. [GH-163]
-
Bootstrap ACLs: Support bootstrapACLs for users setting the
nameOverride
config. [GH-165]
Bug Fixes:
-
Sync: Add Kubernetes namespace as a suffix to the service names via
-add-k8s-namespace-suffix
flag. This prevents service name collisions in Consul when there are two services with the same name in different namespaces in Kubernetes [GH-139] -
Connect: Only write a
service-defaults
config during Connect injection if the protocol is set explicitly [GH-169]
Bug Fixes:
-
Sync: Now changing the annotation
consul.hashicorp.com/service-sync
tofalse
or deleting the annotation will un-sync the service. [GH-76] -
Sync: Rewrite Consul services to lowercase so they're valid Kubernetes services. [GH-110]
Bug Fixes:
-
Add new delete-completed-job command that is used to delete the server-acl-init Kubernetes Job once it's completed. [GH-152]
-
Fixes a bug where even if the ACL Tokens for the other components existed (e.g. client or sync-catalog) we'd try to generate new tokens and update the secrets. [GH-152]
Improvements:
- Allow users to set annotations on their Kubernetes services that get synced into
Consul meta when using the Connect Inject functionality.
To use, set one or more
consul.hashicorp.com/service-meta-<key>: <value>
annotations which will result in Consul meta<key>: <value>
[GH-141]
Bug Fixes:
-
Fix bug during connect-inject where the
-default-protocol
flag was being ignored [GH-141] -
Fix bug during connect-inject where service-tag annotations were being ignored [GH-141]
-
Fix bug during
server-acl-init
where if any step errored then the command would exit and subsequent commands would fail. Now this command runs until completion, i.e. it retries failed steps indefinitely and is idempotent [GH-138]
Deprecations:
- The
consul.hashicorp.com/connect-service-tags
annotation is deprecated. Useconsul.hashicorp.com/service-tags
instead.
Improvements:
- Allow users to set tags on their Kubernetes services that get synced into
Consul service tags via the
consul.hashicorp.com/connect-service-tags
annotation [GH-115]
Bug fixes:
- Fix bootstrap acl issue when Consul was installed into a namespace other than
default
[GH-106] - Fix sync bug where
ClusterIP
services had theirService
port instead of theirEndpoint
port registered. If theService
'stargetPort
was different thenport
then the wrong port would be registered [GH-132]
Improvements:
- Allow creation of ACL token for Snapshot Agents
- Allow creation of ACL token for Mesh Gateways
- Allows client ACL token creation to be optional
Bug fixes:
- Fix central configuration write command to handle the case where the service already exists
Improvements:
- Use the endpoint IP address when generating a service id for NodePort services to prevent possible overlap of what are supposed to be unique ids
- Support adding a prefix for Kubernetes -> Consul service sync [GH 140]
- Support automatic bootstrapping of ACLs in a Consul cluster that is run fully in Kubernetes.
- Support automatic registration of a Kubernetes AuthMethod for use with Connect (available in Consul 1.5+).
- Support central configuration for services, including proxy defaults (available in Consul 1.5+).
Bug fixes:
- Exclude Kubernetes system namespaces from Connect injection
Improvements:
- Use service's namespace when registering endpoints
- Update the Coalesce method to pass go vet tests
- Register Connect services along with the proxy. This allows the services to appear in the intention dropdown in the UI.[GH 77]
- Add
-log-level
CLI flag for catalog sync
Improvements:
- Add support for prepared queries in the Connect upstream annotation
- Add a health endpoint to the catalog sync process that can be used for Kubernetes health and readiness checks
Improvements:
- Clarify the format of the
consul-write-interval
flag forconsul-k8s
[GH 61] - Add datacenter support to inject annotation
- Update connect injector logging to remove healthcheck log spam and make important messages more visible
Bug fixes:
- Fix service registration naming when using Connect [GH 36]
- Fix catalog sync so that agents don't incorrectly deregister Kubernetes services [GH 40][GH 59]
- Fix performance issue for the k8s -> Consul catalog sync [GH 60]
Improvements:
- Supports a configurable tag for the k8s -> Consul sync [GH 42]
Bug fixes:
- Register NodePort services with the node's ip address [GH 8]
- Add the metadata/annotations field if needed before patching annotations [GH 20]
Improvements:
- Support syncing ClusterIP services [GH 4]
Bug fixes:
- Allow unnamed container ports to be used in connect-inject default annotations.
Bug fixes:
- Fix single direction catalog sync [GH 7]
Features:
- New subcommand:
inject-connect
runs a mutating admission webhook for automatic Connect sidecar injection in Kubernetes. While this can be setup manually, we recommend using the Consul helm chart.
- Initial release