demo-app-gw-env

This project provides a product demonstration of the Application Gateway, Azure Kubernetes Service and App Gateway Ingress Controller (AGIC) stack. Two applications are deployed to AKS where AGIC configures ingress access through App Gateway using the ARM API. Look for the following:

• End to End TLS (client, server and backend verification);
• Path based routing;
• Path based custom WAF policies.

Getting Started

This repository provides the primitives needed to provision the required infrastructure in Microsoft Azure (bring-your-own-subscription), configure the cluster, and deploy the applications. A Makefile is also provided with a series of targets to automate these steps. In a product environment, each of these make targets would be analogous to a pipeline stage.

First, we need to format a .env file for some of our sensitive parameters. A template follows:

TF_VAR_azure_client_secret=<>
TF_VAR_aks_service_principal_client_secret=<>

Gather the appropriate parameters and build the file. You will need at least one service principal for this.

az ad sp create-for-rbac --name app-gw-sp --role Contributor --scopes /subscriptions/<subscription_id>

az ad sp list --display-name app-gw-sp --query "[].{\"Object ID\":objectId}" --output table

Record these values in your .env file with the following mappings:

• Set TF_VAR_azure_client_secret and TF_VAR_aks_service_principal_client_secret in .env to the password value returned.
• Set azure_client_id and aks_service_principal_app_id in demo.tfvars to the appId field returned.
• Set azure_tenant_id in demo.tfvars to the tenant field returned.
• Set aks_service_principal_object_id to the returned Object ID field.
• Set azure_dns_resource_group to the name of the esource group that holds the DNS Zone.
• Set azure_dns_zone_name to the DNS Zone name.
• Set domain_name to the domain contained in the DNS zone.
• Set registration_email to a valid email. You will be contacted at this email before the certificate expires.

We are now ready to build the Azure resources using Terraform.

make build

Once this is completed, deploy cert-manager + ClusterIssuer, agic and some necessary CRDs.

make agic

Now we can install the applications:

make services

Give the services, ingresses and Lets Encrypt certificates time to generate (~30s, depending on the speed of the DNS challenge for ACME certificate request) and then issue the following command to deploy the pods.

make pods

Open your browser and verify that the endpoints work:

• https://api.jacobnosal.com/app-a
• https://api.jacobnosal.com/app-b

Environment Walkthrough

The AKS cluster is hosting two applications: app-a and app-b. App Gateway exposes these using path based routing as api.jacobnosal.com/{app-a,app-b} and applies a rewrite rule set to forward traffic to the appropriate services and pods with the /{app-a,app-b} prefix removed.

Annotations on the ingress resource apply a custom WAF policy to all paths contained in that resource. As path based routing implies each application is exposed under a prefix to the URL path, we can apply a custom WAF policy to each application.

SSL Policy

The Application Gateway is configured with a default SSL Policy to all incoming requests. Application Gateway Ingress Controller does not support the conifguration of a per-ingress SSL Profile but Application Gateway does. This issue tracks a feature parity request between App gateway and AGIC. Predefined policies are explained here.

ssl_policy {
    policy_type = "Predefined"
    policy_name = "AppGwSslPolicy20220101S"
}

WAF Policies

Each Azure WAF Policy allows for the configuration of custom rules in addition a core rule set. WAF Policy configurations are as follows:

• app-a-waf-policy
  • OWASP Ruleset v3.1
  • Match X-CUSTOM-HEADER contains 'something-suspicious' => BLOCK
• app-b-waf-policy
  • OWASP Ruleset v3.1
  • Match RequestUri contains '/blocked-page', '/blocked-pages/*' => BLOCK

command	result	reason
curl -v https://api.jacobnosal.com/app-a	✅	The request does not trigger a block due to headers
curl -v https://api.jacobnosal.com/app-b	✅	The request does not trigger a block due to routes
curl -v https://api.jacobnosal.com/app-a -H "X-CUSTOM-HEADER: this-is-something-suspicious"	❌	The app-a-waf-policy matches RequestHeaders/X-CUSTOM-HEADER against something-suspicious with the contains operator
curl -v https://api.jacobnosal.com/app-b -H "X-CUSTOM-HEADER: this-is-something-suspicious"	❌	The app-b-waf-policy does not block this header value
curl -v https://api.jacobnosal.com/app-a/blocked-pages/aaa	✅	The app-a-waf-policy does not block this route
curl -v https://api.jacobnosal.com/app-b/blocked-page	❌	The URL matches against the /blocked-page value with the contains operator
curl -v https://api.jacobnosal.com/app-b/blocked-pages/bbb	❌	The URL matches against the /blocked-page value with the contains operator
curl -v https://api.jacobnosal.com/app-a --tlsv1.1 --tls-max 1.1	❌	TLS v1.1 is not an allowed TLS version
curl -v https://api.jacobnosal.com/app-a --tlsv1.2 --tls-max 1.2	✅	TLS v1.2 is allowed by SSL Policy
curl -v https://api.jacobnosal.com/app-a --tlsv1.3	✅	TLS v1.3 is allowed by SSL Policy

Requirements

Name	Version
terraform	>= 0.14
azurerm	>= 3.4.0

Providers

Name	Version
azurerm	3.26.0

Modules

Name	Source	Version
waf-policies	./waf_policy	n/a

Resources

Name	Type
azurerm_application_gateway.network	resource
azurerm_dns_a_record.api_jacobnosal_com	resource
azurerm_kubernetes_cluster.k8s	resource
azurerm_public_ip.pip	resource
azurerm_virtual_network.test	resource
azurerm_client_config.current	data source
azurerm_resource_group.rg	data source
azurerm_subnet.appgwsubnet	data source
azurerm_subnet.kubesubnet	data source
azurerm_user_assigned_identity.app-gw-id	data source

Inputs

Name	Description	Type	Default	Required
aks_agent_count	The number of agent nodes for the cluster.	number	3	no
aks_agent_os_disk_size	Disk size (in GB) to provision for each of the agent pool nodes. This value ranges from 0 to 1023. Specifying 0 applies the default disk size for that agentVMSize.	number	40	no
aks_agent_vm_size	VM size	string	"Standard_D3_v2"	no
aks_dns_prefix	Optional DNS prefix to use with hosted Kubernetes API server FQDN.	string	"aks"	no
aks_dns_service_ip	DNS server IP address	string	"10.0.0.10"	no
aks_docker_bridge_cidr	CIDR notation IP for Docker bridge.	string	"172.17.0.1/16"	no
aks_enable_rbac	Enable RBAC on the AKS cluster. Defaults to false.	string	"false"	no
aks_name	AKS cluster name	string	"aks-cluster1"	no
aks_service_cidr	CIDR notation IP range from which to assign service cluster IPs	string	"10.0.0.0/16"	no
aks_service_principal_app_id	Application ID/Client ID of the service principal. Used by AKS to manage AKS related resources on Azure like vms, subnets.	any	n/a	yes
aks_service_principal_client_secret	Secret of the service principal. Used by AKS to manage Azure.	any	n/a	yes
aks_service_principal_object_id	Object ID of the service principal.	any	n/a	yes
aks_subnet_address_prefix	Subnet address prefix.	string	"192.168.0.0/24"	no
aks_subnet_name	Subnet Name.	string	"kubesubnet"	no
app_gateway_name	Name of the Application Gateway	string	"ApplicationGateway1"	no
app_gateway_sku	Name of the Application Gateway SKU	string	"WAF_v2"	no
app_gateway_subnet_address_prefix	Subnet server IP address.	string	"192.168.1.0/24"	no
app_gateway_tier	Tier of the Application Gateway tier	string	"WAF_v2"	no
azure_client_id	n/a	any	n/a	yes
azure_client_secret	n/a	any	n/a	yes
azure_dns_resource_group	n/a	any	n/a	yes
azure_dns_zone_name	n/a	any	n/a	yes
azure_subscription_id	n/a	any	n/a	yes
azure_tenant_id	n/a	any	n/a	yes
domain_name	DNS name of the certificate subject.	any	n/a	yes
kubernetes_version	Kubernetes version	string	"1.11.5"	no
location	n/a	string	"eastus"	no
managed_identity_name	Name of the managed identity.	string	"ApplicationGatewayIdentity"	no
public_ssh_key_path	Public key path for SSH.	string	"~/.ssh/id_rsa.pub"	no
registration_email	Email to register with ACME for this cert.	any	n/a	yes
resource_group_name	n/a	string	"rg-app-gw-demo"	no
tags	n/a	map	{ "environment": "demo" }	no
virtual_network_address_prefix	VNET address prefix	string	"192.168.0.0/16"	no
virtual_network_name	Virtual network name	string	"aksVirtualNetwork"	no
vm_user_name	User name for the VM	string	"vmuser1"	no
waf_resource_group	n/a	string	"rg-waf-policies"	no

Outputs

Name	Description
aks_rbac_enabled	n/a
application_domain_name	n/a
application_gateway_name	n/a
application_ip_address	n/a
client_certificate	n/a
client_id	n/a
client_key	n/a
cluster_ca_certificate	n/a
cluster_name	n/a
cluster_password	n/a
cluster_username	n/a
dns_resource_group_name	n/a
dns_zone_name	n/a
host	n/a
kube_config	n/a
registration_email	n/a
resource_group_name	n/a
subscription_id	n/a
tenant_id	n/a