Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support running with restricted PSA enforcement enabled (part 1) #2572

Merged
merged 14 commits into from
Jul 24, 2023
Merged

Support running with restricted PSA enforcement enabled (part 1) #2572

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
45d3af0
Support restricted PSA enforcement in basic setup
Jul 14, 2023
cf9715f
Test: Support `TestConnectInject` on OpenShift + CNI
Jul 14, 2023
ff5c24b
fix: Use security context template
Jul 14, 2023
9e83262
tests: Add static-client and static-server fixtures for OpenShift
Jul 14, 2023
e45cc5a
fix: Respect user-provided securityContext settings
Jul 14, 2023
0671525
Merge remote-tracking branch 'origin/main' into pglass/NET-185/basic-…
Jul 19, 2023
cbc48f7
Remove -cni-namespace flag
Jul 19, 2023
2790c63
Update bats tests
Jul 20, 2023
7b2e6d3
Add changelog entry
Jul 20, 2023
2b2eefb
Set runAsUser on vanilla kube
Jul 20, 2023
d0af4f0
Fix acceptance framework unit test
Jul 20, 2023
fb98d10
Merge remote-tracking branch 'origin/main' into pglass/NET-185/basic-…
Jul 21, 2023
57c89ab
Fix rebase errors
Jul 21, 2023
b8c37fb
Rename flag to -enabled-restricted-psa-enforcement
Jul 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion acceptance/framework/config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"path/filepath"
"strconv"
"strings"
"testing"

"github.com/hashicorp/go-version"
"gopkg.in/yaml.v2"
Expand All @@ -33,14 +34,18 @@ type TestConfig struct {
SecondaryKubeContext string
SecondaryKubeNamespace string

AppNamespace string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want a bunch of these flags/settings as top level flags as I could see them having unintended consequences. For example, AppNamespace/SecondaryAppNamespace; I am sure that if you set this globably in CI and started installing static-client/static-server in different namespaces that a large chunk of the tests will fail.

Now you need to modify all the tests to basically handle -n in the kubectl apply. Not really a bad thing to handle overall but probalby not a framework change you want in this PR.

I'd make AppNamespace/SecondaryAppNamespace test specific or OpenShift specific if needed.

For CNINamespace, we made the design decision that CNI was always installed alongside consul. Othere CNI plugins do it differently and install their plugins in their own namespace. I don't think OpenShift needs the CNI to be installed somewhere else to work? If you want it for setting SCC parameters then you should be able to use {{ release-name }} from helm maybe?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For CNINamespace, we made the design decision that CNI was always installed alongside consul. Othere CNI plugins do it differently and install their plugins in their own namespace. I don't think OpenShift needs the CNI to be installed somewhere else to work?

Right. But if someone wants to run Consul in a namespace with restricted PSA enforcement, then they can't run the CNI in that same namespace because the CNI requires privilege. Eventually, (we assume) restricted PSA enforcement is going to be the OpenShift default, so the goal here was to enable testing with that potential future configuration.

I'd make AppNamespace/SecondaryAppNamespace test specific or OpenShift specific if needed.

This is still relevant to vanilla K8s because you can enable restricted PSA enforcement there as well (although it likely will never be the default like on OpenShift). So I didn't want to make the app namespace settings OpenShift-specific, because it's useful and (and easier) to be able to test this same configuration on vanilla K8s as well.

Copy link
Contributor Author

@pglass pglass Jul 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another aspect of this choice is that I eventually (if I had infinite time) want to run all existing tests with PSA enforcement enabled, so I wanted a way that could apply to all tests, so that every test could deploy Consul and the CNI and apps in a compatible way. Then I can run the entire suite for coverage.

That is more of a dream at this point, because I still have to touch every test and it's unlikely I'd finish that work. But I haven't come up with a way to test with restricted PSA enforcement enabled that doesn't involve touching potentially every test case.

Copy link
Contributor Author

@pglass pglass Jul 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I replaced all these flags with -restricted-psa-enforcement-enabled (default false). When true:

  1. The Consul namespace(s) passed to the tests via -kube-namespaces need to be configured with the restricted PSA enforcement label before tests are run.
  2. The CNI is deployed into the kube-system namespace, which is privileged and already exists (so I removed the -cni-namespace flag)
  3. I removed the app namespace flags. Instead, ConnectHelper supports a UseAppNamespace setting that tests can optionally use (based on whether -restricted-psa-enforcement-enabled is set)

Let me know what you think.

SecondaryAppNamespace string

EnableEnterprise bool
EnterpriseLicense string

EnableOpenshift bool

EnablePodSecurityPolicies bool

EnableCNI bool
EnableCNI bool
CNINamespace string

EnableTransparentProxy bool

Expand Down Expand Up @@ -101,10 +106,18 @@ func (t *TestConfig) HelmValuesFromConfig() (map[string]string, error) {

if t.EnableCNI {
setIfNotEmpty(helmValues, "connectInject.cni.enabled", "true")
setIfNotEmpty(helmValues, "connectInject.cni.logLevel", "debug")
// GKE is currently the only cloud provider that uses a different CNI bin dir.
if t.UseGKE {
setIfNotEmpty(helmValues, "connectInject.cni.cniBinDir", "/home/kubernetes/bin")
}
if t.EnableOpenshift {
setIfNotEmpty(helmValues, "connectInject.cni.multus", "true")
setIfNotEmpty(helmValues, "connectInject.cni.cniBinDir", "/var/lib/cni/bin")
setIfNotEmpty(helmValues, "connectInject.cni.cniNetDir", "/etc/kubernetes/cni/net.d")
}

setIfNotEmpty(helmValues, "connectInject.cni.namespace", t.CNINamespace)
}

setIfNotEmpty(helmValues, "connectInject.transparentProxy.defaultEnabled", strconv.FormatBool(t.EnableTransparentProxy))
Expand Down Expand Up @@ -169,6 +182,12 @@ func (t *TestConfig) entImage() (string, error) {
return fmt.Sprintf("hashicorp/consul-enterprise:%s%s-ent", consulImageVersion, preRelease), nil
}

func (c *TestConfig) SkipWhenOpenshiftAndCNI(t *testing.T) {
if c.EnableOpenshift && c.EnableCNI {
t.Skip("skipping because -enable-cni and -enable-openshift are set and this test doesn't deploy apps correctly")
}
}

// setIfNotEmpty sets key to val in map m if value is not empty.
func setIfNotEmpty(m map[string]string, key, val string) {
if val != "" {
Expand Down
80 changes: 61 additions & 19 deletions acceptance/framework/connhelper/connect_helper.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,11 @@ type ConnectHelper struct {
// ReleaseName is the name of the Consul cluster.
ReleaseName string

// Ctx is used to deploy Consul
Ctx environment.TestContext
Cfg *config.TestConfig
// AppCtx is used to deploy applications. If nil, then Ctx is used.
AppCtx environment.TestContext
Cfg *config.TestConfig

// consulCluster is the cluster to use for the test.
consulCluster consul.Cluster
Expand Down Expand Up @@ -82,6 +85,14 @@ func (c *ConnectHelper) Upgrade(t *testing.T) {
c.consulCluster.Upgrade(t, c.helmValues())
}

// appCtx returns the context where apps are deployed.
func (c *ConnectHelper) appCtx() environment.TestContext {
if c.AppCtx != nil {
return c.AppCtx
}
return c.Ctx
}

// DeployClientAndServer deploys a client and server pod to the Kubernetes
// cluster which will be used to test service mesh connectivity. If the Secure
// flag is true, a pre-check is done to ensure that the ACL tokens for the test
Expand All @@ -108,23 +119,47 @@ func (c *ConnectHelper) DeployClientAndServer(t *testing.T) {

logger.Log(t, "creating static-server and static-client deployments")

k8s.DeployKustomize(t, c.Ctx.KubectlOptions(t), c.Cfg.NoCleanupOnFailure, c.Cfg.DebugDirectory, "../fixtures/cases/static-server-inject")
if c.Cfg.EnableTransparentProxy {
k8s.DeployKustomize(t, c.Ctx.KubectlOptions(t), c.Cfg.NoCleanupOnFailure, c.Cfg.DebugDirectory, "../fixtures/cases/static-client-tproxy")
ctx := c.appCtx()
opts := ctx.KubectlOptions(t)

if c.Cfg.EnableCNI && c.Cfg.EnableOpenshift {
// On OpenShift with the CNI, we need to create a network attachment definition in the namespace
// where the applications are running, and the app deployment configs need to reference that network
// attachment definition.

// TODO: A base fixture is the wrong place for these files
k8s.KubectlApply(t, opts, "../fixtures/bases/openshift/")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of using the AppNamespace flag, you could set the namespace here using opts when CNI + OpenShift is set.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really clear what you mean. This is in ConnectHelper so how does it know the namespace to use?

Right now, it's using a namespace sourced from the test flags via the context (bc that's what I set up). Do you mean I should instead have each test pass a namespace to ConnectHelper if needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried something else:

  • I added ConnectHelper.UseAppNamespace = true|false option
    • If true then it creates, configures, and deploys into a namespace named <consul-namespace>-apps.

This way there is a shared/common way to easily create an app namespace that tests can use if needed and there are no more -app-namespace flags.

I also added a -restricted-psa-enforcement-enabled flag and tests key off of that to determine whether to deploy apps into a separate namespace. (I don't know if I really like this, so I'm open to alternatives)

Copy link
Contributor Author

@pglass pglass Jul 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Also, I accidentally did all of this while rebasing, so the latest commit that looks like a merge from main actually contains the changes - sorry!)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the -restricted-psa-enforcement-enabled flag. I think this is a good + global way to do things as I could see us wanting to test this on all the clouds in the future.

helpers.Cleanup(t, c.Cfg.NoCleanupOnFailure, func() {
k8s.KubectlDelete(t, opts, "../fixtures/bases/openshift/")
})

k8s.DeployKustomize(t, opts, c.Cfg.NoCleanupOnFailure, c.Cfg.DebugDirectory, "../fixtures/cases/static-server-openshift")
curtbushko marked this conversation as resolved.
Show resolved Hide resolved
if c.Cfg.EnableTransparentProxy {
k8s.DeployKustomize(t, opts, c.Cfg.NoCleanupOnFailure, c.Cfg.DebugDirectory, "../fixtures/cases/static-client-openshift-tproxy")
} else {
k8s.DeployKustomize(t, opts, c.Cfg.NoCleanupOnFailure, c.Cfg.DebugDirectory, "../fixtures/cases/static-client-openshift-inject")
}
} else {
k8s.DeployKustomize(t, c.Ctx.KubectlOptions(t), c.Cfg.NoCleanupOnFailure, c.Cfg.DebugDirectory, "../fixtures/cases/static-client-inject")
}
k8s.DeployKustomize(t, opts, c.Cfg.NoCleanupOnFailure, c.Cfg.DebugDirectory, "../fixtures/cases/static-server-inject")
if c.Cfg.EnableTransparentProxy {
k8s.DeployKustomize(t, opts, c.Cfg.NoCleanupOnFailure, c.Cfg.DebugDirectory, "../fixtures/cases/static-client-tproxy")
} else {
k8s.DeployKustomize(t, opts, c.Cfg.NoCleanupOnFailure, c.Cfg.DebugDirectory, "../fixtures/cases/static-client-inject")
}

}
// Check that both static-server and static-client have been injected and
// now have 2 containers.
retry.RunWith(
&retry.Timer{Timeout: 30 * time.Second, Wait: 100 * time.Millisecond}, t,
func(r *retry.R) {
for _, labelSelector := range []string{"app=static-server", "app=static-client"} {
podList, err := c.Ctx.KubernetesClient(t).CoreV1().Pods(c.Ctx.KubectlOptions(t).Namespace).List(context.Background(), metav1.ListOptions{
LabelSelector: labelSelector,
FieldSelector: `status.phase=Running`,
})
podList, err := ctx.KubernetesClient(t).CoreV1().
Pods(opts.Namespace).
List(context.Background(), metav1.ListOptions{
LabelSelector: labelSelector,
FieldSelector: `status.phase=Running`,
})
require.NoError(r, err)
require.Len(r, podList.Items, 1)
require.Len(r, podList.Items[0].Spec.Containers, 2)
Expand All @@ -136,7 +171,7 @@ func (c *ConnectHelper) DeployClientAndServer(t *testing.T) {
// and intentions. This helper is primarly used to ensure that the virtual-ips are persisted to consul properly.
func (c *ConnectHelper) CreateResolverRedirect(t *testing.T) {
logger.Log(t, "creating resolver redirect")
options := c.Ctx.KubectlOptions(t)
options := c.appCtx().KubectlOptions(t)
kustomizeDir := "../fixtures/cases/resolver-redirect-virtualip"
k8s.KubectlApplyK(t, options, kustomizeDir)

Expand All @@ -149,10 +184,12 @@ func (c *ConnectHelper) CreateResolverRedirect(t *testing.T) {
// server fails when no intentions are configured.
func (c *ConnectHelper) TestConnectionFailureWithoutIntention(t *testing.T) {
logger.Log(t, "checking that the connection is not successful because there's no intention")
ctx := c.appCtx()
opts := ctx.KubectlOptions(t)
if c.Cfg.EnableTransparentProxy {
k8s.CheckStaticServerConnectionFailing(t, c.Ctx.KubectlOptions(t), StaticClientName, "http://static-server")
k8s.CheckStaticServerConnectionFailing(t, opts, StaticClientName, "http://static-server")
} else {
k8s.CheckStaticServerConnectionFailing(t, c.Ctx.KubectlOptions(t), StaticClientName, "http://localhost:1234")
k8s.CheckStaticServerConnectionFailing(t, opts, StaticClientName, "http://localhost:1234")
}
}

Expand All @@ -177,11 +214,13 @@ func (c *ConnectHelper) CreateIntention(t *testing.T) {
// static-client pod once the intention is set.
func (c *ConnectHelper) TestConnectionSuccess(t *testing.T) {
logger.Log(t, "checking that connection is successful")
ctx := c.appCtx()
opts := ctx.KubectlOptions(t)
if c.Cfg.EnableTransparentProxy {
// todo: add an assertion that the traffic is going through the proxy
k8s.CheckStaticServerConnectionSuccessful(t, c.Ctx.KubectlOptions(t), StaticClientName, "http://static-server")
k8s.CheckStaticServerConnectionSuccessful(t, opts, StaticClientName, "http://static-server")
} else {
k8s.CheckStaticServerConnectionSuccessful(t, c.Ctx.KubectlOptions(t), StaticClientName, "http://localhost:1234")
k8s.CheckStaticServerConnectionSuccessful(t, opts, StaticClientName, "http://localhost:1234")
}
}

Expand All @@ -192,8 +231,11 @@ func (c *ConnectHelper) TestConnectionFailureWhenUnhealthy(t *testing.T) {
// Test that kubernetes readiness status is synced to Consul.
// Create a file called "unhealthy" at "/tmp/" so that the readiness probe
// of the static-server pod fails.
ctx := c.appCtx()
opts := ctx.KubectlOptions(t)

logger.Log(t, "testing k8s -> consul health checks sync by making the static-server unhealthy")
k8s.RunKubectl(t, c.Ctx.KubectlOptions(t), "exec", "deploy/"+StaticServerName, "--", "touch", "/tmp/unhealthy")
k8s.RunKubectl(t, opts, "exec", "deploy/"+StaticServerName, "--", "touch", "/tmp/unhealthy")

// The readiness probe should take a moment to be reflected in Consul,
// CheckStaticServerConnection will retry until Consul marks the service
Expand All @@ -205,20 +247,20 @@ func (c *ConnectHelper) TestConnectionFailureWhenUnhealthy(t *testing.T) {
// other tests.
logger.Log(t, "checking that connection is unsuccessful")
if c.Cfg.EnableTransparentProxy {
k8s.CheckStaticServerConnectionMultipleFailureMessages(t, c.Ctx.KubectlOptions(t), StaticClientName, false, []string{
k8s.CheckStaticServerConnectionMultipleFailureMessages(t, opts, StaticClientName, false, []string{
"curl: (56) Recv failure: Connection reset by peer",
"curl: (52) Empty reply from server",
"curl: (7) Failed to connect to static-server port 80: Connection refused",
}, "", "http://static-server")
} else {
k8s.CheckStaticServerConnectionMultipleFailureMessages(t, c.Ctx.KubectlOptions(t), StaticClientName, false, []string{
k8s.CheckStaticServerConnectionMultipleFailureMessages(t, opts, StaticClientName, false, []string{
"curl: (56) Recv failure: Connection reset by peer",
"curl: (52) Empty reply from server",
}, "", "http://localhost:1234")
}

// Return the static-server to a "healthy state".
k8s.RunKubectl(t, c.Ctx.KubectlOptions(t), "exec", "deploy/"+StaticServerName, "--", "rm", "/tmp/unhealthy")
k8s.RunKubectl(t, opts, "exec", "deploy/"+StaticServerName, "--", "rm", "/tmp/unhealthy")
}

// helmValues uses the Secure and AutoEncrypt fields to set values for the Helm
Expand Down
20 changes: 18 additions & 2 deletions acceptance/framework/environment/environment.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,10 @@ import (
)

const (
DefaultContextName = "default"
SecondaryContextName = "secondary"
DefaultContextName = "default"
SecondaryContextName = "secondary"
AppContextName = "app-default"
SecondaryAppContextName = "app-secondary"
)

// TestEnvironment represents the infrastructure environment of the test,
Expand Down Expand Up @@ -59,6 +61,20 @@ func NewKubernetesEnvironmentFromConfig(config *config.TestConfig) *KubernetesEn
kenv.contexts[SecondaryContextName] = NewContext(config.SecondaryKubeNamespace, config.SecondaryKubeconfig, config.SecondaryKubeContext)
}

// Optionally, deploy apps into a separate namespace.
pglass marked this conversation as resolved.
Show resolved Hide resolved
// Maybe not the right place for this?
pglass marked this conversation as resolved.
Show resolved Hide resolved
if config.AppNamespace != "" {
kenv.contexts[AppContextName] = NewContext(config.AppNamespace, config.Kubeconfig, config.KubeContext)
if config.EnableMultiCluster {
kenv.contexts[SecondaryAppContextName] = NewContext(config.SecondaryAppNamespace, config.SecondaryKubeconfig, config.SecondaryKubeContext)
}
} else {
kenv.contexts[AppContextName] = defaultContext
if config.EnableMultiCluster {
kenv.contexts[SecondaryAppContextName] = kenv.contexts[SecondaryContextName]
}
}

return kenv
}

Expand Down
17 changes: 15 additions & 2 deletions acceptance/framework/flags/flags.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,18 @@ type TestFlags struct {
flagSecondaryKubecontext string
flagSecondaryNamespace string

flagAppNamespace string
flagSecondaryAppNamespace string

flagEnableEnterprise bool
flagEnterpriseLicense string

flagEnableOpenshift bool

flagEnablePodSecurityPolicies bool

flagEnableCNI bool
flagEnableCNI bool
flagCNINamespace string

flagEnableTransparentProxy bool

Expand Down Expand Up @@ -75,6 +79,9 @@ func (t *TestFlags) init() {
"the context set as the current context will be used by default.")
flag.StringVar(&t.flagNamespace, "namespace", "", "The Kubernetes namespace to use for tests.")

flag.StringVar(&t.flagAppNamespace, "app-namespace", "", "The Kubernetes namespace where mesh services should be deployed.")
flag.StringVar(&t.flagSecondaryAppNamespace, "secondary-app-namespace", "", "The secondary Kubernetes namespace where mesh services should be deployed [multi-cluster only].")

flag.StringVar(&t.flagConsulImage, "consul-image", "", "The Consul image to use for all tests.")
flag.StringVar(&t.flagConsulK8sImage, "consul-k8s-image", "", "The consul-k8s image to use for all tests.")
flag.StringVar(&t.flagConsulDataplaneImage, "consul-dataplane-image", "", "The consul-dataplane image to use for all tests.")
Expand Down Expand Up @@ -112,6 +119,8 @@ func (t *TestFlags) init() {
flag.BoolVar(&t.flagEnableCNI, "enable-cni", false,
"If true, the test suite will run tests with consul-cni plugin enabled. "+
"In general, this will only run against tests that are mesh related (connect, mesh-gateway, peering, etc")
flag.StringVar(&t.flagCNINamespace, "cni-namespace", "",
"If configured, the CNI will be deployed in this namespace.")

flag.BoolVar(&t.flagEnableTransparentProxy, "enable-transparent-proxy", false,
"If true, the test suite will run tests with transparent proxy enabled. "+
Expand Down Expand Up @@ -172,14 +181,18 @@ func (t *TestFlags) TestConfigFromFlags() *config.TestConfig {
SecondaryKubeContext: t.flagSecondaryKubecontext,
SecondaryKubeNamespace: t.flagSecondaryNamespace,

AppNamespace: t.flagAppNamespace,
SecondaryAppNamespace: t.flagSecondaryAppNamespace,

EnableEnterprise: t.flagEnableEnterprise,
EnterpriseLicense: t.flagEnterpriseLicense,

EnableOpenshift: t.flagEnableOpenshift,

EnablePodSecurityPolicies: t.flagEnablePodSecurityPolicies,

EnableCNI: t.flagEnableCNI,
EnableCNI: t.flagEnableCNI,
CNINamespace: t.flagCNINamespace,

EnableTransparentProxy: t.flagEnableTransparentProxy,

Expand Down
2 changes: 2 additions & 0 deletions acceptance/tests/connect/connect_external_servers_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ func TestConnectInject_ExternalServers(t *testing.T) {
caseName := fmt.Sprintf("secure: %t", secure)
t.Run(caseName, func(t *testing.T) {
cfg := suite.Config()
cfg.SkipWhenOpenshiftAndCNI(t)

ctx := suite.Environment().DefaultContext(t)

serverHelmValues := map[string]string{
Expand Down
2 changes: 2 additions & 0 deletions acceptance/tests/connect/connect_inject_namespaces_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ func TestConnectInjectNamespaces(t *testing.T) {
if !cfg.EnableEnterprise {
t.Skipf("skipping this test because -enable-enterprise is not set")
}
cfg.SkipWhenOpenshiftAndCNI(t)

cases := []struct {
name string
Expand Down Expand Up @@ -246,6 +247,7 @@ func TestConnectInjectNamespaces_CleanupController(t *testing.T) {
if !cfg.EnableEnterprise {
t.Skipf("skipping this test because -enable-enterprise is not set")
}
cfg.SkipWhenOpenshiftAndCNI(t)

consulDestNS := "consul-dest"
cases := []struct {
Expand Down
Loading