antctl query

[jakesokol1]

Describe what you are trying to solve
It is difficult to track network policies for very large clusters. To determine which policies apply to any given endpoint, a user must examine the entire set of network policies and the entire set of selectors for each policy to find a match. This represents a burden for large scale kubernetes clusters with complex policy structures, and one which has been addressed in some other CNI solutions. To fix this issue for developers choosing between solutions, and to provide support for current users of Antrea, we suggest a solution in the form of a subcommand of antctl which allows users to filter for relevant resources through more sophisticated querying capabilities.

Describe the solution you have in mind + user flow
antctl query endpoint --pod --namespace

User selects a network endpoint (for now, using pod and namespace). Relevant network policies (to the selected endpoint) are returned to the user in three groups:

• Applied policies are policies which directly apply to the endpoint.

• Ingress rules show the policies which reference the endpoint in at least one ingress rule.

• Egress rules show the policies which reference the endpoint in at least one egress rule.

Here is a sample return object which shows the return fields of a typical response.

(Note, Index represents the index of the rule (relative to the policy containing it) which references the endpoint.

Support may be added in the future for “antctl query networkpolicy”, for the reverse query, i.e. to find out which endpoints are affected by a specific network policy.

Describe the main design/architecture of your solution
Processing: The process of filtering network policies relevant to the selected endpoint occurs in controller_query_replier. EndpointQuerier is initialized with a pointer to NetworkPolicyController. New indexers are added to the NetworkPolicyController stores (appliedToGroups, addressGroups), to allow indexing by endpoint (podIndex). This allows the querier to have a more direct mapping from endpoint directly to applied policy.

appliedToGroupStore.GetByIndex() → networkPolicyStore.GetByIndex() → network policies

For processing policies relevant for the rules sections (ingress + egress), some additional filtering is needed in the querier.

addressGroupStore.GetByIndex() → networkPolicyStore.GetByIndex() → potential network policies

Potential network policies need to be further filtered to see if they belong in the ingress or egress rules section. This is performed by iterating through the rules in the To and From section for each network to see if it matches the addressGroup in question.

E2e interface flow:

antctl → (http) handler → querier replier → internal stores → (back through http to antctl) → output transform → stdout

Antctl changes: in antctl, we create a new subcommand (antctl query), and implement transforms and output functions (tableOutputForQueryCommand) in the same pattern as the existing commands and output functions (for transform look in transforms/endpoint).

Handler creation: we create a handler installation and handler function in the same pattern as in the agent to handle http requests to the controller (Note, this requires installing handlers to the apiserver of the controller, which wasn’t done previously). Http requests from antctl invoke the handler to perform the filtering processes in the querier replier. Look into controller/apiserver for handler function and installation methods and cmd/antrea-controller/controller.go for installation of new http endpoint.

Querier Replier: we create a querier replier object for centralized filtering operations for antctl query. This is the source of actual policy processing, it is instantiated along with the controller and has a pointer to the NetworkPolicyController for internal store access.

Alternative solutions that you considered
We have considered not using indexing and instead filtering at the time of each call to query (instead of getting by indexing). This requires iterating through all policies and matching appropriate policies based off of their selectors for each call to query. We found this approach high in complexity (for the implementation) and slow at scale.

Test plan

• handler_test.go

• controller_query_test.go

• controller_query_perf_test.go

• networkpolicy_controller_perf_test.go

In networkpolicy_controller_perf_test.go TestInitXLargeScaleWithOneNamespaces tests the execution time and the memory usage of computing a scale of 1 Namespaces, 10k NetworkPolicies, 10k Pods where each network policy selects each pod (applied + ingress). This test is important for testing memory effects of indexing.

Potential Issues
My main concern of our implementation is the use of indexers in the core networkpolicy, addressGroups, and appliedToGroups stores. I have tested memory usage for a variety of cases of controller initialization and have found marginal increases in memory usage. Though, I believe that this is an important note, as it is the only part of the implementation which requires any significant changes in underlying Antrea processes, as other aspects of the implementation are disjoint from normal operation.

Open Questions
I would appreciate feedback on all aspects of the implementation. I think some areas of improvement could be the response structure / presentation to the user, the selection mechanism for endpoints (perhaps something other than --pod and --namespace), and ideas for command flags.

Future Work
To determine which policies are relevant to a given policy selects, a user must examine the entire set of endpoints to see which labels match the policy selector. For future work, we propose antctl query network policy, which would work in the reverse direction of antctl query endpoint, returning relevant endpoints instead of relevant network policies in the context of a network policy instead of an endpoint (Note, a version of this is already supported in Octant, look into network policy visualization).

The tool can expand to support a more abstract selection capable of selecting a set of endpoints instead of a single endpoint, only using pod and namespace arguments. We are currently evaluating multiple designs, and are considering the possibility of utilizing labels towards this end.

Antctl query’s ability to filter large sets of kubernetes resources may be able to apply to visualization tools, such as Octant’s NetworkPolicy visualization. This could give a more centralized policy tracking experience, along with encouraging migration between services.