kind + apisnoop

Setup

Feel free to modify **kind+apisnoop.yaml** to your needs.

It may take a while for snoopdb to download and process the ~900MB~ or so of audit logs from the most recent conformance runs.

This will allow you to compare the API operations your application or test hits to the existing conformance test coverage.

git clone https://github.com/cncf/apisnoop
cd apisnoop/kind
kind create cluster --config=kind+apisnoop.yaml
# can take a while, but will ensure your are working against fresh data
kubectl wait --for=condition=Ready --selector=app.kubernetes.io/name=auditlogger --timeout=600s pod
export PGUSER=apisnoop
export PGHOST=localhost
psql -c "select distinct useragent from testing.audit_event;"

Any software hitting the kubernetes apiserver will show up in the audit_event table.

This includes tests as they are being written, and to study software that is hitting API Operations that are as of yet, untested.

Sample queries

Info about an endpoint

APISnoop loads in kubernetes endpoint data from the open api spec, which gives a quick way to get info about any particular endpoint.

select endpoint, description, path
  from open_api
 where endpoint = 'getApiextensionsAPIGroup'
 -- get information from latest spec
   and release = (
    select release
      from open_api
     order by release::semver desc
     limit 1
    );

Or group, like all stable endpoints that are a part of networking

select endpoint, description
  from open_api
 where level = 'stable'
   and category =  'networking'
 -- get information from latest spec
   and release = (
    select release
      from open_api
     order by release::semver desc
     limit 1
    );

Coverage Information

APISnoop also loads in data from the most recent end-to-end test runs, which lets us see which endpoints in the api are conformance tested.

As an example, we can see those same GA networking endpoints, but now whether they are tested.

select endpoint,
       conf_tested as "conformance tested"
  from endpoint_coverage
 where level = 'stable'
    and category = 'networking'
   and release = (
    select release
      from open_api
     order by release::semver desc
     limit 1
    )
order by "conformance tested" desc;

Coverage information for your endpoints

Lastly, our auditlogger is tracking all interactions happening on the cluster you brought up and mapping it to the endpoints being hit. This lets us see whether an application running on our cluster is hitting untested endpoints.

A simple way to do this is to take a look at the useragents hitting the cluster (excluding the kubelet api itself)

select distinct useragent
  from testing.audit_event
  where useragent not like 'kube-%'
 group by useragent;

We can then combine some tables to see whether our useragent is hitting anything untested.

select live.endpoint, conf_tested
  from testing.audit_event live
  join endpoint_coverage using(endpoint)
 where useragent like 'coredns%'
   and endpoint_coverage.release = (
        select release
        from open_api
        order by release::semver desc
        limit 1
       )
 group by live.endpoint, conf_tested
 order by live.endpoint, conf_tested asc;

Helper queries

Reset the live audit events

Since the logger is tracking everyting, its audit_event table can get pretty big pretty fast. To delete everyting from it, you want to run

delete from testing.audit_event;

This will delete all events, while keeping all other relevant info and the auditlogger will continue to run. It can be useful if you are wanting to test something on the cluster to reset the list, run the test, check the events, and repeat.

Get info on our tables and views

We have a couple helper functions in the db, that are basically a wrapper on psql’s existing functions

To see a description of all the relations in the db, you can run

select * from describe_relations();

And to see all the columns, and their definition, for a given relation you can run

select * from describe_columns('public', 'endpoint_coverage');
-- first arg is the schema, second is the relation name

Envoy

Let’s see if we can get envoy to trigger some untested endpoints. It may be that Envoy itself doesn’t hit any k8s endpoints.

helm repo add stable https://kubernetes-charts.storage.googleapis.com/
helm install stable/envoy --name my-release
psql -c "select distinct useragent from testing.audit_event where useragent not like 'kube%' and useragent not like 'local-path%' and useragent not like 'coredns%' and useragent not like 'kindnetd%'"

select distinct useragent
  from testing.audit_event
 where useragent not like 'kube%'
   and useragent not like 'local-path%'
   and useragent not like 'coredns%'
   and useragent not like 'kindnetd%';

Logs

Not necessary, but may help in debugging.

Snoop DB

kubectl logs  --selector=app.kubernetes.io/name=auditlogger -c snoopdb -f

Auditlogger

kubectl logs  --selector=app.kubernetes.io/name=auditlogger -c auditlogger -f