Start tracking k8s liveness probes #1320
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mstemm
force-pushed
the
identify-k8s-liveness-probes
branch
from
a657c17 to
83af9e0
Compare
mstemm
force-pushed
the
identify-k8s-liveness-probes
branch
from
8aa0a4a to
5a79c19
Compare
anoop-sysd
reviewed
Feb 26, 2019
mstemm
force-pushed
the
identify-k8s-liveness-probes
branch
from
db912c3 to
34b74bd
Compare
|
@gnosek I think this is ready to merge now, can you take a look? |
gnosek
reviewed
Mar 1, 2019
|
I'm going to wait for #1326 to be merged before making additional updates. They both change the same container_engine code and I'd rather resolve conflicts just once. |
mstemm
force-pushed
the
identify-k8s-liveness-probes
branch
4 times, most recently
from
ffd575a to
800ebd8
Compare
|
This is ready for review again @gnosek could you take another look? |
mstemm
force-pushed
the
identify-k8s-liveness-probes
branch
from
800ebd8 to
1cc9055
Compare
gnosek
approved these changes
May 10, 2019
mstemm
force-pushed
the
identify-k8s-liveness-probes
branch
2 times, most recently
from
1a30b95 to
555deed
Compare
Not used in this header file, so shouldn't be needed.
K8s has a similar but not identical method as docker for container
health checks. They are called liveness/readiness probes and are a part
of the pod specification, and not a part of the image.
Luckily, the pod configuration *is* a part of the container metadata as
stringified json, with a label
"annotation.kubectl.kubernetes.io/last-applied-configuration", so we can
use that label to identify liveness/readiness probes.
New methods in the docker container resolver handle parsing the pod
specification (and healthcheck info) out of the container json and
creating health probes from them.
A new class sinsp_container_info::container_health_probe represents one
of these health probes. It has a probe
type (healthcheck/liveness/readiness), the executable and arguments, and
methods to serialize/unserialize from json. The serialization doesn't
preserve the original container json--they only keep the exe + args.
The container info now has a list of possible health probe objects and
iterates over them when dumping the container to json.
For threads, switch everything to use a threadinfo category instead of a
simple bool for has healthcheck. The possible values for the category
are:
- CAT_NONE: no specific category
- CAT_CONTAINER: a process run in a container and *not* any
of the following more specific categories.
- CAT_HEALTHCHECK: part of a container healthcheck
- CAT_LIVENESS_PROBE: part of a k8s liveness probe
- CAT_READINESS_PROBE: part of a k8s readiness probe
Identify_healthcheck becomes identify_category() but
otherwise behaves the same (passing categories down and checking the
args list otherwise).
The logic in indentify_healthcheck tries to handle the common cases
first:
- not running in a container or container info not present: CAT_NONE
- vpid=1: CAT_CONTAINER
- inherit categories other than CAT_NONE directly from parent
If those fail, the more expensive steps of matching against the health
check args and possibly traversing the parent state are done.
The filterchecks aren't quite as generic as the threadinfo categories to
keep the filtering simple. A new field
proc.is_container_{liveness,readiness}_probe checks for k8s
liveness/readiness probes, and container.{liveness,readiness}_probe
prints the exe + args.
mstemm
force-pushed
the
identify-k8s-liveness-probes
branch
from
555deed to
21f1139
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
K8s has a similar but not identical method as docker for container
health checks. They are called liveness probes and are a different
property of the container json.
Read both out of the container json, load/save them both when writing
to/from container json information events, and use a shared "health
probe" command and args.
For threads, now that we have two properties, switch everything to use a
threadinfo category instead of a simple bool for has healthcheck. The
possible values for the category are CAT_NONE, CAT_HEALTHCHECK, and
CAT_LIVENESS_PROBE. identify_healthcheck becomes identify_category() but
otherwise behaves the same (passing categories down and checking the
args list otherwise).
The filterchecks aren't quite as generic as the threadinfo categories to
keep the filtering simple. A new field proc.is_container_liveness_probe
checks for k8s liveness probes, and container.liveness_probe prints the
exe + args.