Skip to content

Commit

Permalink
Update docs to reflect removal of SANDBOX log level and replacement T…
Browse files Browse the repository at this point in the history
…RACE level.
  • Loading branch information
craigwalton-dsit committed Jan 3, 2025
1 parent 1ce0891 commit e2becef
Show file tree
Hide file tree
Showing 2 changed files with 20 additions and 31 deletions.
37 changes: 13 additions & 24 deletions docs/docs/tips/debugging-k8s-sandboxes.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,55 +4,44 @@ This section explains features of [Inspect](https://inspect.ai-safety-institute.
and [k9s](https://k9scli.io/) which are particularly relevant to debugging evals which
use K8s sandboxes. Please see the dedicated docs pages of each for more information.

## Capture Inspect `SANDBOX`-level logs { #sandbox-log-level }
## View Inspect's `TRACE`-level logs { #trace-log-level }

Useful sandbox-related messages like Helm installs/uninstalls, pod operations (`exec()`
Useful sandbox-related messages like Helm installs/uninstalls, Pod operations (`exec()`
executions including the result, `read_file()`, `write_file()`) etc. are logged at the
`SANDBOX` log level.
`TRACE` log level. See the [Inspect tracing
docs](https://inspect.ai-safety-institute.org.uk/tracing.html) for more information on
where these are stored and how to read them.

Set Inspect's log level to `SANDBOX` or lower via one of these methods:

* passing `--log-level sandbox` on the command line
* setting `INSPECT_LOG_LEVEL=sandbox` environment variable
* passing the `log_level` argument to `eval()` or `eval_set()`

Example:
Example (additional fields removed for brevity):

```raw
SANDBOX - K8S: Installing Helm chart. {
K8s installing Helm chart: {
"chart": "/home/ubuntu/.../k8s_sandbox/resources/helm/agent-env",
"release": "uo4w7mvq",
"values": "/home/ubuntu/.../helm-values.yaml",
"namespace": "agent",
"task": "xss-attack"
}
SANDBOX - K8S: Available sandboxes: ['default', 'default', 'victim']
SANDBOX - K8S: Starting: Execute command in pod. {
[K8s] Available sandboxes: ['default', 'default', 'victim']
K8s execute command in Pod: {
"pod": "agent-env-uo4w7mvq-default-0",
"task_name": "xss-attack", "cmd": "['python3']",
"stdin": "print('Hello, world!')", "cwd": "None", "timeout": "300"
}
SANDBOX - K8S: Completed: Execute command in pod. {
[K8s] Completed: K8s execute command in Pod. {
"result": "ExecResult(success=True, returncode=0, stdout=\"...\", stderr\"\")"
"pod": "agent-env-uo4w7mvq-attacker-0", "task_name": "xss-attack",
"cmd": "['python3']", "stdin": "print('Hello, world!')", "cwd": "None",
"timeout": "300"
}
```

Additionally, ensure the content of the `logging` module is written to a file on disk:
All K8s-relevant entries contain "K8s" as a substring within the "action" field which
may be useful for filtering.

```sh
mkdir -p logs
export INSPECT_PY_LOGGER_FILE="logs/inspect_py_log.log"
```

These will include timestamps and are invaluable when piecing together an ordered
The trace logs include timestamps and are invaluable when piecing together an ordered
sequence of events.

Consider including the datetime or other identifier in the log file name to keep logs
separate.

## Disabling Inspect Cleanup

By default, Inspect will clean up sandboxes (i.e. uninstall Helm releases) after an eval
Expand Down
14 changes: 7 additions & 7 deletions docs/docs/tips/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@
For general K8s and Inspect sandbox debugging, see the [Debugging K8s
Sandboxes](debugging-k8s-sandboxes.md) guide.

## Capture Inspect `SANDBOX`-level logs
## View Inspect's `TRACE`-level logs

A good starting point to most issues is to capture the output of the Python `logging`
module at `SANDBOX` level. See the [`SANDBOX` log level
section](debugging-k8s-sandboxes.md#sandbox-log-level).
A good starting point to many issues is to view the `TRACE`-level logs written by
Inspect. See the [`TRACE` log level
section](debugging-k8s-sandboxes.md#trace-log-level).

## I'm seeing "Helm install: context deadline exceeded" errors

Expand All @@ -23,8 +23,8 @@ Therefore, this error can be an indication of:

Try installing the chart again (this can also be [done
manually](../helm/built-in-chart.md#manual-chart-install)) and check the Pod statuses
and logs using a tool like K9s. Use the helm release name (will be in error message and
`SANDBOX` -level logs) to filter the Pods.
and logs using a tool like K9s. Use the helm release name (will be in error message) to
filter the Pods.

## I'm seeing "Helm uninstall failed" errors

Expand Down Expand Up @@ -80,7 +80,7 @@ kubectl get events --sort-by='.metadata.creationTimestamp' \
--field-selector involvedObject.name=agent-env-xxxxxxxx-default-0
```

Find the Pod name (including the random 8-character identifier) in the `SANDBOX`-level
Find the Pod name (including the random 8-character identifier) in the `TRACE`-level
logs or the stack trace.

To specify a namespace other than the default, use the `-n` flag.

0 comments on commit e2becef

Please sign in to comment.