Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1862426: gather the audit logs for oauth apiserver #144

Merged
merged 4 commits into from
Sep 23, 2020
Merged

Bug 1862426: gather the audit logs for oauth apiserver #144

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7701ea2
gather_audit_logs: add logging and oauth-apiserver logs
sanchezl Mar 11, 2020
9ba1a64
gather_service_logs: limit service logs to one week
sanchezl Jul 31, 2020
7b3fff5
gather: stop gathering audit logs by default
sanchezl Aug 1, 2020
d3bd409
gather_service_logs: fix collection & add mco services
sanchezl Sep 6, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions collection-scripts/gather
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,6 @@ done
# Gather etcd information
/usr/bin/gather_etcd

# Collect System Audit Logs
/usr/bin/gather_audit_logs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can't do this until the e2e test works.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that e2e you're talking about already explicitly calls this script see https://github.com/openshift/origin/blob/1a83021f4dbc91de606a71d3a27cd80664aae26c/test/extended/cli/mustgather.go#L160 so it shouldn't block this PR


# Gather Service Logs (using a suplamental Script); Scoped to Masters.
/usr/bin/gather_service_logs master

Expand Down
61 changes: 25 additions & 36 deletions collection-scripts/gather_audit_logs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,40 +1,29 @@
#!/bin/bash
BASE_COLLECTION_PATH="/must-gather"
AUDIT_LOG_PATH="${BASE_COLLECTION_PATH}/audit_logs/"

### Helper Functions
function queue() {
local TARGET="${1}"
shift
local LIVE="$(jobs | wc -l)"
while [[ "${LIVE}" -ge 45 ]]; do
sleep 1
LIVE="$(jobs | wc -l)"
done
echo "${@}"
if [[ -n "${FILTER}" ]]; then
"${@}" | "${FILTER}" >"${TARGET}" &
else
"${@}" >"${TARGET}" &
fi
}

# Collect System Audit Logs
function collect_audit_logs { ### Takes an input of PATH
### (openshift-apiserver or kube-apiserver)

echo "WARNING: Collecting one or more audit logs on ALL masters in your cluster. This could take a large amount of time." >&2
mkdir -p ${AUDIT_LOG_PATH}/${1}
/usr/bin/oc adm node-logs --role=master --path=${1}/ | grep -v ".terminating" > ${AUDIT_LOG_PATH}/${1}.audit_logs_listing
while IFS=$'\n' read -r line; do
IFS=' ' read -ra log <<< "${line}"
FILTER=gzip queue ${AUDIT_LOG_PATH}/${1}/"${log[0]}"-"${log[1]}".gz /usr/bin/oc adm node-logs "${log[0]}" --path=${1}/"${log[1]}"
done < ${AUDIT_LOG_PATH}/${1}.audit_logs_listing
echo "INFO: Audit logs for $1 collected."
}

collect_audit_logs openshift-apiserver
collect_audit_logs kube-apiserver
# Downloads the audit.log (and its rotated copies) from
# /var/logs/{kube-apiserver,openshift-apiserver} on each
# master node.
BASE_COLLECTION_PATH="${BASE_COLLECTION_PATH:-/must-gather}"
echo "WARNING: Collecting one or more audit logs on ALL masters in your cluster. This could take a large amount of time." >&2
# the command executed by xargs below expects four parameters:
# $1 - node path under /var/logs to download
# $2 - local output path
# $3 - node name
# $4 - log file name
paths=(openshift-apiserver kube-apiserver oauth-apiserver)
for path in "${paths[@]}" ; do
output_dir="${BASE_COLLECTION_PATH}/audit_logs/$path"
mkdir -p "$output_dir"
oc adm node-logs --role=master --path="$path" | \
tee "${BASE_COLLECTION_PATH}/audit_logs/$path.audit_logs_listing" | \
grep -v ".terminating" | \
sed "s|^|$path $output_dir |"
done | \
xargs --max-args=4 --max-procs=45 bash -c \
'echo "INFO: Started downloading $1/$4 from $3";
oc adm node-logs $3 --path=$1/$4 | gzip > $2/$3-$4.gz;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add --since ${SINCE_TIMEFRAME} to this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sferich888 , I have added the --since parameter, but it doesn't seem to actually do anything.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The --since parameter is only applied to journal logs. It it ignored for --path logs, which are simply served by an instance of http.FileServer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What other ways can we limit the collection?

Is an awk script needed?

#!/usr/bin/gawk -f BEGIN { starttime = mktime(starttime) endtime = mktime(endtime) } func in_range(n, start, end) { return start <= n && n < end } match($0, /^([0-9]{4})-([0-9]{2})-([0-9]{2})\s/, m) && in_range(mktime(m[1] " " m[2] " " m[3] " 00 00 00"),

Source: https://askubuntu.com/questions/800769/how-to-get-text-from-range-of-dates-using-grep-sed-in-large-text-file

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sferich888 I think I can work something out, but as a followup.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In others scripts I see the full path of command, like:/usr/bin/oc

echo "INFO: Finished downloading $1/$4 from $3"' \
bash
echo "INFO: Audit logs collected."

# force disk flush to ensure that all data gathered is accessible in the copy container
sync
14 changes: 7 additions & 7 deletions collection-scripts/gather_service_logs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@ BASE_COLLECTION_PATH="/must-gather"
ROLES=${1:-master} ### Defaults to only collecting things from Masters

# Service Lists
GENERAL_SERVICES=(kubelet crio)
MASTER_SERVICES+=${2:-${GENERAL_SERVICES[@]}}
GENERAL_SERVICES=(kubelet crio machine-config-daemon-firstboot machine-config-daemon-host)
MASTER_SERVICES+=(${2:-"${GENERAL_SERVICES[@]}"})
MASTER_SERVICES+=() ### Placeholder to extend Master only services
NODE_SERVICES+=${2:-${GENERAL_SERVICES[@]}}
NODE_SERVICES+=(${2:-"${GENERAL_SERVICES[@]}"})
NODE_SERVICES+=() ### Placeholder to extend Node only services

# Collect System Service Logs
Expand All @@ -16,14 +16,14 @@ function collect_serivce_logs { ## Takes a node role input (master or worker)
PIDS=()
DIR_PATH=${SERVICE_LOG_PATH}/${1}s
echo "WARNING: Collecting one or more service logs on ALL linux $1 nodes in your cluster. This could take a large amount of time." >&2
mkdir -p ${DIR_PATH}
for service in ${NODE_SERVICES[@]}; do
mkdir -p "${DIR_PATH}"
for service in "${NODE_SERVICES[@]}"; do
echo "INFO: Collecting host service logs for $service"
/usr/bin/oc adm node-logs --role=$1 -l kubernetes.io/os=linux -u $service > ${DIR_PATH}/${service}_service.log &
/usr/bin/oc adm node-logs --role=$1 -l kubernetes.io/os=linux -u "$service" --since "${SINCE_TIMEFRAME:--7d}"> "${DIR_PATH}/${service}_service.log" &
PIDS+=($!)
done
echo "INFO: Waiting for worker host service log collection to complete ..."
wait ${PIDS[@]}
wait "${PIDS[@]}"
echo "INFO: Worker host service log collection to complete."
}

Expand Down