Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs release 1.8.0 csi 3573 external log collection (PR2) #418

Merged
Merged
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
2cb8e29
formatting updates
rivkap Nov 21, 2021
022853f
started adding in information from ticket
rivkap Nov 21, 2021
213e986
wiki updates
rivkap Nov 25, 2021
607d1c4
xref linking trial
rivkap Nov 25, 2021
eff7104
pathing updates
rivkap Nov 25, 2021
3b1ed65
summary.md typo fixes
rivkap Nov 25, 2021
8bd5c2b
typo correction
rivkap Nov 25, 2021
45f97df
log status collect updates
rivkap Dec 8, 2021
9f29bc2
spacing updates
rivkap Dec 8, 2021
9dfac0f
language updates per acrolinx
rivkap Dec 8, 2021
5ba59ee
final updates per ticket
rivkap Dec 9, 2021
3ddf088
Merge branch 'docs_release-1.8.0' into docs_release-1.8.0-CSI-3573_ex…
rivkap Dec 9, 2021
a8e73f7
update default to namespace placeholder
rivkap Dec 12, 2021
fbf20a6
updated /tmp/logs_collection to logs/
rivkap Dec 12, 2021
f21f2eb
updated leftover -l csi to product=ibm-block-csi-driver
rivkap Dec 12, 2021
28b15e4
fixed logs/
rivkap Dec 12, 2021
9010806
combined examples and main command into one and added note
rivkap Dec 12, 2021
d069e00
typo fix
rivkap Dec 12, 2021
009c729
node pod typo fix
rivkap Dec 12, 2021
d020137
updates per meeting
rivkap Dec 12, 2021
282871e
typo fix
rivkap Dec 12, 2021
61ccbcc
updated per input
rivkap Dec 13, 2021
826a128
put wider collection commands under ibm support heading
rivkap Dec 13, 2021
aea080b
added pod and pvc_namespace
rivkap Dec 13, 2021
e7eb607
updated pvc bound status check
rivkap Dec 14, 2021
7f632ff
Merge branch 'docs_release-1.8.0' into docs_release-1.8.0-CSI-3573_ex…
rivkap Dec 14, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 29 additions & 45 deletions docs/content/troubleshooting/log_status_collect.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ To help pinpoint potential causes for stateful pod failure:

1. Verify that all CSI pods are running.
```
kubectl get pods -n <namespace> -l csi
kubectl get pods -n <namespace> -l product=ibm-block-csi-driver
```

2. If a pod is not in a _Running_ state, run the following command:
Expand All @@ -19,6 +19,7 @@ To help pinpoint potential causes for stateful pod failure:
```
View the logs.


## Status and log collection
To collect and display status and logs related to the different components of IBM® block storage CSI driver, use the Kubernetes commands that are found in this section.

Expand All @@ -29,9 +30,11 @@ For example:
```
mkdir logs
```
Save logs and status reports directly to the created directory by adding in the following string at the end of collection command: `> logs/<log_filename>`.
Save logs and status reports directly to the created directory by adding in the following string at the end of the collection command: `> logs/<log_filename>`.

**Important:** Be sure that the logs cover any relevant time frames for the specific issues that you are trying to debug when gathering logs from the storage system.

**Important:** Be sure that the logs cover any relevant timeframes for the specific issues that you are trying to debug when gathering logs from the storage system.
**Note:** All commands here are listed with the collection command with the `logs` folder name example. Change the folder name according as needed.

### General status and log collection for IBM Support
Be sure to run the following steps and copy the output to an external file, when engaging IBM Support and sending log collections.
Expand All @@ -41,81 +44,62 @@ Be sure to run the following steps and copy the output to an external file, when
`kubectl get nodes`
2. Check the CSI driver component status.

`kubectl get -n <namespace> pod -o wide | grep ibm-block-csi`
`kubectl get -all-namespaces pod -o wide | grep ibm-block-csi`
3. Check if the PersistentVolumeClaims (PVCs) are _Bound_.

`kubectl get pvc`
`kubectl get -n default pvc -o=jsonpath='{range .items[?(@.metadata.annotations.volume\.beta\.kubernetes\.io/storage-provisioner=="block.csi.ibm.com")]}{"PVC NAME: "}{@.metadata.name}{" PVC STATUS: "}{@.status.phase}{"\n"}{end}'`

The output should be similar to the following:

- If the PVCs are not in the _Bound_ state collect the events of all unbound PVCs. (See [Log collection for unbound PVCs](#log-collection-for-unbound-pvcs).)
`PVC NAME: demo-pvc-file-system PVC STATUS: Bound`

### Log collection for all CSI nodepods and their containers
**Note:** If the PVCs are not in the _Bound_ state collect the events of all unbound PVCs. (See [Log collection for unbound PVCs](#log-collection-for-unbound-pvcs).)

To collect logs for all nodepods, use the following commands:
#### Log collection for all CSI driver node pods and their containers

nodepods=`kubectl get -n <namespace> pod -l app.kubernetes.io/component=csi-node --output=jsonpath={.items..metadata.name}`
To collect logs for all CSI driver node pods, use the following commands:

nodepods=`kubectl get pods -l product=ibm-block-csi-driver -l app.kubernetes.io/component=csi-node --output=jsonpath={.items..metadata.name}`

for pod in $nodepods; kubectl logs -n <namespace> $pod -c $container > /tmp/logs_collection/${pod}_${container}.log;done;done
for pod in $nodepods;do for container in `kubectl get -n <namespace> pod $pod -o jsonpath='{.spec.containers[*].name}'`;do kubectl logs -n <namespace> $pod -c $container > logs/${pod}_${container}.log;done;done


### Log collection for all CSI controller containers
#### Log collection for all CSI controller containers

To collect logs for all pods and containers, use the following commands:
To collect logs for all controller containers, use the following commands:

for container in kubectl get -n default pod ibm-block-csi-controller-0 -o jsonpath='{.spec.containers[*].name}';do kubectl logs -n default ibm-block-csi-controller-0 -c $container > /tmp/logs_collection/ibm-block-csi-controller-0_${container}.log;done
for container in `kubectl get -n <namespace> pod ibm-block-csi-controller-0 -o jsonpath='{.spec.containers[*].name}'`;do kubectl logs -n <namespace> ibm-block-csi-controller-0 -c $container > logs/ibm-block-csi-controller-0_${container}.log;done


### Log collection for CSI operator logs
#### Log collection for CSI operator logs
To collect CSI operator logs, use the following commands:

operatorpod=`kubectl get pods --all-namespaces |grep ibm-block-csi-operator|awk '{print $2}'`
kubectl logs $operatorpod -n <namespace> > logs/operator


### Collecting details of all CSI objects and components

`kubectl describe all -l product=ibm-block-csi-driver -n <namespace>`

For example:

kubectl describe all -l csi -n <namespace> > logs/describe_ibm-block-csi-driver
`kubectl describe all -l product=ibm-block-csi-driver -n <namespace> > logs/describe_ibm-block-csi-driver`


### Status collection for CSI pods, daemonset, and statefulset
`kubectl get all -n <namespace> -l product=ibm-block-csi-driver`

For example:

kubectl get all -n <namespace> -l csi > get_all_ibm-block-csi-driver
`kubectl get all -n <namespace> -l product=ibm-block-csi-driver > logs/get_all_ibm-block-csi-driver`



### Log collection for the CSI driver controller
`kubectl log -f -n <namespace> ibm-block-csi-controller-0 -c ibm-block-csi-controller`

For example:

kubectl log -f -n <namespace> ibm-block-csi-controller-0 -c ibm-block-csi-controller > logs/ibm-block-csi-controller
`kubectl logs -f -n <namespace> ibm-block-csi-controller-0 -c ibm-block-csi-controller > logs/ibm-block-csi-controller`


### Log collection for the CSI driver node (per worker node or PODID)
`kubectl log -f -n <namespace> ibm-block-csi-node-<PODID> -c ibm-block-csi-node`

For example:

kubectl log -f -n <namespace> ibm-block-csi-node-<PODID> -c ibm-block-csi-node > logs/csi-node-<PODID>

`kubectl logs -f -n <namespace> ibm-block-csi-node-<PODID> -c ibm-block-csi-node > logs/csi-node-<PODID>`

### Log collection for unbound PVCs
`kubectl describe pvc <pvc-name>`

For example:

kubectl describe pvc <pvc-name> > logs/pvc_not_bounded
### Details collection for unbound PVCs
`kubectl describe -n <pvc_namespace> pvc <pvc-name> > logs/pvc_not_bounded`


### Log collection for pods not in the _Running_ state
`kubectl describe pod <not-running-pod-name>`

For example:

kubectl describe pod <pod-name> > logs/pod_not_running
### Details collection for pods not in the _Running_ state
`kubectl describe -n <pod_namespace> pod <not-running-pod-name> > logs/pod_not_running`