Where to find the created snapshot

[felikjunvianto]

Hi, sorry if the question is quite dummy, I am still new in Kubernetes thing. Where can I view the created snapshots? My first impression is I can view it under Snapshots menu in GCE (https://console.cloud.google.com/compute/snapshots), but I can't find any.

Also, could it be the case that my deployment failed to create snapshot? I found this Exception logged on created k8s-snapshots pod.

2017-12-30T17:29:06.303183Z watch-resources.worker.error   [k8s_snapshots.kube] message=watch-resources.worker.error resource_type_name=PersistentVolume severity=ERROR
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 99, in raise_for_status
    resp.raise_for_status()
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://10.15.240.1:443/api/v1/persistentvolumes?watch=true

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/kube.py", line 181, in worker
    for event in sync_iterator:
  File "/usr/local/lib/python3.6/site-packages/pykube/query.py", line 156, in object_stream
    self.api.raise_for_status(r)
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 106, in raise_for_status
    raise HTTPError(resp.status_code, payload["message"])
pykube.exceptions.HTTPError: persistentvolumes is forbidden: User "system:serviceaccount:core:default" cannot watch persistentvolumes at the cluster scope: Unknown user "system:serviceaccount:core:default"
2017-12-30T17:29:08.295234Z watch-resources.worker.error   [k8s_snapshots.kube] message=watch-resources.worker.error resource_type_name=PersistentVolumeClaim severity=ERROR
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 99, in raise_for_status
    resp.raise_for_status()
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://10.15.240.1:443/api/v1/persistentvolumeclaims?watch=true

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/kube.py", line 181, in worker
    for event in sync_iterator:
  File "/usr/local/lib/python3.6/site-packages/pykube/query.py", line 156, in object_stream
    self.api.raise_for_status(r)
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 106, in raise_for_status
    raise HTTPError(resp.status_code, payload["message"])
pykube.exceptions.HTTPError: persistentvolumeclaims is forbidden: User "system:serviceaccount:core:default" cannot watch persistentvolumeclaims at the cluster scope: Unknown user "system:serviceaccount:core:default"
2017-12-30T17:29:09.293575Z watch-resources.worker.error   [k8s_snapshots.kube] message=watch-resources.worker.error resource_type_name=SnapshotRule severity=ERROR
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 99, in raise_for_status
    resp.raise_for_status()
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://10.15.240.1:443/apis/k8s-snapshots.elsdoerfer.com/v1/snapshotrules?watch=true

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/kube.py", line 181, in worker
    for event in sync_iterator:
  File "/usr/local/lib/python3.6/site-packages/pykube/query.py", line 156, in object_stream
    self.api.raise_for_status(r)
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 106, in raise_for_status
    raise HTTPError(resp.status_code, payload["message"])
pykube.exceptions.HTTPError: snapshotrules.k8s-snapshots.elsdoerfer.com is forbidden: User "system:serviceaccount:core:default" cannot watch snapshotrules.k8s-snapshots.elsdoerfer.com at the cluster scope: Unknown user "system:serviceaccount:core:default"

[miracle2k]

The place to look for the snapshots is the right one. It does not look to me like backups are being created; It seems the error message is "Unknown user "system:serviceaccount:core:default"". I am not sure about this; the permissions might be off. Can you find something helpful in this thread maybe? Also have a look at the rbac definition file that was recently added to the repository.

kubernetes/dashboard#2326

[felikjunvianto]

Thank you for the response. I will take a look and post my finding soon.

[felikjunvianto]

Hi @miracle2k , the link was very helpful, thank you very much. I supposed the fix is I need to create new clusterrolebinding for not existing user (as mentioned in kubernetes/dashboard#2326 (comment)).