Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SegmentStore failed to start during Helm chart installation of Pravega with "externalAccess.enabled=true" #204

Closed
deenav opened this issue May 31, 2019 · 2 comments · Fixed by #227 or #249
Assignees
Labels
kind/bug Something isn't working priority/P1 Recoverable error, functionality/performance impaired but not lost, no permanent damage version 0.4.0 Issue with Operator 0.4.0

Comments

@deenav
Copy link

deenav commented May 31, 2019

Problem Description
SegmentStore failed to start with error Unexpected service type . Exiting... when Pravega external connectivity deployed using helm method of installation $helm install charts/pravega --name bar --set zookeeperUri=pravega-zk-client:2181,externalAccess.enabled=true

This issue is same as pravega/pravega#3614, due to the service account that is configured in the SegmentStore pod doesn't have proper permissions when enabling external-connectivity.

deploynment details

Pravega version : 0.5.0-2236.5228e2d
Pravega Operator: pravega/pravega-operator:0.4.0
Zookeeper Operator : pravega/zookeeper-operator:0.2.1
# kubectl get po --all-namespaces
NAMESPACE     NAME                                                 READY   STATUS      RESTARTS   AGE
default       bar-pravega-bookie-0                                 1/1     Running     0          79s
default       bar-pravega-bookie-1                                 1/1     Running     0          79s
default       bar-pravega-bookie-2                                 1/1     Running     0          78s
default       bar-pravega-pravega-controller-6d7fc79-lx8zf         1/1     Running     0          79s
default       bar-pravega-pravega-segmentstore-0                   0/1     Error       2          78s
default       foo-pravega-operator-778ff44c8-wpmvv                 1/1     Running     0          2m56s
default       pravega-benchmark                                    1/1     Running     0          16d
default       pravega-zk-0                                         1/1     Running     0          127m
default       pravega-zk-1                                         1/1     Running     0          126m
default       pravega-zk-2                                         1/1     Running     0          126m
default       shaka-zulu-nfs-client-provisioner-59d7f8f84c-tvh7w   1/1     Running     0          17d

# kubectl get svc
NAME                                        TYPE           CLUSTER-IP       EXTERNAL-IP                  PORT(S)                          AGE
bar-pravega-bookie-headless                 ClusterIP      None             <none>                       3181/TCP                         2m16s
bar-pravega-pravega-controller              LoadBalancer   10.100.200.55    10.240.119.88,100.64.64.69   10080:31364/TCP,9090:31791/TCP   2m16s
bar-pravega-pravega-segmentstore-0          LoadBalancer   10.100.200.139   10.240.119.89,100.64.64.69   12345:30431/TCP                  2m15s
bar-pravega-pravega-segmentstore-headless   ClusterIP      None             <none>                       12345/TCP                        2m16s
kubernetes                                  ClusterIP      10.100.200.1     <none>                       443/TCP                          29d
pravega-admission-webhook                   ClusterIP      10.100.200.241   <none>                       443/TCP                          3m46s
pravega-zk-client                           ClusterIP      10.100.200.46    <none>                       2181/TCP                         128m
pravega-zk-headless                         ClusterIP      None             <none>                       2888/TCP,3888/TCP                128m


Segmentstore logs

# kubectl logs -f bar-pravega-pravega-segmentstore-0
Waiting for bar-pravega-bookie-0.bar-pravega-bookie-headless.default:3181
Connected to bar-pravega-bookie-0.bar-pravega-bookie-headless.default:3181
Waiting for bar-pravega-bookie-1.bar-pravega-bookie-headless.default:3181
Connected to bar-pravega-bookie-1.bar-pravega-bookie-headless.default:3181
Waiting for bar-pravega-bookie-2.bar-pravega-bookie-headless.default:3181
Connected to bar-pravega-bookie-2.bar-pravega-bookie-headless.default:3181
Running in a Kubernetes environment and managed by the Pravega Operator with external access enabled
Unexpected service type . Exiting...

Problem location
Helm Chart with externalAccess.enabled=true doesn't create a new service account, role, and role binding, with minimum required permissions to obtain the external address, and configure and enable it on the PravegaCluster manifest. Ref

@sumit-bm sumit-bm added kind/bug Something isn't working priority/P1 Recoverable error, functionality/performance impaired but not lost, no permanent damage labels May 31, 2019
@pbelgundi pbelgundi self-assigned this Jun 4, 2019
@pbelgundi pbelgundi assigned pbelgundi and SrishT and unassigned pbelgundi Jun 14, 2019
@pbelgundi pbelgundi added the version 0.4.0 Issue with Operator 0.4.0 label Jul 2, 2019
@pbelgundi
Copy link
Contributor

pbelgundi commented Aug 7, 2019

I'm still seeing this issue with using latest chart from master. So re-opening this.
When I create a new pravega cluster with external access enabled using this chart I see sgement store going into CrashLoopBackOff state..
Also serviceAccount for the segment store pod is set to 'default' instead of pravega-components

pravega-pravega-pravega-segmentstore-0                    0/1     CrashLoopBackOff   8          20m
apiVersion: v1
kind: Pod
metadata:
  annotations:
    pravega.version: 0.5.0-2303.7b5f02f
  creationTimestamp: "2019-08-07T06:11:05Z"
  generateName: pravega-pravega-pravega-segmentstore-
  labels:
    app: pravega-cluster
    component: pravega-segmentstore
. . .
restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: pravega-segmentstore
  terminationGracePeriodSeconds: 30
. . .

@pbelgundi pbelgundi reopened this Aug 7, 2019
@pbelgundi
Copy link
Contributor

segmentstore pod logs show this:

pbelgundi@ubuntu:~/deploy_external$ k logs -f pravega-pravega-pravega-segmentstore-0 
Waiting for pravega-pravega-bookie-0.pravega-pravega-bookie-headless.default:3181
Connected to pravega-pravega-bookie-0.pravega-pravega-bookie-headless.default:3181
Waiting for pravega-pravega-bookie-1.pravega-pravega-bookie-headless.default:3181
Connected to pravega-pravega-bookie-1.pravega-pravega-bookie-headless.default:3181
Waiting for pravega-pravega-bookie-2.pravega-pravega-bookie-headless.default:3181
Connected to pravega-pravega-bookie-2.pravega-pravega-bookie-headless.default:3181
Running in a Kubernetes environment and managed by the Pravega Operator with external access enabled
Unexpected service type . Exiting...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working priority/P1 Recoverable error, functionality/performance impaired but not lost, no permanent damage version 0.4.0 Issue with Operator 0.4.0
Projects
None yet
4 participants