grafana-prom-import-dashboards pod always fail in seldon-core-analytics chat

[theone4ever]

Describe the bug

I am using Seldon Core 1.0.2 as I need MAB feature. So I also installed seldon-core-analytics with following command:

helm install seldon-core-analytics seldon-core-analytics --repo https://storage.googleapis.com/seldon-charts --set persistence.enabled=false --namespace seldon-system --version 1.0.2

In addition, I have Seldon-core 1.0.2 installed at the same seldon-system namespace and I am using istio as ingress.

When install Seldon-core-analytics, the grafana-prom-import-dashboards pod always failed, error the container log below"

Using grafana-prom:80
checking connection to grafana [1]
Sleeping 1 secs...
checking connection to grafana [2]
Sleeping 1 secs...
checking connection to grafana [3]
Sleeping 1 secs...
checking connection to grafana [4]
curl: (22) The requested URL returned error: 503 Service Unavailable
curl: (22) The requested URL returned error: 503 Service Unavailable

importing grafana-net-2115-dashboard.json
curl: (22) The requested URL returned error: 503 Service Unavailable

importing metrics-test-dashboard.json
curl: (22) The requested URL returned error: 503 Service Unavailable

importing predictions-analytics-dashboard.json
curl: (22) The requested URL returned error: 503 Service Unavailable

To reproduce

1. First install Seldon-core 1.0.2: helm install seldon-core seldon-core-operator --repo https://storage.googleapis.com/seldon-charts --set istio.enabled=true --set istio.gateway=istio-system/seldon-gateway --set usageMetrics.enabled=true --set executor.enabled=false --namespace seldon-system --version 1.0.2
2. Then install seldon-core-analytics 1.0.2: helm install seldon-core-analytics seldon-core-analytics --repo https://storage.googleapis.com/seldon-charts --set persistence.enabled=false --namespace seldon-system --version 1.0.2

Expected behaviour

Environment

seldon-core 1.0.2 with istio on Azure Kubernetes service
seldon-core-analytics 1.0.2

• Cloud Provider: Azure Kubernetes Service
• Kubernetes Cluster Version: v1.17.11
• Deployed Seldon System Images:

        value: docker.io/seldonio/engine:1.0.2
        image: docker.io/seldonio/seldon-core-operator:1.0.2

Model Details

[ryandawsonuk]

Looks rather like there's an istio configuration in your cluster blocking the dashboard load pod from connecting to grafana. Do you know how istio was installed and configured? Have you tried it without istio?

[axsaucedo]

@theone4ever do you have any updates? Closing for now, please reopen with further info if still an issue