says "In the middle of a leadership election" only in minikube.

[medyagh]

I am using sarama library (1.11.0) for kafka in K8Guard .
I have both docker-compose and minikube version for k8guard. it used to work in both but now recently suddenly it stopped working in minikube.

I tried different version of minikube 18, and 20.

the error I get is

panic: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes.

the funny thing is when I log in to kafka contrainer and try to describe the leader, it says it has a leader 1001 and it says it is on partition 0. so for sure it DOES have a leader.

here is how my docker-compose looks like (which works)

---
services:
  cassandra:
    image: bitnami/cassandra
    environment:
      CASSANDRA_CLUSTER_NAME: cassandra
  zookeeper:
     image: wurstmeister/zookeeper
     ports:
       - "2181:2181"
  kafka:
    environment:
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
  memcached:
    image: "memcached:alpine"

version: "2"

---
services:
  k8guard-action:
    build:
      context: ../k8guard-action/
      dockerfile: Dockerfile-local
    external_links:
      - k8guardstartfromhere_kafka_1:kafka
      - k8guardstartfromhere_cassandra_1:cassandra
    networks:
      - k8guardstartfromhere_default
    env_file:
      - .env-creds
      - .env
    volumes:
      - ${HOME}/.kube:/root/.kube
      - ${HOME}/.minikube:${HOME}/.minikube
version: "2"
networks:
  k8guardstartfromhere_default:
    external: true

and here is how my minikube looks like (which stopped working)
only difference I have made is change kubernetes version to 1.6.3 on my minikube.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: k8guard-kafka
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: k8guard-kafka
    spec:
      containers:
      - name: zookeeper
        image: wurstmeister/zookeeper
        # image: satlank/microzookeeper:3.4.9-1
        ports:
        - containerPort: 2181
      - name: kafka
        image: wurstmeister/kafka
        env:
          - name: KAFKA_ZOOKEEPER_CONNECT
            value: "localhost:2181"
          - name: KAFKA_ADVERTISED_HOST_NAME
            value: "k8guard-kafka-service.default.svc.cluster.local"
          - name: KAFKA_ADVERTISED_PORT
            value: "9092"
          - name: KAFKA_CREATE_TOPICS
            value: "k8guard-to-action-k8s-lab:1:1"
        # image: satlank/microkafka:0.10.1.0-1
        ports:
        - containerPort: 9092

apiVersion: v1
kind: Service
metadata:
  labels:
    name: k8guard-kafka-label
  name: k8guard-kafka-service
spec:
  ports:
  - port: 2181
    targetPort: 2181
    name: zk
  - port: 9092
    targetPort: 9092
    name: kafka
  # Label keys and values that must match in order to receive traffic for this service.
  selector:
     app: k8guard-kafka

[medyagh]

it seems like this is a minikube bug, I am closing this one.

[lghinet]

hello
i am having the same problem, and i am not using minikube,
ERROR: Failed to produce message: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes
for non existing topics, for an existing one, works ok.
there was a parameter waitForElection but i cant find it now !?

[lghinet]

solved by increasing config.Metadata.Retry.Backoff = 2*time.Second