How to move a partition with leader -1 to another broker?

Question

Lets say all the nodes which has a partition's leader and replica crashed and nodes are not recoverable.

How to elect another Kafka broker as leader of that partition? (I am fine with data loss)

I tried "kafka-preferred-replica-election.sh" and "kafka-reassign-partitions.sh" but no success.

Setup:

Two node kafka cluster with broker id 10 and 11.
Topic "test" with replica factor 1 and partition 1. Broker 11 is the leader for topic "test" partition 0
Broker 11 goes down.
Kafka version - kafka_2.11-0.10.1.0

Topic Describe

Topic:test  PartitionCount:1    ReplicationFactor:1 Configs:
Topic: test Partition: 0    Leader: -1  Replicas: 11    Isr:

kafka-reassign-partitions.sh output:

bin/kafka-reassign-partitions.sh --zookeeper myvm:2181 --reassignment-json-file manual-assignment.json --execute
Partitions reassignment failed due to The proposed assignment contains non-existent partitions: List([test,10])
kafka.common.AdminCommandFailedException: The proposed assignment contains non-existent partitions: List([test,10])
    at kafka.admin.ReassignPartitionsCommand$.parseAndValidate(ReassignPartitionsCommand.scala:201)
    at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:153)
    at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:149)
    at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:46)
    at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)

cat manual-assignment.json
{
 "partitions":
  [
    {"topic": "test", "partition": 10, "replicas": [10]}
  ],
  "version":1
}

kafka-preferred-replica-election.sh

bin/kafka-preferred-replica-election.sh --zookeeper myvm:2181 --path-to-json-file topicPartitionList.json
[2016-11-08 03:27:01,993] ERROR Skipping preferred replica leader election for partition [test,10] since it doesn't exist (kafka.admin.PreferredReplicaLeaderElectionCommand)
Successfully started preferred replica election for partitions Set([test,10])

cat topicPartitionList.json
{
 "partitions":
  [
    {"topic": "test", "partition": 10}
  ]
}

amethystic amethystic · Accepted Answer · 2016-11-08T03:55:48

Kafka only elects a leader from ISR and assigned replicas(when enabling unclean leader election). For you case when replication-factor is set to 1, there is no broker left in ISR after the leader broker crashed. No candidate can be elected to be the new leader, so this is as design.

How to move a partition with leader -1 to another broker?

1 Answers