Detecting new partitions in a kafka topic

Question

I am new to Kafka and have a use case where I will be reading data from a kafka topic. While reading it is possible that new partitions are added. Is there a way with golang Sarama client to detect if a new partition has been added to the topic.

One way to detect this is to list the topics at a regular interval.

Do you see any other way or method provided by Sarama package that does that?

Answer 1

If partitions are added to a topic "in flight", then any existing consumer group(s) subscribed to that topic will be re-balanced. The documentation for the Sarama client has this to say on that topic [sic]:

Note that Sarama's Consumer implementation does not currently support automatic consumer-group rebalancing and offset tracking. Blockquote

q.v.: https://pkg.go.dev/github.com/shopify/sarama

If you have a genuine need to respond to re-balancing events then you may need to consider switching to an alternative client, such as Confluent.

fwiw, having used both Sarama the Confluent client is (imho and ime) far easier to work with and more generally and comprehensively capable.