Is it possible to lose Consumer group Offsets by Kafka brokers?

Question

I was consuming from a Kafka topic (with a retention of 21 days) in a Kafka Cluster as a consumer (earliest/from beginning) for 15 days continously with x consumer group and on 15th day producer's team stopped producing and I stopped consumer on my side conforming that no messages left over to consume. Then Kafka Cluster was also turned off. Then on 16th day Kafka Cluster turned on and Producer started his producer on 23rd day and I started my consumer on myside as well. But when I started, I was getting messages from beginning not from where I left out eventhough am consuming with same x consumer group. So my question is why this happened? Does Kafka Broker lost information about consumer group?

Answer 1

When a consumer group loses all its consumers its offsets will be kept for the period configured in the broker property offsets.retention.minutes. This property defaults to 10080 which is the equivalent to 7 days — roughly the time taken when your consumer stopped (the 16th day) and when it was resumed (the 23rd day).

You can tweak this property to increase the retention period of offsets. Alternatively, you can also tweak the property offsets.retention.check.interval.ms that dictates how often a check for stale offsets will occur.