My scenario is as follows: I have topics A and B with two partitions. I also have two consumers C1 and C2 that read from topic A, do some work and then use producer P1 and P2 to write to topic B. Note that C1 and P1 run in one service-instance and C2 and P2 run in one instance.
I need exactly-once semantics, so I make my producers idempotent. This means that the broker will give my producers a unique producer-id (P1, P2) and each producer will have a monotonically increasing sequence number for each message.
Now a message is written into topic A, partition 1. C1 picks it up, transforms it and P1 produces it into topic B. Now let's say that the message was successfully delivered to the broker but the ACK was not received by P1 because the instance died, including C1. This will lead to a rebalance and C2 will now consume both partition 1 and partition 2. As the ACK was not received, it will re-read the message, process it and P2 will produce again into topic B. This results in the idempotency being broken and the same input message results in two output messages.
Is my logic correct? If so, how can I really achieve idempotency in Kafka across multiple consumer and producer instances on one topic? If not, where is my example wrong?