When setting up a kafka producer to use idempotent behaviour, and transactional behaviour:
I understand that for idempotency we set: enable.idempotence=true and that by changing this one flag on our producer, we are guaranteed exactly-once event delivery?
and for transactions, we must go further and set the transaction.id=some value but by setting this value, it also sets idempotence to true?
Also, by setting one or both of the above to true, the producer will also set acks=all.
With the above should I be able to add 'exactly once delivery' by simply changing the enable idempotency setting? If i wanted to go further and enable transactional support, On the Consumer side, I would only need to change their setting, isolation.level=read_committed? Does this image reflect how to setup the producer in terms of EOS?
By enabling idempotence, the producer automatically sets acks to all and guarantees message delivery for the lifetime of the Producer instance.Please note "
lifetime of the Producer instance", If the Producer instance dies a new Producer instance comes up the sequence will start from 0.
By enabling transactions, the producer automatically enables idempotence (and acks=all). Transactions allow to group produce requests and offset commits and ensure all or nothing gets committed to Kafka.
When using transactions, you can configure if consumers should only see records from committed transactions by setting isolation.level to read_committed, otherwise by default they see all records including from discarded transactions.
Update for Apache Kafka 3.0 According to the Announcement of Apache Kafka 3.0 the producer enables the strongest delivery guarantees by default (acks=all, enable.idempotence=true). This means that users now get ordering and durability by default.
The Idempotent Producer only has guarantees within the life of the Producer process. If it crashes, the new Idempotent Producer will have a different ProducerId and will start its own sequence. The Sequence number simply starts from 0 and monotically increases for each record. If a record fails being delivered, it is sent again with its existing sequence number so it can be deduplicated (if needed) by the brokers. The sequence number is per producer and per partition. Currently Kafka does not offer a way to "continue" an Idempotent Producer session. Each time you start one it gets a new and unique ProducerId (generated by the cluster)
The configuration "idempotent" only works when the producer does not crash. However with the transactions, you cand send data accross different partitions exactly once. You set a transaction id with your producer id (automatically created). If a new producer id arrives with the same transaction id, it means that you have a problem. Then, the records will be written exactly once.
This is a feature that is sorely missing from Kafka, and I don't see an elegant and efficient way to solve it without modifying Kafka itself. As a preliminary, if you want true idempotency across any failure (producer or broker), then you absolutely positively need some kind of id in the business layer (rather than the lower level transport layer). What you could do with such an id in Kafka is this: Your producer writes to a topic at-least once, and then you have a Kafka Streams process deduplicating messages from that topic using your business layer id and publishing the remaining unique messages to another topic. In order to be efficient, you should use a monotonically increasing id, aka sequence number, otherwise you would have to keep around (and persist) every id you have ever seen, which amounts to a memory leak, unless you restrict the ability to deduplicate to the last x days / hours / minutes and retain only the latest ids. Or, you give Apache Pulsar a try, which, besides addressing other sore spots of Kafka (having to do a costly manual and error prone rebalance in order to scale out a topic, to name just one) has this feature built in.
Update for Apache Kafka 3.0 According to the Announcement of Apache Kafka 3.0 the producer enables the strongest delivery guarantees by default (acks=all, enable.idempotence=true). This means that users now get ordering and durability by default.
The Idempotent Producer only has guarantees within the life of the Producer process. If it crashes, the new Idempotent Producer will have a different ProducerId and will start its own sequence. The Sequence number simply starts from 0 and monotically increases for each record. If a record fails being delivered, it is sent again with its existing sequence number so it can be deduplicated (if needed) by the brokers. The sequence number is per producer and per partition. Currently Kafka does not offer a way to "continue" an Idempotent Producer session. Each time you start one it gets a new and unique ProducerId (generated by the cluster)
The configuration "idempotent" only works when the producer does not crash. However with the transactions, you cand send data accross different partitions exactly once. You set a transaction id with your producer id (automatically created). If a new producer id arrives with the same transaction id, it means that you have a problem. Then, the records will be written exactly once.
This is a feature that is sorely missing from Kafka, and I don't see an elegant and efficient way to solve it without modifying Kafka itself. As a preliminary, if you want true idempotency across any failure (producer or broker), then you absolutely positively need some kind of id in the business layer (rather than the lower level transport layer). What you could do with such an id in Kafka is this: Your producer writes to a topic at-least once, and then you have a Kafka Streams process deduplicating messages from that topic using your business layer id and publishing the remaining unique messages to another topic. In order to be efficient, you should use a monotonically increasing id, aka sequence number, otherwise you would have to keep around (and persist) every id you have ever seen, which amounts to a memory leak, unless you restrict the ability to deduplicate to the last x days / hours / minutes and retain only the latest ids. Or, you give Apache Pulsar a try, which, besides addressing other sore spots of Kafka (having to do a costly manual and error prone rebalance in order to scale out a topic, to name just one) has this feature built in.
No comments:
Post a Comment