Monday, April 14, 2025

Kafka endOffset

endOffsets()

🔢 1. Understanding endOffsets()

When you call:

long endOffset = consumer.endOffsets(List.of(tp)).get(tp);

You get the next offset after the last available message in that partition.

Think of endOffset as a marker:
“If a new message arrives, this is the offset it will be assigned.”


🧠 Example:

Message Offset
A 0
B 1
C 2
  • endOffset = 3 (no message at offset 3 yet)

  • So to read last message, we must seek to endOffset - 1 = 2


🔍 Why Not Use endOffset Directly?

If you call:

consumer.seek(tp, endOffset); // BAD!

It will position the consumer after the last message, and poll() will return nothing, because there's nothing at that offset yet.


✅ Summary

Offset Meaning
currentOffset Where the next poll() will start
endOffset The next available offset, NOT the last existing record
endOffset - 1 Points to the actual last message in the topic/partition

🧠 Interview Line:

“Kafka offsets are forward-pointing. The endOffset indicates where the next message will be written, not where the last message lives. So to retrieve the last message, we seek to endOffset - 1.”


Let me know if you want this as a diagram or visual timeline — it's perfect for interview slides.

No comments:

Post a Comment

Kafka Partition

🧩 What Exactly Is a Partition in Kafka? A partition is the fundamental unit of storage, parallelism, and scalability in Kafka. Think of ...