Kafka Consumer Fetch Configs

Kafka Consumer Fetch Tuning - Interview Guide

Purpose of Kafka Consumer Group -> To paralleize the message consumption and achieve fault tolerance.

⚙️ Core Settings

• fetch.min.bytes – Minimum data size (bytes) broker should collect before replying to consumer.
• fetch.max.wait.ms – Maximum time broker will wait to meet the fetch.min.bytes requirement.
• fetch.max.bytes – Maximum bytes the consumer is willing to receive in a single fetch response.

Kafka sends a fetch response if it has ≥ fetch.min.bytes data available OR if fetch.max.wait.ms expires.

fetch.max.bytes is the hard cap on total response size per fetch from all partitions.

📈 Example Scenario

fetch.min.bytes = 1000
fetch.max.wait.ms = 3000
fetch.max.bytes = 1500

✔ Message A = 800 bytes
✔ Message B = 900 bytes

Case:
- Broker receives Message A → waits for Message B
- A+B = 1700 bytes → exceeds fetch.max.bytes
- Kafka will only return Message A in this fetch cycle

🧠 Diagram - Fetch Behavior Matrix

+----------------------+--------------------------+----------------------+---------------------+ | Scenario | fetch.min.bytes | fetch.max.wait.ms | fetch.max.bytes | +----------------------+--------------------------+----------------------+---------------------+ | Total data = 400 | < 1000 → wait | Waits up to 3s | Not exceeded | | Total data = 1200 | >= min.bytes → ✅ send | Immediately | Not exceeded | | Message = 2000 | > max.bytes → ❌ skipped | Will not be fetched | Must raise config | | Total = 1600 | >= min.bytes | Time ok | Too large → partial | +----------------------+--------------------------+----------------------+---------------------+

🎯 Interview Questions & Answers

Q1. What does fetch.max.bytes do in Kafka consumer configuration?

It defines the maximum number of bytes a consumer can receive in a single fetch response. It is a hard cap across all partitions in a fetch cycle.

Q2. What happens if a record is larger than fetch.max.bytes?

Kafka will not return that record in the fetch response. The message is effectively "skipped" until fetch.max.bytes is increased to accommodate it.

Q3. Will Kafka throw an exception if a message is too large for fetch.max.bytes?

No. It silently skips the record. The consumer just keeps polling and receives nothing.

Q4. What if multiple messages together exceed fetch.max.bytes?

Kafka includes only as many messages as it can fit within the limit. The rest will be delivered in the next fetch call.

Q5. How does fetch.max.bytes differ from max.partition.fetch.bytes?

• fetch.max.bytes – Total limit across all partitions
• max.partition.fetch.bytes – Limit per partition
• Both must be adjusted if your message size increases

📘 Bonus: Java Config Snippet

Properties props = new Properties();
props.put("fetch.min.bytes", "1000");
props.put("fetch.max.wait.ms", "3000");
props.put("fetch.max.bytes", "1500");

KafkaConsumer consumer = new KafkaConsumer<>(props);
ConsumerRecords records = consumer.poll(Duration.ofMillis(5000));

if (records.isEmpty()) {
    System.out.println("No records fetched — possibly due to fetch limits.");
}

💡 Interview Tips

• Understand that these fetch configs optimize between latency and throughput.
• Mention how fetch.max.bytes is critical for avoiding starvation with large messages.
• Be ready to talk about how fetch.max.bytes and max.partition.fetch.bytes should be tuned together.