Kafka Group Coordinator

Kafka Group Coordinator – Interview Perspective

Interview Question:
“What is the Group Coordinator in Kafka? How is it selected, what are its responsibilities, and where can we find its code in Kafka’s source?”

1. What is a Group Coordinator?

The Group Coordinator in Kafka is a designated Kafka broker responsible for managing a consumer group. It handles consumer group membership, rebalancing, and offset commits for the group.

2. How is the Group Coordinator Selected?

• Every broker in Kafka is capable of being a group coordinator.
• When a consumer joins a group, it sends a FindCoordinatorRequest to any broker.
• That broker uses a consistent hashing strategy on the group.id to determine which broker will be the Group Coordinator for that group.
• The broker holding the partition for the internal topic __consumer_offsets relevant to that group.id becomes the Group Coordinator.

3. Responsibilities of the Group Coordinator

The Group Coordinator performs several key functions:

• Manage Consumer Membership: Tracks all consumers in a group.
• Coordinate Rebalances: When consumers join/leave, it triggers a rebalance and assigns partitions.
• Assign Leader: Appoints one consumer as the group leader to help coordinate partition assignment.
• Track Offsets: Stores committed offsets in the __consumer_offsets topic.
• Handle Heartbeats: Keeps track of active consumers to detect failures quickly.

4. Services Provided by the Group Coordinator

It acts as a centralized service point for:

• Tracking active consumers in the group
• Reassigning partitions during consumer join/leave events
• Persisting committed offsets
• Coordinating heartbeat checks and group timeouts

5. Internal Communication

• Consumers communicate with the group coordinator using RPC-style requests:
• JoinGroupRequest
• SyncGroupRequest
• HeartbeatRequest
• LeaveGroupRequest
• OffsetCommitRequest

6. Where Is This in Kafka’s Source Code?

The core logic for group coordination lives in the Kafka broker codebase:

• Class: GroupCoordinator
• Location: core/src/main/scala/kafka/coordinator/group/GroupCoordinator.scala
• Handles all consumer group membership and offset commit logic.
• Works closely with the __consumer_offsets internal topic.

Additional Code References:

• kafka.server.KafkaApis – entry point that handles requests from consumers
• FindCoordinatorRequest – initiates coordinator discovery
• GroupMetadataManager – helps manage group metadata and offset storage

8. What If There Are Multiple Consumer Groups?

Kafka is designed to handle many consumer groups efficiently. Each consumer group is managed independently by potentially different group coordinators, depending on the group ID.

How Kafka Handles Multiple Groups:

• Each group.id is mapped (via hashing) to a broker using the __consumer_offsets topic.
• This ensures that different consumer groups can be coordinated by different brokers for load distribution.
• Even if multiple groups consume from the same topic, they operate independently and can have different offsets and members.

Example Scenario:

• Group A might be coordinated by Broker 1.
• Group B might be coordinated by Broker 2.
• Both could be consuming from the same topic orders, but maintain their own committed offsets.

Advantages of This Design:

• Isolation: Each group is logically separated — no interference or data duplication.
• Scalability: Multiple brokers help distribute coordination load.
• Flexibility: You can have many groups with different configurations and processing logic.

Interview Summary:
“Kafka supports multiple consumer groups natively. Each group is handled independently, and group coordinators are selected using the group ID’s hash. This ensures that different groups are isolated in terms of membership and offset tracking, even when reading from the same topic. It's a powerful feature that enables multi-team, multi-application consumption from a single Kafka topic.”

7. Summary (Interview Style)

“The Group Coordinator is a Kafka broker responsible for managing a specific consumer group. It is selected via consistent hashing based on the group ID and coordinates group membership, partition assignments, and offset tracking. It plays a critical role in ensuring consumer groups are balanced and fault-tolerant. You can find the implementation in GroupCoordinator.scala within the Kafka broker code.”