====== Kafka Offset ======

===== Theory =====

An ''offset'' is a sequential number that identifies the position of a message inside a partition.

Important:

  * One partition contains many messages.
  * One message has exactly one offset.
  * Offsets are unique only within a partition.
  * Offsets increase monotonically.
  * Offsets never change after assignment.

Relationship:

<code>
Topic
  └── Partition
        └── Offset
              └── Message
</code>

Formula:

<code>
Message ID = (topic, partition, offset)
</code>

Offset alone is not globally unique.

Example:

<code>
Partition 0, Offset 5
Partition 1, Offset 5
</code>

These are two different messages.

You must use both:

<code>
(partition, offset)
</code>

to uniquely identify a message.

===== Storage Model =====

Kafka stores messages as append-only logs.

When a producer sends a new message:

<code>
Producer --> Topic --> Partition --> Append to end
</code>

Example:

<code>
Topic: orders

Partition 0

Offset 0 --> ORD-1001
Offset 1 --> ORD-1002
Offset 2 --> ORD-1003
</code>

New message:

<code>
ORD-1004
</code>

Kafka appends it:

<code>
Topic: orders

Partition 0

Offset 0 --> ORD-1001
Offset 1 --> ORD-1002
Offset 2 --> ORD-1003
Offset 3 --> ORD-1004
</code>

===== Consumer Theory =====

Consumers do not remove messages.

Instead, each consumer group stores its progress.

Formula:

<code>
(group.id, partition) --> committed offset
</code>

Example:

<code>
Group: email-service

Partition 0 --> Offset 2
</code>

Meaning:

<code>
email-service has processed messages up to offset 2
</code>

Next message:

<code>
Offset 3
</code>

===== Internal Offset Storage =====

Kafka stores committed offsets in an internal topic:

<code>
__consumer_offsets
</code>

Example:

<code>
Group: email-service

orders-P0 --> 2
orders-P1 --> 5

Group: analytics-service

orders-P0 --> 10
orders-P1 --> 12
</code>

Each consumer group has independent offsets.

===== Complete Example =====

Topic:

<code>
orders
</code>

Partitions:

<code>
Partition 0

Offset 0 --> ORD-1001
Offset 1 --> ORD-1002
Offset 2 --> ORD-1003

Partition 1

Offset 0 --> ORD-1004
Offset 1 --> ORD-1005
</code>

Consumer group:

<code>
email-service
</code>

Workers:

<code>
worker-1
worker-2
</code>

Partition assignment:

<code>
worker-1 --> Partition 0
worker-2 --> Partition 1
</code>

Committed offsets:

<code>
email-service

Partition 0 --> 1
Partition 1 --> 0
</code>

Worker-1 calls:

<code>
poll()
</code>

Kafka logic:

<code>
1. Find assigned partitions:
   worker-1 --> Partition 0

2. Find committed offset:
   Partition 0 --> 1

3. Calculate next offset:
   1 + 1 = 2

4. Read message:
   Partition 0, Offset 2
</code>

Kafka returns:

<code>
ORD-1003
</code>

After processing:

<code>
commit(offset=2)
</code>

Kafka updates:

<code>
email-service

Partition 0 --> 2
</code>

===== Batch Consumption =====

One offset always represents one message.

Example:

<code>
Partition 0

Offset 10 --> M1
Offset 11 --> M2
Offset 12 --> M3
</code>

A single poll request may return multiple messages:

<code>
poll()

[
  (P0, 10, M1),
  (P0, 11, M2),
  (P0, 12, M3)
]
</code>

But the rule remains:

<code>
1 offset = 1 message
</code>

Batching is only a performance optimization.

===== Summary =====

^ Concept ^ Description ^
| Topic | Logical stream of messages |
| Partition | Physical shard of a topic |
| Offset | Position of a message in a partition |
| Consumer Group | Logical application consuming messages |
| Consumer | Worker process inside a group |
| Committed Offset | Last processed offset for a group and partition |

Key formulas:

<code>
Message = topic + partition + offset
</code>

<code>
Progress = group.id + partition --> committed offset
</code>

<code>
Next message offset = committed offset + 1
</code>

Golden rules:

<code>
1 partition --> many offsets

1 offset --> 1 message

1 consumer group --> 1 committed offset per partition
</code>