Package io.confluent.parallelconsumer

Class WorkManager<K,V>

java.lang.Object

io.confluent.parallelconsumer.WorkManager<K,V>

Type Parameters:

K -

V -

All Implemented Interfaces:

org.apache.kafka.clients.consumer.ConsumerRebalanceListener

public class WorkManager<K,V>
extends java.lang.Object
implements org.apache.kafka.clients.consumer.ConsumerRebalanceListener

Sharded, prioritised, offset managed, order controlled, delayed work queue.

Field Summary

Fields

Modifier and Type	Field	Description
static double	USED_PAYLOAD_THRESHOLD_MULTIPLIER	Best efforts attempt to prevent usage of offset payload beyond X% - as encoding size test is currently only done per batch, we need to leave some buffer for the required space to overrun before hitting the hard limit where we have to drop the offset payload entirely.

Constructor Summary

Constructors

Constructor	Description
WorkManager(ParallelConsumerOptions options, org.apache.kafka.clients.consumer.Consumer<K,V> consumer)	Use a private DynamicLoadFactor, useful for testing.
WorkManager(ParallelConsumerOptions newOptions, org.apache.kafka.clients.consumer.Consumer<K,V> consumer, DynamicLoadFactor dynamicExtraLoadFactor)

Method Summary

Modifier and Type	Method	Description
int	getNumberOfEntriesInPartitionQueues()
int	getNumberRecordsOutForProcessing()
ParallelConsumerOptions	getOptions()
int	getTotalWorkWaitingProcessing()
WorkContainer<K,V>	getWorkContainerForRecord(org.apache.kafka.clients.consumer.ConsumerRecord<K,V> rec)
int	getWorkQueuedInShardsCount()
protected void	handleFutureResult(WorkContainer<K,V> wc)
boolean	hasWorkInCommitQueues()
boolean	hasWorkInFlight()
boolean	hasWorkInMailboxes()
boolean	isClean()
<R> java.util.List<WorkContainer<K,V>>	maybeGetWork()	Get work with no limit on quantity, useful for testing.
java.util.List<WorkContainer<K,V>>	maybeGetWork(int requestedMaxWorkToRetrieve)	Depth first work retrieval.
void	onFailure(WorkContainer<K,V> wc)
void	onOffsetCommitSuccess(java.util.Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> offsetsToSend)	Truncate our tracked offsets as a commit was successful, so the low water mark rises, and we dont' need to track as much anymore.
void	onPartitionsAssigned(java.util.Collection<org.apache.kafka.common.TopicPartition> partitions)	Load offset map for assigned partitions
void	onPartitionsLost(java.util.Collection<org.apache.kafka.common.TopicPartition> partitions)	Clear offset map for lost partitions
void	onPartitionsRevoked(java.util.Collection<org.apache.kafka.common.TopicPartition> partitions)	Clear offset map for revoked partitions
void	onSuccess(WorkContainer<K,V> wc)
void	registerWork(java.util.List<org.apache.kafka.clients.consumer.ConsumerRecords<K,V>> records)
void	registerWork(org.apache.kafka.clients.consumer.ConsumerRecords<K,V> records)
static void	setUSED_PAYLOAD_THRESHOLD_MULTIPLIER(double USED_PAYLOAD_THRESHOLD_MULTIPLIER)	Best efforts attempt to prevent usage of offset payload beyond X% - as encoding size test is currently only done per batch, we need to leave some buffer for the required space to overrun before hitting the hard limit where we have to drop the offset payload entirely.
boolean	shouldThrottle()
boolean	workIsWaitingToBeProcessed()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

USED_PAYLOAD_THRESHOLD_MULTIPLIER

public static double USED_PAYLOAD_THRESHOLD_MULTIPLIER

Best efforts attempt to prevent usage of offset payload beyond X% - as encoding size test is currently only done per batch, we need to leave some buffer for the required space to overrun before hitting the hard limit where we have to drop the offset payload entirely.

Constructor Details

WorkManager

public WorkManager(ParallelConsumerOptions options, org.apache.kafka.clients.consumer.Consumer<K,V> consumer)

Use a private DynamicLoadFactor, useful for testing.

WorkManager

public WorkManager(ParallelConsumerOptions newOptions, org.apache.kafka.clients.consumer.Consumer<K,V> consumer, DynamicLoadFactor dynamicExtraLoadFactor)

Method Details

onPartitionsAssigned

public void onPartitionsAssigned(java.util.Collection<org.apache.kafka.common.TopicPartition> partitions)

Load offset map for assigned partitions

Specified by:

onPartitionsAssigned in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener

onPartitionsRevoked

public void onPartitionsRevoked(java.util.Collection<org.apache.kafka.common.TopicPartition> partitions)

Clear offset map for revoked partitions

ParallelEoSStreamProcessor.onPartitionsRevoked(java.util.Collection<org.apache.kafka.common.TopicPartition>) handles committing off offsets upon revoke

Specified by:

onPartitionsRevoked in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener

See Also:

ParallelEoSStreamProcessor.onPartitionsRevoked(java.util.Collection<org.apache.kafka.common.TopicPartition>)

onPartitionsLost

public void onPartitionsLost(java.util.Collection<org.apache.kafka.common.TopicPartition> partitions)

Clear offset map for lost partitions

Specified by:

onPartitionsLost in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener

registerWork

public void registerWork(java.util.List<org.apache.kafka.clients.consumer.ConsumerRecords<K,V>> records)

registerWork

public void registerWork(org.apache.kafka.clients.consumer.ConsumerRecords<K,V> records)

maybeGetWork

public <R> java.util.List<WorkContainer<K,V>> maybeGetWork()

Get work with no limit on quantity, useful for testing.

maybeGetWork

public java.util.List<WorkContainer<K,V>> maybeGetWork(int requestedMaxWorkToRetrieve)

Depth first work retrieval.

onSuccess

public void onSuccess(WorkContainer<K,V> wc)

onFailure

public void onFailure(WorkContainer<K,V> wc)

getNumberOfEntriesInPartitionQueues

public int getNumberOfEntriesInPartitionQueues()

getTotalWorkWaitingProcessing

public int getTotalWorkWaitingProcessing()

Returns:

Work count in mailbox plus work added to the processing shards

getWorkQueuedInShardsCount

public int getWorkQueuedInShardsCount()

Returns:

Work ready in the processing shards, awaiting selection as work to do

getWorkContainerForRecord

public WorkContainer<K,V> getWorkContainerForRecord(org.apache.kafka.clients.consumer.ConsumerRecord<K,V> rec)

onOffsetCommitSuccess

public void onOffsetCommitSuccess(java.util.Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> offsetsToSend)

Truncate our tracked offsets as a commit was successful, so the low water mark rises, and we dont' need to track as much anymore.

When commits are made to broker, we can throw away all the individually tracked offsets before the committed offset.

shouldThrottle

public boolean shouldThrottle()

workIsWaitingToBeProcessed

public boolean workIsWaitingToBeProcessed()

hasWorkInFlight

public boolean hasWorkInFlight()

isClean

public boolean isClean()

hasWorkInMailboxes

public boolean hasWorkInMailboxes()

hasWorkInCommitQueues

public boolean hasWorkInCommitQueues()

handleFutureResult

protected void handleFutureResult(WorkContainer<K,V> wc)

setUSED_PAYLOAD_THRESHOLD_MULTIPLIER

public static void setUSED_PAYLOAD_THRESHOLD_MULTIPLIER(double USED_PAYLOAD_THRESHOLD_MULTIPLIER)

Best efforts attempt to prevent usage of offset payload beyond X% - as encoding size test is currently only done per batch, we need to leave some buffer for the required space to overrun before hitting the hard limit where we have to drop the offset payload entirely.

getOptions

public ParallelConsumerOptions getOptions()

getNumberRecordsOutForProcessing

public int getNumberRecordsOutForProcessing()