java.lang.Object

io.confluent.parallelconsumer.state.WorkManager<K,V>

Type Parameters:: K -; V -

All Implemented Interfaces:: org.apache.kafka.clients.consumer.ConsumerRebalanceListener

public class WorkManager<K,V>
extends Object
implements org.apache.kafka.clients.consumer.ConsumerRebalanceListener

Sharded, prioritised, offset managed, order controlled, delayed work queue.

Low Water Mark - the highest offset (continuously successful) with all it's previous messages succeeded (the offset one commits to broker)

High Water Mark - the highest offset which has succeeded (previous may be incomplete)

Highest seen offset - the highest ever seen offset

This state is shared between the BrokerPollSystem thread and the AbstractParallelEoSStreamProcessor.

Constructor Summary

Constructors
Constructor	Description
`WorkManager(ParallelConsumerOptions<K,V> options, org.apache.kafka.clients.consumer.Consumer<K,V> consumer)`
`WorkManager(ParallelConsumerOptions<K,V> newOptions, org.apache.kafka.clients.consumer.Consumer<K,V> consumer, DynamicLoadFactor dynamicExtraLoadFactor, Clock clock)`
`WorkManager(ParallelConsumerOptions<K,V> options, org.apache.kafka.clients.consumer.Consumer<K,V> consumer, Clock clock)`	Use a private `DynamicLoadFactor`, useful for testing.

Method Summary

Modifier and Type	Method	Description
`boolean`	`checkIfWorkIsStale(WorkContainer<K,V> workContainer)`	Have our partitions been revoked?
`boolean`	`checkIfWorkIsStale(List<WorkContainer<K,V>> workContainers)`	Have our partitions been revoked? Can a batch contain messages of different epochs?
`Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata>`	`collectCommitDataForDirtyPartitions()`
`Optional<Duration>`	`getLowestRetryTime()`
`long`	`getNumberOfEntriesInPartitionQueues()`
`long`	`getNumberOfWorkQueuedInShardsAwaitingSelection()`
`int`	`getNumberRecordsOutForProcessing()`
`ParallelConsumerOptions<K,V>`	`getOptions()`
`PartitionStateManager<K,V>`	`getPm()`
`ShardManager<K,V>`	`getSm()`
`List<Consumer<WorkContainer<K,V>>>`	`getSuccessfulWorkListeners()`	Useful for testing
`List<WorkContainer<K,V>>`	`getWorkIfAvailable()`	Get work with no limit on quantity, useful for testing.
`List<WorkContainer<K,V>>`	`getWorkIfAvailable(int requestedMaxWorkToRetrieve)`	Depth first work retrieval.
`void`	`handleFutureResult(WorkContainer<K,V> wc)`
`boolean`	`hasWorkInCommitQueues()`
`boolean`	`hasWorkInFlight()`
`boolean`	`isNoRecordsOutForProcessing()`
`boolean`	`isRecordsAwaitingProcessing()`
`boolean`	`isRecordsAwaitingToBeCommitted()`
`boolean`	`isSufficientlyLoaded()`
`boolean`	`isWorkInFlightMeetingTarget()`
`void`	`onFailureResult(WorkContainer<K,V> wc)`
`void`	`onOffsetCommitSuccess(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> committed)`	Can run from controller or poller thread, depending on which is responsible for committing
`void`	`onPartitionsAssigned(Collection<org.apache.kafka.common.TopicPartition> partitions)`	Load offset map for assigned partitions
`void`	`onPartitionsLost(Collection<org.apache.kafka.common.TopicPartition> partitions)`	Clear offset map for lost partitions
`void`	`onPartitionsRevoked(Collection<org.apache.kafka.common.TopicPartition> partitions)`	Clear offset map for revoked partitions
`void`	`onSuccessResult(WorkContainer<K,V> wc)`
`void`	`registerWork(EpochAndRecordsMap<K,V> records)`
`boolean`	`shouldThrottle()`
`boolean`	`workIsWaitingToBeProcessed()`

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details
- WorkManager
  
  public WorkManager(ParallelConsumerOptions<K,V> options, org.apache.kafka.clients.consumer.Consumer<K,V> consumer)
- WorkManager
  
  public WorkManager(ParallelConsumerOptions<K,V> options, org.apache.kafka.clients.consumer.Consumer<K,V> consumer, Clock clock)
  
  Use a private DynamicLoadFactor, useful for testing.
- WorkManager
  
  public WorkManager(ParallelConsumerOptions<K,V> newOptions, org.apache.kafka.clients.consumer.Consumer<K,V> consumer, DynamicLoadFactor dynamicExtraLoadFactor, Clock clock)
Method Details
- onPartitionsAssigned
  
  public void onPartitionsAssigned(Collection<org.apache.kafka.common.TopicPartition> partitions)
  
  Load offset map for assigned partitions
  
  Specified by:
  
  onPartitionsAssigned in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener
- onPartitionsRevoked
  
  public void onPartitionsRevoked(Collection<org.apache.kafka.common.TopicPartition> partitions)
  
  Clear offset map for revoked partitions
  AbstractParallelEoSStreamProcessor.onPartitionsRevoked(java.util.Collection<org.apache.kafka.common.TopicPartition>) handles committing off offsets upon revoke
  
  Specified by:
  
  onPartitionsRevoked in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener
  
  See Also:
  
  AbstractParallelEoSStreamProcessor.onPartitionsRevoked(java.util.Collection<org.apache.kafka.common.TopicPartition>)
- onPartitionsLost
  
  public void onPartitionsLost(Collection<org.apache.kafka.common.TopicPartition> partitions)
  
  Clear offset map for lost partitions
  
  Specified by:
  
  onPartitionsLost in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener
- registerWork
  
  public void registerWork(EpochAndRecordsMap<K,V> records)
- getWorkIfAvailable
  
  public List<WorkContainer<K,V>> getWorkIfAvailable()
  
  Get work with no limit on quantity, useful for testing.
- getWorkIfAvailable
  
  public List<WorkContainer<K,V>> getWorkIfAvailable(int requestedMaxWorkToRetrieve)
  
  Depth first work retrieval.
- onSuccessResult
  
  public void onSuccessResult(WorkContainer<K,V> wc)
- onOffsetCommitSuccess
  
  public void onOffsetCommitSuccess(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> committed)
  
  Can run from controller or poller thread, depending on which is responsible for committing
  
  See Also:
  
  PartitionStateManager.onOffsetCommitSuccess(Map)
- onFailureResult
  
  public void onFailureResult(WorkContainer<K,V> wc)
- getNumberOfEntriesInPartitionQueues
  
  public long getNumberOfEntriesInPartitionQueues()
- collectCommitDataForDirtyPartitions
  
  public Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> collectCommitDataForDirtyPartitions()
- checkIfWorkIsStale
  
  public boolean checkIfWorkIsStale(List<WorkContainer<K,V>> workContainers)
  
  Have our partitions been revoked? Can a batch contain messages of different epochs?
  
  Returns:
  
  true if any epoch is stale, false if not
  
  See Also:
  
  checkIfWorkIsStale(WorkContainer)
- checkIfWorkIsStale
  
  public boolean checkIfWorkIsStale(WorkContainer<K,V> workContainer)
  
  Have our partitions been revoked?
  
  Returns:
  
  true if epoch doesn't match, false if ok
- shouldThrottle
  
  public boolean shouldThrottle()
- isSufficientlyLoaded
  
  public boolean isSufficientlyLoaded()
  
  Returns:
  
  true if there's enough messages downloaded from the broker already to satisfy the pipeline, false if more should be downloaded (or pipelined in the Consumer)
- workIsWaitingToBeProcessed
  
  public boolean workIsWaitingToBeProcessed()
- hasWorkInFlight
  
  public boolean hasWorkInFlight()
- isWorkInFlightMeetingTarget
  
  public boolean isWorkInFlightMeetingTarget()
- getNumberOfWorkQueuedInShardsAwaitingSelection
  
  public long getNumberOfWorkQueuedInShardsAwaitingSelection()
- hasWorkInCommitQueues
  
  public boolean hasWorkInCommitQueues()
- isRecordsAwaitingProcessing
  
  public boolean isRecordsAwaitingProcessing()
- isRecordsAwaitingToBeCommitted
  
  public boolean isRecordsAwaitingToBeCommitted()
- handleFutureResult
  
  public void handleFutureResult(WorkContainer<K,V> wc)
- isNoRecordsOutForProcessing
  
  public boolean isNoRecordsOutForProcessing()
- getLowestRetryTime
  
  public Optional<Duration> getLowestRetryTime()
- getOptions
  
  public ParallelConsumerOptions<K,V> getOptions()
- getPm
  
  public PartitionStateManager<K,V> getPm()
- getSm
  
  public ShardManager<K,V> getSm()
- getNumberRecordsOutForProcessing
  
  public int getNumberRecordsOutForProcessing()
- getSuccessfulWorkListeners
  
  public List<Consumer<WorkContainer<K,V>>> getSuccessfulWorkListeners()
  
  Useful for testing

Class WorkManager<K,​V>

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Details

Method Details

Class WorkManager<K,V>