Package io.confluent.parallelconsumer.internal

Class AbstractParallelEoSStreamProcessor<K,V>

java.lang.Object

io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor<K,V>

All Implemented Interfaces:

DrainingCloseable, ParallelConsumer<K,V>, Closeable, AutoCloseable, org.apache.kafka.clients.consumer.ConsumerRebalanceListener

Direct Known Subclasses:

ExternalEngine, ParallelEoSStreamProcessor

public abstract class AbstractParallelEoSStreamProcessor<K,V>
extends Object
implements ParallelConsumer<K,V>, org.apache.kafka.clients.consumer.ConsumerRebalanceListener, Closeable

See Also:

ParallelConsumer

Nested Class Summary

Nested classes/interfaces inherited from interface io.confluent.parallelconsumer.internal.DrainingCloseable

DrainingCloseable.DrainingMode

Nested classes/interfaces inherited from interface io.confluent.parallelconsumer.ParallelConsumer

ParallelConsumer.Tuple<L,R>

Field Summary

Fields

Modifier and Type	Field	Description
protected DynamicLoadFactor	dynamicExtraLoadFactor	Multiple of ParallelConsumerOptions.getMaxConcurrency() to have in our processing queue, in order to make sure threads always have work to do.
static String	MDC_INSTANCE_ID
static String	MDC_OFFSET_MARKER
protected ParallelConsumerOptions	options
protected WorkManager<K,V>	wm
protected ThreadPoolExecutor	workerThreadPool	The pool which is used for running the users's supplied function

Fields inherited from interface io.confluent.parallelconsumer.internal.DrainingCloseable

DEFAULT_TIMEOUT

Constructor Summary

Constructors

Constructor	Description
AbstractParallelEoSStreamProcessor(ParallelConsumerOptions<K,V> newOptions)	Construct the AsyncConsumer by wrapping this passed in conusmer and producer, which can be configured any which way as per normal.

Method Summary

Modifier and Type	Method	Description
void	addLoopEndCallBack(Runnable r)	Plugin a function to run at the end of each main loop.
protected void	addToMailbox(WorkContainer<K,V> wc)
protected void	addToMailBoxOnUserFunctionSuccess(WorkContainer<K,V> wc, List<?> resultsFromUserFunction)
protected int	calculateQuantityToRequest()
protected void	checkPipelinePressure()	Checks the system has enough pressure in the pipeline of work, if not attempts to step up the load factor.
void	close()	Close the system, without draining.
void	close(Duration timeout, DrainingCloseable.DrainingMode drainMode)	Close the consumer.
Exception	getFailureCause()
Optional<String>	getMyId()	Optional ID of this instance.
int	getNumberOfAssignedPartitions()
protected ParallelConsumerOptions	getOptions()
protected Optional<ProducerManager<K,V>>	getProducerManager()
protected int	getQueueTargetLoaded()
protected int	getTargetOutForProcessing()
Duration	getTimeBetweenCommits()	Time between commits.
WorkManager<K,V>	getWm()
protected BlockingQueue<io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor.ControllerEventMessage<K,V>>	getWorkMailBox()	Collection of work waiting to be
boolean	isClosedOrFailed()
void	notifySomethingToDo()	Early notify of work arrived.
void	onPartitionsAssigned(Collection<org.apache.kafka.common.TopicPartition> partitions)	Delegate to WorkManager
void	onPartitionsLost(Collection<org.apache.kafka.common.TopicPartition> partitions)	Delegate to WorkManager
void	onPartitionsRevoked(Collection<org.apache.kafka.common.TopicPartition> partitions)	Commit our offsets
protected void	onUserFunctionSuccess(WorkContainer<K,V> wc, List<?> resultsFromUserFunction)
void	registerWork(EpochAndRecordsMap<K,V> polledRecords)
void	requestCommitAsap()	Request a commit as soon as possible (ASAP), overriding other constraints.
protected <R> List<ParallelConsumer.Tuple<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>,R>>	runUserFunction(Function<PollContextInternal<K,V>,List<R>> usersFunction, Consumer<R> callback, List<WorkContainer<K,V>> workContainerBatch)	Run the supplied function.
void	setLongPollTimeout(Duration ofMillis)
void	setMyId(Optional<String> myId)	Optional ID of this instance.
void	setTimeBetweenCommits(Duration timeBetweenCommits)	Time between commits.
protected ThreadPoolExecutor	setupWorkerPool(int poolSize)
protected <R> void	submitWorkToPool(Function<PollContextInternal<K,V>,List<R>> usersFunction, Consumer<R> callback, List<WorkContainer<K,V>> workToProcess)	Submit a piece of work to the processing pool.
void	subscribe(Collection<String> topics)
void	subscribe(Collection<String> topics, org.apache.kafka.clients.consumer.ConsumerRebalanceListener callback)
void	subscribe(Pattern pattern)
void	subscribe(Pattern pattern, org.apache.kafka.clients.consumer.ConsumerRebalanceListener callback)
protected <R> void	supervisorLoop(Function<PollContextInternal<K,V>,List<R>> userFunctionWrapped, Consumer<R> callback)	Supervisor loop for the main loop.
void	waitForProcessedNotCommitted(Duration timeout)	Deprecated. no longer used, will be removed in next version
long	workRemaining()	Of the records consumed from the broker, how many do we have remaining in our local queues

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.confluent.parallelconsumer.internal.DrainingCloseable

closeDontDrainFirst, closeDontDrainFirst, closeDrainFirst, closeDrainFirst

Field Details

MDC_INSTANCE_ID

public static final String MDC_INSTANCE_ID

See Also:

Constant Field Values

MDC_OFFSET_MARKER

public static final String MDC_OFFSET_MARKER

See Also:

Constant Field Values

options

protected final ParallelConsumerOptions options

workerThreadPool

protected final ThreadPoolExecutor workerThreadPool

The pool which is used for running the users's supplied function

wm

protected final WorkManager<K,V> wm

dynamicExtraLoadFactor

protected final DynamicLoadFactor dynamicExtraLoadFactor

Multiple of ParallelConsumerOptions.getMaxConcurrency() to have in our processing queue, in order to make sure threads always have work to do.

Constructor Details

AbstractParallelEoSStreamProcessor

public AbstractParallelEoSStreamProcessor(ParallelConsumerOptions<K,V> newOptions)

Construct the AsyncConsumer by wrapping this passed in conusmer and producer, which can be configured any which way as per normal.

See Also:

ParallelConsumerOptions

Method Details

isClosedOrFailed

public boolean isClosedOrFailed()

getFailureCause

public Exception getFailureCause()

Returns:

if the system failed, returns the recorded reason.

setupWorkerPool

protected ThreadPoolExecutor setupWorkerPool(int poolSize)

subscribe

public void subscribe(Collection<String> topics)

Specified by:

subscribe in interface ParallelConsumer<K,V>

See Also:

KafkaConsumer.subscribe(Collection)

subscribe

public void subscribe(Pattern pattern)

Specified by:

subscribe in interface ParallelConsumer<K,V>

See Also:

KafkaConsumer.subscribe(Pattern)

subscribe

public void subscribe(Collection<String> topics, org.apache.kafka.clients.consumer.ConsumerRebalanceListener callback)

Specified by:

subscribe in interface ParallelConsumer<K,V>

See Also:

KafkaConsumer.subscribe(Collection, ConsumerRebalanceListener)

subscribe

public void subscribe(Pattern pattern, org.apache.kafka.clients.consumer.ConsumerRebalanceListener callback)

Specified by:

subscribe in interface ParallelConsumer<K,V>

See Also:

KafkaConsumer.subscribe(Pattern, ConsumerRebalanceListener)

onPartitionsRevoked

public void onPartitionsRevoked(Collection<org.apache.kafka.common.TopicPartition> partitions)

Commit our offsets

Make sure the calling thread is the thread which performs commit - i.e. is the OffsetCommitter.

Specified by:

onPartitionsRevoked in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener

onPartitionsAssigned

public void onPartitionsAssigned(Collection<org.apache.kafka.common.TopicPartition> partitions)

Delegate to WorkManager

Specified by:

onPartitionsAssigned in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener

See Also:

WorkManager.onPartitionsAssigned(java.util.Collection<org.apache.kafka.common.TopicPartition>)

onPartitionsLost

public void onPartitionsLost(Collection<org.apache.kafka.common.TopicPartition> partitions)

Delegate to WorkManager

Specified by:

onPartitionsLost in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener

See Also:

WorkManager.onPartitionsAssigned(java.util.Collection<org.apache.kafka.common.TopicPartition>)

close

public void close()

Close the system, without draining.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Specified by:

close in interface DrainingCloseable

See Also:

State.draining

close

public void close(Duration timeout, DrainingCloseable.DrainingMode drainMode)

Description copied from interface: DrainingCloseable

Close the consumer.

Specified by:

close in interface DrainingCloseable

Parameters:

timeout - how long to wait before giving up

drainMode - wait for messages already consumed from the broker to be processed before closing

waitForProcessedNotCommitted

@Deprecated public void waitForProcessedNotCommitted(Duration timeout)

Deprecated.

no longer used, will be removed in next version

Block the calling thread until no more messages are being processed.

Used for testing.

supervisorLoop

protected <R> void supervisorLoop(Function<PollContextInternal<K,V>,List<R>> userFunctionWrapped, Consumer<R> callback)

Supervisor loop for the main loop.

See Also:

supervisorLoop(Function, Consumer)

submitWorkToPool

protected <R> void submitWorkToPool(Function<PollContextInternal<K,V>,List<R>> usersFunction, Consumer<R> callback, List<WorkContainer<K,V>> workToProcess)

Submit a piece of work to the processing pool.

Parameters:

workToProcess - the polled records to process

calculateQuantityToRequest

protected int calculateQuantityToRequest()

Returns:

number of WorkContainer to try to get

getTargetOutForProcessing

protected int getTargetOutForProcessing()

getQueueTargetLoaded

protected int getQueueTargetLoaded()

checkPipelinePressure

protected void checkPipelinePressure()

Checks the system has enough pressure in the pipeline of work, if not attempts to step up the load factor.

runUserFunction

protected <R> List<ParallelConsumer.Tuple<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>,R>> runUserFunction(Function<PollContextInternal<K,V>,List<R>> usersFunction, Consumer<R> callback, List<WorkContainer<K,V>> workContainerBatch)

Run the supplied function.

addToMailBoxOnUserFunctionSuccess

protected void addToMailBoxOnUserFunctionSuccess(WorkContainer<K,V> wc, List<?> resultsFromUserFunction)

onUserFunctionSuccess

protected void onUserFunctionSuccess(WorkContainer<K,V> wc, List<?> resultsFromUserFunction)

addToMailbox

protected void addToMailbox(WorkContainer<K,V> wc)

registerWork

public void registerWork(EpochAndRecordsMap<K,V> polledRecords)

notifySomethingToDo

public void notifySomethingToDo()

Early notify of work arrived.

Only wake up the thread if it's sleeping while polling the mail box.

See Also:

processWorkCompleteMailBox(), blockableControlThread

workRemaining

public long workRemaining()

Description copied from interface: DrainingCloseable

Of the records consumed from the broker, how many do we have remaining in our local queues

Specified by:

workRemaining in interface DrainingCloseable

Returns:

the number of consumed but outstanding records to process

addLoopEndCallBack

public void addLoopEndCallBack(Runnable r)

Plugin a function to run at the end of each main loop.

Useful for testing and controlling loop progression.

setLongPollTimeout

public void setLongPollTimeout(Duration ofMillis)

requestCommitAsap

public void requestCommitAsap()

Request a commit as soon as possible (ASAP), overriding other constraints.

getOptions

protected ParallelConsumerOptions getOptions()

setTimeBetweenCommits

public void setTimeBetweenCommits(Duration timeBetweenCommits)

Time between commits. Using a higher frequency will put more load on the brokers.

getTimeBetweenCommits

public Duration getTimeBetweenCommits()

Time between commits. Using a higher frequency will put more load on the brokers.

getProducerManager

protected Optional<ProducerManager<K,V>> getProducerManager()

getWm

public WorkManager<K,V> getWm()

getWorkMailBox

protected BlockingQueue<io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor.ControllerEventMessage<K,V>> getWorkMailBox()

Collection of work waiting to be

getNumberOfAssignedPartitions

public int getNumberOfAssignedPartitions()

setMyId

public void setMyId(Optional<String> myId)

Optional ID of this instance. Useful for testing.

getMyId

public Optional<String> getMyId()

Optional ID of this instance. Useful for testing.