Package io.confluent.parallelconsumer.internal

Class ExternalEngine<K,V>

java.lang.Object

io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor<K,V>

io.confluent.parallelconsumer.internal.ExternalEngine<K,V>

All Implemented Interfaces:

DrainingCloseable, ParallelConsumer<K,V>, Closeable, AutoCloseable, org.apache.kafka.clients.consumer.ConsumerRebalanceListener

public abstract class ExternalEngine<K,V>
extends AbstractParallelEoSStreamProcessor<K,V>

Overrides key aspects required in common for other threading engines like Vert.x and Reactor

Nested Class Summary

Nested classes/interfaces inherited from interface io.confluent.parallelconsumer.internal.DrainingCloseable

DrainingCloseable.DrainingMode

Nested classes/interfaces inherited from interface io.confluent.parallelconsumer.ParallelConsumer

ParallelConsumer.Tuple<L,R>

Field Summary

Fields inherited from class io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor

dynamicExtraLoadFactor, MDC_INSTANCE_ID, MDC_OFFSET_MARKER, options, wm, workerThreadPool

Fields inherited from interface io.confluent.parallelconsumer.internal.DrainingCloseable

DEFAULT_TIMEOUT

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	ExternalEngine(ParallelConsumerOptions<K,V> newOptions)

Method Summary

Modifier and Type	Method	Description
protected void	addToMailBoxOnUserFunctionSuccess(WorkContainer<K,V> wc, List<?> resultsFromUserFunction)	With Vertx and Reactor, a function hasn't succeeded until the inner vertx function has also succeeded no op
protected void	checkPipelinePressure()	Checks the system has enough pressure in the pipeline of work, if not attempts to step up the load factor.
protected int	getTargetOutForProcessing()
protected abstract boolean	isAsyncFutureWork(List<?> resultsFromUserFunction)	System dependent way for the core engine to know if an external scheduler needs to be relied on for completion, as opposed to the internal threading system in the core module.
protected void	onUserFunctionSuccess(WorkContainer<K,V> wc, List<?> resultsFromUserFunction)	With Vertx and Reactor, a function hasn't succeeded until the inner vertx function has also succeeded logging
protected ThreadPoolExecutor	setupWorkerPool(int poolSize)	The vert.x module doesn't use any thread pool for dispatching work, as the work is all done by the vert.x engine.

Methods inherited from class io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor

addLoopEndCallBack, addToMailbox, calculateQuantityToRequest, close, close, getFailureCause, getMyId, getNumberOfAssignedPartitions, getOptions, getProducerManager, getQueueTargetLoaded, getTimeBetweenCommits, getWm, getWorkMailBox, isClosedOrFailed, notifySomethingToDo, onPartitionsAssigned, onPartitionsLost, onPartitionsRevoked, registerWork, requestCommitAsap, runUserFunction, setLongPollTimeout, setMyId, setTimeBetweenCommits, submitWorkToPool, subscribe, subscribe, subscribe, subscribe, supervisorLoop, waitForProcessedNotCommitted, workRemaining

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.confluent.parallelconsumer.internal.DrainingCloseable

closeDontDrainFirst, closeDontDrainFirst, closeDrainFirst, closeDrainFirst

Constructor Details

ExternalEngine

protected ExternalEngine(ParallelConsumerOptions<K,V> newOptions)

Method Details

getTargetOutForProcessing

protected int getTargetOutForProcessing()

Overrides:

getTargetOutForProcessing in class AbstractParallelEoSStreamProcessor<K,V>

Returns:

the number of records to try to get, based on the current count of records outstanding - but unlike core, we don't pipeline messages into the executor pool for processing.

checkPipelinePressure

protected void checkPipelinePressure()

Description copied from class: AbstractParallelEoSStreamProcessor

Checks the system has enough pressure in the pipeline of work, if not attempts to step up the load factor.

Overrides:

checkPipelinePressure in class AbstractParallelEoSStreamProcessor<K,V>

setupWorkerPool

protected ThreadPoolExecutor setupWorkerPool(int poolSize)

The vert.x module doesn't use any thread pool for dispatching work, as the work is all done by the vert.x engine. This thread is only used to dispatch the work to vert.x.

TODO optimise thread usage by not using any extra thread here at all - go straight from the control thread to vert.x.

Overrides:

setupWorkerPool in class AbstractParallelEoSStreamProcessor<K,V>

onUserFunctionSuccess

protected void onUserFunctionSuccess(WorkContainer<K,V> wc, List<?> resultsFromUserFunction)

With Vertx and Reactor, a function hasn't succeeded until the inner vertx function has also succeeded logging

Overrides:

onUserFunctionSuccess in class AbstractParallelEoSStreamProcessor<K,V>

addToMailBoxOnUserFunctionSuccess

protected void addToMailBoxOnUserFunctionSuccess(WorkContainer<K,V> wc, List<?> resultsFromUserFunction)

With Vertx and Reactor, a function hasn't succeeded until the inner vertx function has also succeeded no op

Overrides:

addToMailBoxOnUserFunctionSuccess in class AbstractParallelEoSStreamProcessor<K,V>

isAsyncFutureWork

protected abstract boolean isAsyncFutureWork(List<?> resultsFromUserFunction)

System dependent way for the core engine to know if an external scheduler needs to be relied on for completion, as opposed to the internal threading system in the core module.

Parameters:

resultsFromUserFunction - the results to test

Returns:

true if the work needs special treatment