Package io.confluent.parallelconsumer

Class ParallelEoSStreamProcessor<K,V>

java.lang.Object

io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor<K,V>

io.confluent.parallelconsumer.ParallelEoSStreamProcessor<K,V>

All Implemented Interfaces:

DrainingCloseable, ParallelConsumer<K,V>, ParallelStreamProcessor<K,V>, Closeable, AutoCloseable, org.apache.kafka.clients.consumer.ConsumerRebalanceListener

Direct Known Subclasses:

JStreamParallelEoSStreamProcessor

public class ParallelEoSStreamProcessor<K,V>
extends AbstractParallelEoSStreamProcessor<K,V>
implements ParallelStreamProcessor<K,V>

Nested Class Summary

Nested classes/interfaces inherited from interface io.confluent.parallelconsumer.internal.DrainingCloseable

DrainingCloseable.DrainingMode

Nested classes/interfaces inherited from interface io.confluent.parallelconsumer.ParallelConsumer

ParallelConsumer.Tuple<L,R>

Nested classes/interfaces inherited from interface io.confluent.parallelconsumer.ParallelStreamProcessor

ParallelStreamProcessor.ConsumeProduceResult<K,V,KK,VV>

Field Summary

Fields inherited from class io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor

dynamicExtraLoadFactor, MDC_INSTANCE_ID, MDC_OFFSET_MARKER, options, wm, workerThreadPool

Fields inherited from interface io.confluent.parallelconsumer.internal.DrainingCloseable

DEFAULT_TIMEOUT

Constructor Summary

Constructors

Constructor	Description
ParallelEoSStreamProcessor(ParallelConsumerOptions<K,V> newOptions)	Construct the AsyncConsumer by wrapping this passed in conusmer and producer, which can be configured any which way as per normal.

Method Summary

Modifier and Type	Method	Description
void	poll(Consumer<PollContext<K,V>> usersVoidConsumptionFunction)	Register a function to be applied in parallel to each received message
void	pollAndProduce(Function<PollContext<K,V>,org.apache.kafka.clients.producer.ProducerRecord<K,V>> userFunction)	Register a function to be applied in parallel to each received message, which in turn returns a ProducerRecord to be sent back to the broker.
void	pollAndProduce(Function<PollContext<K,V>,org.apache.kafka.clients.producer.ProducerRecord<K,V>> userFunction, Consumer<ParallelStreamProcessor.ConsumeProduceResult<K,V,K,V>> callback)	Register a function to be applied in parallel to each received message, which in turn returns a ProducerRecord to be sent back to the broker.
void	pollAndProduceMany(Function<PollContext<K,V>,List<org.apache.kafka.clients.producer.ProducerRecord<K,V>>> userFunction)	Register a function to be applied in parallel to each received message, which in turn returns one or many ProducerRecords to be sent back to the broker.
void	pollAndProduceMany(Function<PollContext<K,V>,List<org.apache.kafka.clients.producer.ProducerRecord<K,V>>> userFunction, Consumer<ParallelStreamProcessor.ConsumeProduceResult<K,V,K,V>> callback)	Register a function to be applied in parallel to each received message, which in turn returns one or more ProducerRecords to be sent back to the broker.

Methods inherited from class io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor

addLoopEndCallBack, addToMailbox, addToMailBoxOnUserFunctionSuccess, calculateQuantityToRequest, checkPipelinePressure, close, close, getFailureCause, getMyId, getNumberOfAssignedPartitions, getOptions, getProducerManager, getQueueTargetLoaded, getTargetOutForProcessing, getTimeBetweenCommits, getWm, getWorkMailBox, isClosedOrFailed, notifySomethingToDo, onPartitionsAssigned, onPartitionsLost, onPartitionsRevoked, onUserFunctionSuccess, registerWork, requestCommitAsap, runUserFunction, setLongPollTimeout, setMyId, setTimeBetweenCommits, setupWorkerPool, submitWorkToPool, subscribe, subscribe, subscribe, subscribe, supervisorLoop, waitForProcessedNotCommitted, workRemaining

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.confluent.parallelconsumer.internal.DrainingCloseable

close, close, closeDontDrainFirst, closeDontDrainFirst, closeDrainFirst, closeDrainFirst, workRemaining

Methods inherited from interface io.confluent.parallelconsumer.ParallelConsumer

subscribe, subscribe, subscribe, subscribe

Constructor Details

ParallelEoSStreamProcessor

public ParallelEoSStreamProcessor(ParallelConsumerOptions<K,V> newOptions)

Construct the AsyncConsumer by wrapping this passed in conusmer and producer, which can be configured any which way as per normal.

See Also:

ParallelConsumerOptions

Method Details

poll

public void poll(Consumer<PollContext<K,V>> usersVoidConsumptionFunction)

Description copied from interface: ParallelStreamProcessor

Register a function to be applied in parallel to each received message

Specified by:

poll in interface ParallelStreamProcessor<K,V>

Parameters:

usersVoidConsumptionFunction - the function

pollAndProduceMany

public void pollAndProduceMany(Function<PollContext<K,V>,List<org.apache.kafka.clients.producer.ProducerRecord<K,V>>> userFunction, Consumer<ParallelStreamProcessor.ConsumeProduceResult<K,V,K,V>> callback)

Description copied from interface: ParallelStreamProcessor

Register a function to be applied in parallel to each received message, which in turn returns one or more ProducerRecords to be sent back to the broker.

Specified by:

pollAndProduceMany in interface ParallelStreamProcessor<K,V>

callback - applied after the produced message is acknowledged by kafka

pollAndProduceMany

public void pollAndProduceMany(Function<PollContext<K,V>,List<org.apache.kafka.clients.producer.ProducerRecord<K,V>>> userFunction)

Description copied from interface: ParallelStreamProcessor

Register a function to be applied in parallel to each received message, which in turn returns one or many ProducerRecords to be sent back to the broker.

Specified by:

pollAndProduceMany in interface ParallelStreamProcessor<K,V>

pollAndProduce

public void pollAndProduce(Function<PollContext<K,V>,org.apache.kafka.clients.producer.ProducerRecord<K,V>> userFunction)

Description copied from interface: ParallelStreamProcessor

Register a function to be applied in parallel to each received message, which in turn returns a ProducerRecord to be sent back to the broker.

Specified by:

pollAndProduce in interface ParallelStreamProcessor<K,V>

pollAndProduce

public void pollAndProduce(Function<PollContext<K,V>,org.apache.kafka.clients.producer.ProducerRecord<K,V>> userFunction, Consumer<ParallelStreamProcessor.ConsumeProduceResult<K,V,K,V>> callback)

Description copied from interface: ParallelStreamProcessor

Register a function to be applied in parallel to each received message, which in turn returns a ProducerRecord to be sent back to the broker.

Specified by:

pollAndProduce in interface ParallelStreamProcessor<K,V>

callback - applied after the produced message is acknowledged by kafka