Package io.confluent.parallelconsumer

Interface ParallelStreamProcessor<K,V>

All Superinterfaces:

java.lang.AutoCloseable, java.io.Closeable, DrainingCloseable, ParallelConsumer<K,V>

All Known Implementing Classes:

JStreamParallelEoSStreamProcessor, ParallelEoSStreamProcessor

public interface ParallelStreamProcessor<K,V>
extends ParallelConsumer<K,V>, DrainingCloseable

Parallel message consumer which also can optionally produce 0 or many ProducerRecord results to be published back to Kafka.

See Also:

pollAndProduce(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K, V>, org.apache.kafka.clients.producer.ProducerRecord<K, V>>), pollAndProduceMany(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K, V>, java.util.List<org.apache.kafka.clients.producer.ProducerRecord<K, V>>>, java.util.function.Consumer<io.confluent.parallelconsumer.ParallelStreamProcessor.ConsumeProduceResult<K, V, K, V>>)

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static class	ParallelStreamProcessor.ConsumeProduceResult<K,V,KK,VV>	A simple triple structure to capture the set of coinciding data.

Nested classes/interfaces inherited from interface io.confluent.parallelconsumer.DrainingCloseable

DrainingCloseable.DrainingMode

Nested classes/interfaces inherited from interface io.confluent.parallelconsumer.ParallelConsumer

ParallelConsumer.Tuple<L,R>

Field Summary

Fields inherited from interface io.confluent.parallelconsumer.DrainingCloseable

DEFAULT_TIMEOUT

Method Summary

Modifier and Type	Method	Description
static <KK, VV> ParallelStreamProcessor<KK,VV>	createEosStreamProcessor(ParallelConsumerOptions<KK,VV> options)
void	pollAndProduce(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>,org.apache.kafka.clients.producer.ProducerRecord<K,V>> userFunction)	Register a function to be applied in parallel to each received message, which in turn returns a ProducerRecord to be sent back to the broker.
void	pollAndProduce(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>,org.apache.kafka.clients.producer.ProducerRecord<K,V>> userFunction, java.util.function.Consumer<ParallelStreamProcessor.ConsumeProduceResult<K,V,K,V>> callback)	Register a function to be applied in parallel to each received message, which in turn returns a ProducerRecord to be sent back to the broker.
void	pollAndProduceMany(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>,java.util.List<org.apache.kafka.clients.producer.ProducerRecord<K,V>>> userFunction)	Register a function to be applied in parallel to each received message, which in turn returns one or many ProducerRecords to be sent back to the broker.
void	pollAndProduceMany(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>,java.util.List<org.apache.kafka.clients.producer.ProducerRecord<K,V>>> userFunction, java.util.function.Consumer<ParallelStreamProcessor.ConsumeProduceResult<K,V,K,V>> callback)	Register a function to be applied in parallel to each received message, which in turn returns one or more ProducerRecords to be sent back to the broker.

Methods inherited from interface io.confluent.parallelconsumer.DrainingCloseable

close, close, closeDontDrainFirst, closeDontDrainFirst, closeDrainFirst, closeDrainFirst, workRemaining

Methods inherited from interface io.confluent.parallelconsumer.ParallelConsumer

poll, subscribe, subscribe, subscribe, subscribe

Method Details

createEosStreamProcessor

static <KK, VV> ParallelStreamProcessor<KK,VV> createEosStreamProcessor(ParallelConsumerOptions<KK,VV> options)

pollAndProduceMany

void pollAndProduceMany(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>,java.util.List<org.apache.kafka.clients.producer.ProducerRecord<K,V>>> userFunction, java.util.function.Consumer<ParallelStreamProcessor.ConsumeProduceResult<K,V,K,V>> callback)

Register a function to be applied in parallel to each received message, which in turn returns one or more ProducerRecords to be sent back to the broker.

Parameters:

callback - applied after the produced message is acknowledged by kafka

pollAndProduceMany

void pollAndProduceMany(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>,java.util.List<org.apache.kafka.clients.producer.ProducerRecord<K,V>>> userFunction)

Register a function to be applied in parallel to each received message, which in turn returns one or many ProducerRecords to be sent back to the broker.

pollAndProduce

void pollAndProduce(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>,org.apache.kafka.clients.producer.ProducerRecord<K,V>> userFunction)

Register a function to be applied in parallel to each received message, which in turn returns a ProducerRecord to be sent back to the broker.

pollAndProduce

void pollAndProduce(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>,org.apache.kafka.clients.producer.ProducerRecord<K,V>> userFunction, java.util.function.Consumer<ParallelStreamProcessor.ConsumeProduceResult<K,V,K,V>> callback)

Register a function to be applied in parallel to each received message, which in turn returns a ProducerRecord to be sent back to the broker.

Parameters:

callback - applied after the produced message is acknowledged by kafka