Package io.confluent.parallelconsumer

Class ParallelEoSStreamProcessor<K,​V>

java.lang.Object
io.confluent.parallelconsumer.ParallelEoSStreamProcessor<K,​V>
All Implemented Interfaces:
DrainingCloseable, ParallelConsumer<K,​V>, ParallelStreamProcessor<K,​V>, java.io.Closeable, java.lang.AutoCloseable, org.apache.kafka.clients.consumer.ConsumerRebalanceListener
Direct Known Subclasses:
JStreamParallelEoSStreamProcessor
public class ParallelEoSStreamProcessor<K,​V>
extends java.lang.Object
implements ParallelStreamProcessor<K,​V>, org.apache.kafka.clients.consumer.ConsumerRebalanceListener, java.io.Closeable
See Also:
ParallelConsumer

  • Nested Class Summary

    Nested classes/interfaces inherited from interface io.confluent.parallelconsumer.DrainingCloseable

    DrainingCloseable.DrainingMode

    Nested classes/interfaces inherited from interface io.confluent.parallelconsumer.ParallelConsumer

    ParallelConsumer.Tuple<L,​R>

    Nested classes/interfaces inherited from interface io.confluent.parallelconsumer.ParallelStreamProcessor

    ParallelStreamProcessor.ConsumeProduceResult<K,​V,​KK,​VV>

  • Field Summary

    Fields 
    Modifier and Type Field Description
    static java.lang.String MDC_INSTANCE_ID  
    protected WorkManager<K,​V> wm  

    Fields inherited from interface io.confluent.parallelconsumer.DrainingCloseable

    DEFAULT_TIMEOUT

  • Constructor Summary

    Constructors 
    Constructor Description
    ParallelEoSStreamProcessor​(ParallelConsumerOptions newOptions)
    Construct the AsyncConsumer by wrapping this passed in conusmer and producer, which can be configured any which way as per normal.

  • Method Summary

    Modifier and Type Method Description
    protected void addToMailbox​(WorkContainer<K,​V> wc)  
    protected void addToMailBoxOnUserFunctionSuccess​(WorkContainer<K,​V> wc, java.util.List<?> resultsFromUserFunction)  
    void close()
    Close the system, without draining.
    void close​(java.time.Duration timeout, DrainingCloseable.DrainingMode drainMode)
    Close the consumer.
    java.lang.Exception getFailureCause()  
    int getNumberOfAssignedPartitions()  
    java.time.Duration getTimeBetweenCommits()
    Time between commits.
    boolean isClosedOrFailed()  
    void onPartitionsAssigned​(java.util.Collection<org.apache.kafka.common.TopicPartition> partitions)
    Delegate to WorkManager
    void onPartitionsLost​(java.util.Collection<org.apache.kafka.common.TopicPartition> partitions)
    Delegate to WorkManager
    void onPartitionsRevoked​(java.util.Collection<org.apache.kafka.common.TopicPartition> partitions)
    Commit our offsets
    protected void onUserFunctionSuccess​(WorkContainer<K,​V> wc, java.util.List<?> resultsFromUserFunction)  
    void poll​(java.util.function.Consumer<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>> usersVoidConsumptionFunction)
    Register a function to be applied in parallel to each received message
    void pollAndProduce​(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>,​org.apache.kafka.clients.producer.ProducerRecord<K,​V>> userFunction)
    Register a function to be applied in parallel to each received message, which in turn returns a ProducerRecord to be sent back to the broker.
    void pollAndProduce​(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>,​org.apache.kafka.clients.producer.ProducerRecord<K,​V>> userFunction, java.util.function.Consumer<ParallelStreamProcessor.ConsumeProduceResult<K,​V,​K,​V>> callback)
    Register a function to be applied in parallel to each received message, which in turn returns a ProducerRecord to be sent back to the broker.
    void pollAndProduceMany​(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>,​java.util.List<org.apache.kafka.clients.producer.ProducerRecord<K,​V>>> userFunction)
    Register a function to be applied in parallel to each received message, which in turn returns one or many ProducerRecords to be sent back to the broker.
    void pollAndProduceMany​(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>,​java.util.List<org.apache.kafka.clients.producer.ProducerRecord<K,​V>>> userFunction, java.util.function.Consumer<ParallelStreamProcessor.ConsumeProduceResult<K,​V,​K,​V>> callback)
    Register a function to be applied in parallel to each received message, which in turn returns one or more ProducerRecords to be sent back to the broker.
    void requestCommitAsap()
    Request a commit as soon as possible (ASAP), overriding other constraints.
    void setLongPollTimeout​(java.time.Duration ofMillis)  
    void setMyId​(java.util.Optional<java.lang.String> myId)
    Optioanl ID of this instance.
    void setTimeBetweenCommits​(java.time.Duration timeBetweenCommits)
    Time between commits.
    void subscribe​(java.util.Collection<java.lang.String> topics)  
    void subscribe​(java.util.Collection<java.lang.String> topics, org.apache.kafka.clients.consumer.ConsumerRebalanceListener callback)  
    void subscribe​(java.util.regex.Pattern pattern)  
    void subscribe​(java.util.regex.Pattern pattern, org.apache.kafka.clients.consumer.ConsumerRebalanceListener callback)  
    protected <R> void supervisorLoop​(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>,​java.util.List<R>> userFunction, java.util.function.Consumer<R> callback)  
    protected <R> java.util.List<ParallelConsumer.Tuple<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>,​R>> userFunctionRunner​(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>,​java.util.List<R>> usersFunction, java.util.function.Consumer<R> callback, WorkContainer<K,​V> wc)
    Run the supplied function.
    void waitForProcessedNotCommitted​(java.time.Duration timeout)
    Block the calling thread until no more messages are being processed.
    int workRemaining()
    Of the records consumed from the broker, how many do we have remaining in our local queues

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface io.confluent.parallelconsumer.DrainingCloseable

    closeDontDrainFirst, closeDontDrainFirst, closeDrainFirst, closeDrainFirst

  • Field Details

  • Constructor Details

  • Method Details

    • isClosedOrFailed

      public boolean isClosedOrFailed()

    • getFailureCause

      public java.lang.Exception getFailureCause()
      Returns:
      if the system failed, returns the recorded reason.

    • subscribe

      public void subscribe​(java.util.Collection<java.lang.String> topics)
      Specified by:
      subscribe in interface ParallelConsumer<K,​V>
      See Also:
      KafkaConsumer.subscribe(Collection)

    • subscribe

      public void subscribe​(java.util.regex.Pattern pattern)
      Specified by:
      subscribe in interface ParallelConsumer<K,​V>
      See Also:
      KafkaConsumer.subscribe(Pattern)

    • subscribe

      public void subscribe​(java.util.Collection<java.lang.String> topics, org.apache.kafka.clients.consumer.ConsumerRebalanceListener callback)
      Specified by:
      subscribe in interface ParallelConsumer<K,​V>
      See Also:
      KafkaConsumer.subscribe(Collection, ConsumerRebalanceListener)

    • subscribe

      public void subscribe​(java.util.regex.Pattern pattern, org.apache.kafka.clients.consumer.ConsumerRebalanceListener callback)
      Specified by:
      subscribe in interface ParallelConsumer<K,​V>
      See Also:
      KafkaConsumer.subscribe(Pattern, ConsumerRebalanceListener)

    • onPartitionsRevoked

      public void onPartitionsRevoked​(java.util.Collection<org.apache.kafka.common.TopicPartition> partitions)
      Commit our offsets

      Make sure the calling thread is the thread which performs commit - i.e. is the OffsetCommitter.

      Specified by:
      onPartitionsRevoked in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener

    • onPartitionsAssigned

      public void onPartitionsAssigned​(java.util.Collection<org.apache.kafka.common.TopicPartition> partitions)
      Delegate to WorkManager
      Specified by:
      onPartitionsAssigned in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener
      See Also:
      WorkManager.onPartitionsAssigned(java.util.Collection<org.apache.kafka.common.TopicPartition>)

    • onPartitionsLost

      public void onPartitionsLost​(java.util.Collection<org.apache.kafka.common.TopicPartition> partitions)
      Delegate to WorkManager
      Specified by:
      onPartitionsLost in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener
      See Also:
      WorkManager.onPartitionsAssigned(java.util.Collection<org.apache.kafka.common.TopicPartition>)

    • poll

      public void poll​(java.util.function.Consumer<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>> usersVoidConsumptionFunction)
      Description copied from interface: ParallelConsumer
      Register a function to be applied in parallel to each received message
      Specified by:
      poll in interface ParallelConsumer<K,​V>
      Parameters:
      usersVoidConsumptionFunction - the function

    • pollAndProduceMany

      public void pollAndProduceMany​(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>,​java.util.List<org.apache.kafka.clients.producer.ProducerRecord<K,​V>>> userFunction, java.util.function.Consumer<ParallelStreamProcessor.ConsumeProduceResult<K,​V,​K,​V>> callback)
      Description copied from interface: ParallelStreamProcessor
      Register a function to be applied in parallel to each received message, which in turn returns one or more ProducerRecords to be sent back to the broker.
      Specified by:
      pollAndProduceMany in interface ParallelStreamProcessor<K,​V>
      callback - applied after the produced message is acknowledged by kafka

    • pollAndProduceMany

      public void pollAndProduceMany​(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>,​java.util.List<org.apache.kafka.clients.producer.ProducerRecord<K,​V>>> userFunction)
      Description copied from interface: ParallelStreamProcessor
      Register a function to be applied in parallel to each received message, which in turn returns one or many ProducerRecords to be sent back to the broker.
      Specified by:
      pollAndProduceMany in interface ParallelStreamProcessor<K,​V>

    • pollAndProduce

      public void pollAndProduce​(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>,​org.apache.kafka.clients.producer.ProducerRecord<K,​V>> userFunction)
      Description copied from interface: ParallelStreamProcessor
      Register a function to be applied in parallel to each received message, which in turn returns a ProducerRecord to be sent back to the broker.
      Specified by:
      pollAndProduce in interface ParallelStreamProcessor<K,​V>

    • pollAndProduce

      public void pollAndProduce​(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>,​org.apache.kafka.clients.producer.ProducerRecord<K,​V>> userFunction, java.util.function.Consumer<ParallelStreamProcessor.ConsumeProduceResult<K,​V,​K,​V>> callback)
      Description copied from interface: ParallelStreamProcessor
      Register a function to be applied in parallel to each received message, which in turn returns a ProducerRecord to be sent back to the broker.
      Specified by:
      pollAndProduce in interface ParallelStreamProcessor<K,​V>
      callback - applied after the produced message is acknowledged by kafka

    • close

      public void close()
      Close the system, without draining.
      Specified by:
      close in interface java.lang.AutoCloseable
      Specified by:
      close in interface java.io.Closeable
      Specified by:
      close in interface DrainingCloseable
      See Also:
      ParallelEoSStreamProcessor.State.draining

    • close

      public void close​(java.time.Duration timeout, DrainingCloseable.DrainingMode drainMode)
      Description copied from interface: DrainingCloseable
      Close the consumer.
      Specified by:
      close in interface DrainingCloseable
      Parameters:
      timeout - how long to wait before giving up
      drainMode - wait for messages already consumed from the broker to be processed before closing

    • waitForProcessedNotCommitted

      public void waitForProcessedNotCommitted​(java.time.Duration timeout)
      Block the calling thread until no more messages are being processed.

    • supervisorLoop

      protected <R> void supervisorLoop​(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>,​java.util.List<R>> userFunction, java.util.function.Consumer<R> callback)

    • userFunctionRunner

      protected <R> java.util.List<ParallelConsumer.Tuple<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>,​R>> userFunctionRunner​(java.util.function.Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>,​java.util.List<R>> usersFunction, java.util.function.Consumer<R> callback, WorkContainer<K,​V> wc)
      Run the supplied function.

    • addToMailBoxOnUserFunctionSuccess

      protected void addToMailBoxOnUserFunctionSuccess​(WorkContainer<K,​V> wc, java.util.List<?> resultsFromUserFunction)

    • onUserFunctionSuccess

      protected void onUserFunctionSuccess​(WorkContainer<K,​V> wc, java.util.List<?> resultsFromUserFunction)

    • addToMailbox

      protected void addToMailbox​(WorkContainer<K,​V> wc)

    • workRemaining

      public int workRemaining()
      Description copied from interface: DrainingCloseable
      Of the records consumed from the broker, how many do we have remaining in our local queues
      Specified by:
      workRemaining in interface DrainingCloseable
      Returns:
      the number of consumed but outstanding records to process

    • setLongPollTimeout

      public void setLongPollTimeout​(java.time.Duration ofMillis)

    • requestCommitAsap

      public void requestCommitAsap()
      Request a commit as soon as possible (ASAP), overriding other constraints.

    • setTimeBetweenCommits

      public void setTimeBetweenCommits​(java.time.Duration timeBetweenCommits)
      Time between commits. Using a higher frequency will put more load on the brokers.

    • getTimeBetweenCommits

      public java.time.Duration getTimeBetweenCommits()
      Time between commits. Using a higher frequency will put more load on the brokers.

    • getNumberOfAssignedPartitions

      public int getNumberOfAssignedPartitions()

    • setMyId

      public void setMyId​(java.util.Optional<java.lang.String> myId)
      Optioanl ID of this instance. Useful for testing.