Package io.confluent.parallelconsumer

Class ParallelConsumerOptions<K,​V>

java.lang.Object
io.confluent.parallelconsumer.ParallelConsumerOptions<K,​V>
public class ParallelConsumerOptions<K,​V>
extends Object
The options for the AbstractParallelEoSStreamProcessor system.
See Also:
builder(), ParallelConsumerOptions.ParallelConsumerOptionsBuilder

  • Field Details

  • Method Details

    • isUsingBatching

      public boolean isUsingBatching()

    • getTargetAmountOfRecordsInFlight

      public int getTargetAmountOfRecordsInFlight()
      Returns:
      the combined target of the desired concurrency by the configured batch size

    • validate

      public void validate()

    • isUsingTransactionalProducer

      public boolean isUsingTransactionalProducer()

    • isProducerSupplied

      public boolean isProducerSupplied()

    • builder

      public static <K,​ V> ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,​V> builder()

    • toBuilder

      public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,​V> toBuilder()

    • getConsumer

      public org.apache.kafka.clients.consumer.Consumer<K,​V> getConsumer()
      Required parameter for all use.

    • getProducer

      public org.apache.kafka.clients.producer.Producer<K,​V> getProducer()
      Supplying a producer is only needed if using the produce flows.
      See Also:
      ParallelStreamProcessor

    • getManagedExecutorService

      public String getManagedExecutorService()
      Path to Managed executor service for Java EE

    • getManagedThreadFactory

      public String getManagedThreadFactory()
      Path to Managed thread factory for Java EE

    • getOrdering

      public ParallelConsumerOptions.ProcessingOrder getOrdering()
      The ParallelConsumerOptions.ProcessingOrder type to use

    • getCommitMode

      public ParallelConsumerOptions.CommitMode getCommitMode()
      The ParallelConsumerOptions.CommitMode to be used

    • getMaxConcurrency

      public int getMaxConcurrency()
      Controls the maximum degree of concurrency to occur. Used to limit concurrent calls to external systems to a maximum to prevent overloading them or to a degree, using up quotas.

      When using getBatchSize(), this is over and above the batch size setting. So for example, a getMaxConcurrency() of 2 and a batch size of 3 would result in at most 15 records being processed at once.

      A note on quotas - if your quota is expressed as maximum concurrent calls, this works well. If it's limited in total requests / sec, this may still overload the system. See towards the distributed rate limiting feature for this to be properly addressed: https://github.com/confluentinc/parallel-consumer/issues/24 Add distributed rate limiting support #24.

      In the core module, this sets the number of threads to use in the core's thread pool.

      It's recommended to set this quite high, much higher than core count, as it's expected that these threads will spend most of their time blocked waiting for IO. For automatic setting of this variable, look out for issue https://github.com/confluentinc/parallel-consumer/issues/21 Dynamic concurrency control with flow control or tcp congestion control theory #21.

    • getDefaultMessageRetryDelay

      public Duration getDefaultMessageRetryDelay()
      When a message fails, how long the system should wait before trying that message again. Note that this will not be exact, and is just a target.

    • getRetryDelayProvider

      public Function<RecordContext<K,​V>,​Duration> getRetryDelayProvider()
      When present, use this to generate the retry delay, instead of getDefaultMessageRetryDelay().

      Overrides defaultMessageRetryDelay, even if it's set.

    • getSendTimeout

      public Duration getSendTimeout()
      Controls how long to block while waiting for the Producer.send(org.apache.kafka.clients.producer.ProducerRecord<K, V>) to complete for any ProducerRecords returned from the user-function. Only relevant if using one of the produce-flows and providing a producer. If the timeout occurs the record will be re-processed in the user-function.

      Consider aligning the value with the producer-options to avoid unnecessary re-processing and duplicates on slow Producer.send(org.apache.kafka.clients.producer.ProducerRecord<K, V>) calls.

      See Also:
      ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG

    • getOffsetCommitTimeout

      public Duration getOffsetCommitTimeout()
      Controls how long to block while waiting for offsets to be committed. Only relevant if using ParallelConsumerOptions.CommitMode.PERIODIC_CONSUMER_SYNC commit-mode.

    • getBatchSize

      public Integer getBatchSize()
      The maximum number of messages to attempt to pass into the user functions.

      Batch sizes may sometimes be less than this size, but will never be more.

      The system will treat the messages as a set, so if an error is thrown by the user code, then all messages will be marked as failed and be retried (Note that when they are retried, there is no guarantee they will all be in the same batch again). So if you're going to process messages individually, then don't set a batch size.

      Otherwise, if you're going to process messages in sub sets from this batch, it's better to instead adjust the getBatchSize() instead to the actual desired size, and process them as a whole.

      Note that there is no relationship between the ConsumerConfig setting of ConsumerConfig.MAX_POLL_RECORDS_CONFIG and this configured batch size, as this library introduces a large layer of indirection between the managed consumer, and the managed queues we use.

      This indirection effectively disconnects the processing of messages from "polling" them from the managed client, as we do not wait to process them before calling poll again. We simply call poll as much as we need to, in order to keep our queues full of enough work to satisfy demand.

      If we have enough, then we actively manage pausing our subscription so that we can continue calling poll without pulling in even more messages.

      See Also:
      getBatchSize()

    • getThresholdForTimeSpendInQueueWarning

      public Duration getThresholdForTimeSpendInQueueWarning()
      Configure the amount of delay a record experiences, before a warning is logged.

    • getMaxFailureHistory

      public int getMaxFailureHistory()

    • toString

      public String toString()
      Overrides:
      toString in class Object