org.apache.nifi.processors.kafka.pubsub

Class ConsumerPool

java.lang.Object

org.apache.nifi.processors.kafka.pubsub.ConsumerPool

All Implemented Interfaces:

Closeable, AutoCloseable

public class ConsumerPool
extends Object
implements Closeable

A pool of Kafka Consumers for a given topic. Consumers can be obtained by calling 'obtainConsumer'. Once closed the pool is ready to be immediately used again.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) static class	ConsumerPool.PoolStats
private class	ConsumerPool.SimpleConsumerLease

Field Summary

Fields

Modifier and Type	Field and Description
private String	bootstrapServers
private AtomicLong	consumerClosedCountRef
private AtomicLong	consumerCreatedCountRef
private byte[]	demarcatorBytes
private Charset	headerCharacterSet
private Pattern	headerNamePattern
private boolean	honorTransactions
private Map<String,Object>	kafkaProperties
private String	keyEncoding
private AtomicLong	leasesObtainedCountRef
private ComponentLog	logger
private long	maxWaitMillis
private BlockingQueue<ConsumerPool.SimpleConsumerLease>	pooledLeases
private RecordReaderFactory	readerFactory
private String	securityProtocol
private Pattern	topicPattern
private List<String>	topics
private RecordSetWriterFactory	writerFactory

Constructor Summary

Constructors

Constructor and Description
ConsumerPool(int maxConcurrentLeases, byte[] demarcator, Map<String,Object> kafkaProperties, List<String> topics, long maxWaitMillis, String keyEncoding, String securityProtocol, String bootstrapServers, ComponentLog logger, boolean honorTransactions, Charset headerCharacterSet, Pattern headerNamePattern) Creates a pool of KafkaConsumer objects that will grow up to the maximum indicated threads from the given context.
ConsumerPool(int maxConcurrentLeases, byte[] demarcator, Map<String,Object> kafkaProperties, Pattern topics, long maxWaitMillis, String keyEncoding, String securityProtocol, String bootstrapServers, ComponentLog logger, boolean honorTransactions, Charset headerCharacterSet, Pattern headerNamePattern)
ConsumerPool(int maxConcurrentLeases, RecordReaderFactory readerFactory, RecordSetWriterFactory writerFactory, Map<String,Object> kafkaProperties, List<String> topics, long maxWaitMillis, String securityProtocol, String bootstrapServers, ComponentLog logger, boolean honorTransactions, Charset headerCharacterSet, Pattern headerNamePattern)
ConsumerPool(int maxConcurrentLeases, RecordReaderFactory readerFactory, RecordSetWriterFactory writerFactory, Map<String,Object> kafkaProperties, Pattern topics, long maxWaitMillis, String securityProtocol, String bootstrapServers, ComponentLog logger, boolean honorTransactions, Charset headerCharacterSet, Pattern headerNamePattern)

Method Summary

Modifier and Type	Method and Description
void	close() Closes all consumers in the pool.
private void	closeConsumer(org.apache.kafka.clients.consumer.Consumer<?,?> consumer)
protected org.apache.kafka.clients.consumer.Consumer<byte[],byte[]>	createKafkaConsumer() Exposed as protected method for easier unit testing
(package private) ConsumerPool.PoolStats	getPoolStats()
ConsumerLease	obtainConsumer(ProcessSession session, ProcessContext processContext) Obtains a consumer from the pool if one is available or lazily initializes a new one if deemed necessary.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

pooledLeases

private final BlockingQueue<ConsumerPool.SimpleConsumerLease> pooledLeases

topics

private final List<String> topics

topicPattern

private final Pattern topicPattern

kafkaProperties

private final Map<String,Object> kafkaProperties

maxWaitMillis

private final long maxWaitMillis

logger

private final ComponentLog logger

demarcatorBytes

private final byte[] demarcatorBytes

keyEncoding

private final String keyEncoding

securityProtocol

private final String securityProtocol

bootstrapServers

private final String bootstrapServers

honorTransactions

private final boolean honorTransactions

readerFactory

private final RecordReaderFactory readerFactory

writerFactory

private final RecordSetWriterFactory writerFactory

headerCharacterSet

private final Charset headerCharacterSet

headerNamePattern

private final Pattern headerNamePattern

consumerCreatedCountRef

private final AtomicLong consumerCreatedCountRef

consumerClosedCountRef

private final AtomicLong consumerClosedCountRef

leasesObtainedCountRef

private final AtomicLong leasesObtainedCountRef

Constructor Detail

ConsumerPool

public ConsumerPool(int maxConcurrentLeases,
                    byte[] demarcator,
                    Map<String,Object> kafkaProperties,
                    List<String> topics,
                    long maxWaitMillis,
                    String keyEncoding,
                    String securityProtocol,
                    String bootstrapServers,
                    ComponentLog logger,
                    boolean honorTransactions,
                    Charset headerCharacterSet,
                    Pattern headerNamePattern)

Creates a pool of KafkaConsumer objects that will grow up to the maximum indicated threads from the given context. Consumers are lazily initialized. We may elect to not create up to the maximum number of configured consumers if the broker reported lag time for all topics is below a certain threshold.

Parameters:

maxConcurrentLeases - max allowable consumers at once

demarcator - bytes to use as demarcator between messages; null or empty means no demarcator

kafkaProperties - properties to use to initialize kafka consumers

topics - the topics to subscribe to

maxWaitMillis - maximum time to wait for a given lease to acquire data before committing

keyEncoding - the encoding to use for the key of a kafka message if found

securityProtocol - the security protocol used

bootstrapServers - the bootstrap servers

logger - the logger to report any errors/warnings

ConsumerPool

public ConsumerPool(int maxConcurrentLeases,
                    byte[] demarcator,
                    Map<String,Object> kafkaProperties,
                    Pattern topics,
                    long maxWaitMillis,
                    String keyEncoding,
                    String securityProtocol,
                    String bootstrapServers,
                    ComponentLog logger,
                    boolean honorTransactions,
                    Charset headerCharacterSet,
                    Pattern headerNamePattern)

ConsumerPool

public ConsumerPool(int maxConcurrentLeases,
                    RecordReaderFactory readerFactory,
                    RecordSetWriterFactory writerFactory,
                    Map<String,Object> kafkaProperties,
                    Pattern topics,
                    long maxWaitMillis,
                    String securityProtocol,
                    String bootstrapServers,
                    ComponentLog logger,
                    boolean honorTransactions,
                    Charset headerCharacterSet,
                    Pattern headerNamePattern)

ConsumerPool

public ConsumerPool(int maxConcurrentLeases,
                    RecordReaderFactory readerFactory,
                    RecordSetWriterFactory writerFactory,
                    Map<String,Object> kafkaProperties,
                    List<String> topics,
                    long maxWaitMillis,
                    String securityProtocol,
                    String bootstrapServers,
                    ComponentLog logger,
                    boolean honorTransactions,
                    Charset headerCharacterSet,
                    Pattern headerNamePattern)

Method Detail

obtainConsumer

public ConsumerLease obtainConsumer(ProcessSession session,
                                    ProcessContext processContext)

Obtains a consumer from the pool if one is available or lazily initializes a new one if deemed necessary.

Parameters:

session - the session for which the consumer lease will be associated

processContext - the ProcessContext for which the consumer lease will be associated

Returns:

consumer to use or null if not available or necessary

createKafkaConsumer

protected org.apache.kafka.clients.consumer.Consumer<byte[],byte[]> createKafkaConsumer()

Exposed as protected method for easier unit testing

Returns:

consumer

Throws:

org.apache.kafka.common.KafkaException - if unable to subscribe to the given topics

close

public void close()

Closes all consumers in the pool. Can be safely called repeatedly.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

closeConsumer

private void closeConsumer(org.apache.kafka.clients.consumer.Consumer<?,?> consumer)

getPoolStats

ConsumerPool.PoolStats getPoolStats()