Package org.apache.nifi.processors.azure.eventhub.checkpoint

Class ComponentStateCheckpointStore

java.lang.Object
org.apache.nifi.processors.azure.eventhub.checkpoint.ComponentStateCheckpointStore
All Implemented Interfaces:
com.azure.messaging.eventhubs.CheckpointStore
public class ComponentStateCheckpointStore extends Object implements com.azure.messaging.eventhubs.CheckpointStore
The CheckpointStore is responsible for managing the storage of partition ownership and checkpoint information for Azure Event Hubs consumers. The underlying storage has to be persistent and centralized (shared across the consumer clients in the consumer group).

There exist one ownership entry and one checkpoint entry for each partition in the store. They represent PartitionOwnership and Checkpoint entities in a storage-specific serialized form.

The checkpoint store is plugged into EventProcessorClient and directly used by the load balancer algorithm running in each consumer client instance.

ComponentStateCheckpointStore stores the partition ownership and checkpoint information in the component's (that is ConsumeAzureEventHub processor's) state using NiFi's StateManager in the background.

The format of the ownership entry in the state map: 

    ownership/event-hub-namespace/event-hub-name/consumer-group/partition-id -> client-id/last-modified-time/etag

The format of the checkpoint entry in the state map: 

    checkpoint/event-hub-namespace/event-hub-name/consumer-group/partition-id -> offset/sequence-number

The checkpoint store is required to provide optimistic locking mechanism in order to avoid concurrent updating of the same ownership entry and therefore owning the same partition by multiple client instances at the same time. The optimistic locking is supposed to be based on the eTag field of PartitionOwnership and should be supported at entry level (only updating the same partition ownership is conflicting, claiming ownership of 2 different partitions or updating 2 checkpoints in parallel are valid operations as they are independent changes).

StateManager.replace(StateMap, Map, Scope) method supports optimistic locking but only globally, in the scope of the whole state map (which may or may not contain conflicting changes after update). For this reason, the state update had to be implemented in 2 phases in claimOwnership(List):

  • in the 1st phase the algorithm gets the current state and tries to set the ownership in memory based on eTag, the claim request is skipped if eTag does not match (the original eTag was retrieved in listOwnership(String, String, String))
  • in the 2nd phase StateManager.replace(StateMap, Map, Scope) is called to persist the new state and if it is not successful - meaning that another client instance changed the state in the meantime which may or may not be conflicting -, then the whole process needs to be started over with the 1st phase

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    private final String
    clientId
     
    private static final org.slf4j.Logger
    LOGGER
     
    private final org.apache.nifi.components.state.StateManager
    stateManager
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    ComponentStateCheckpointStore(String clientId, org.apache.nifi.components.state.StateManager stateManager)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    private void
    checkDisconnectedNode(org.apache.nifi.components.state.StateMap state)
     
    reactor.core.publisher.Flux<com.azure.messaging.eventhubs.models.PartitionOwnership>
    claimOwnership(List<com.azure.messaging.eventhubs.models.PartitionOwnership> requestedPartitionOwnerships)
     
    void
    cleanUp(String fullyQualifiedNamespace, String eventHubName, String consumerGroup)
    Cleans up the underlying state map and retains only items matching the "EventHub coordinates" passed in (fullyQualifiedNamespace, eventHubName and consumerGroup).
    (package private) reactor.core.publisher.Mono<Void>
    cleanUpMono(String fullyQualifiedNamespace, String eventHubName, String consumerGroup)
     
    private reactor.util.retry.Retry
    createRetrySpec(String methodName)
     
    private void
    debug(String message, Object... arguments)
     
    private reactor.core.publisher.Flux<com.azure.messaging.eventhubs.models.Checkpoint>
    getCheckpoints(org.apache.nifi.components.state.StateMap state)
     
    private <T> reactor.core.publisher.Flux<T>
    getEntries(org.apache.nifi.components.state.StateMap state, String kind, BiFunction<String,String,T> converter)
     
    private reactor.core.publisher.Flux<com.azure.messaging.eventhubs.models.PartitionOwnership>
    getOwnerships(org.apache.nifi.components.state.StateMap state)
     
    private reactor.core.publisher.Mono<org.apache.nifi.components.state.StateMap>
    getState()
     
    reactor.core.publisher.Flux<com.azure.messaging.eventhubs.models.Checkpoint>
    listCheckpoints(String fullyQualifiedNamespace, String eventHubName, String consumerGroup)
     
    reactor.core.publisher.Flux<com.azure.messaging.eventhubs.models.PartitionOwnership>
    listOwnership(String fullyQualifiedNamespace, String eventHubName, String consumerGroup)
     
    reactor.core.publisher.Mono<Void>
    updateCheckpoint(com.azure.messaging.eventhubs.models.Checkpoint checkpoint)
     
    private reactor.core.publisher.Mono<Void>
    updateState(org.apache.nifi.components.state.StateMap oldState, Map<String,String> newMap)
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • LOGGER

      private static final org.slf4j.Logger LOGGER

    • clientId

      private final String clientId

    • stateManager

      private final org.apache.nifi.components.state.StateManager stateManager

  • Constructor Details

    • ComponentStateCheckpointStore

      public ComponentStateCheckpointStore(String clientId, org.apache.nifi.components.state.StateManager stateManager)

  • Method Details

    • cleanUp

      public void cleanUp(String fullyQualifiedNamespace, String eventHubName, String consumerGroup)
      Cleans up the underlying state map and retains only items matching the "EventHub coordinates" passed in (fullyQualifiedNamespace, eventHubName and consumerGroup). The method should be called once in the initialization phase in order to remove the obsolete items but the checkpoint store can operate properly without doing that too.
      Parameters:
      fullyQualifiedNamespace - the fullyQualifiedNamespace of the items to be retained
      eventHubName - the eventHubName of the items to be retained
      consumerGroup - the consumerGroup of the items to be retained

    • cleanUpMono

      reactor.core.publisher.Mono<Void> cleanUpMono(String fullyQualifiedNamespace, String eventHubName, String consumerGroup)

    • listOwnership

      public reactor.core.publisher.Flux<com.azure.messaging.eventhubs.models.PartitionOwnership> listOwnership(String fullyQualifiedNamespace, String eventHubName, String consumerGroup)
      Specified by:
      listOwnership in interface com.azure.messaging.eventhubs.CheckpointStore

    • claimOwnership

      public reactor.core.publisher.Flux<com.azure.messaging.eventhubs.models.PartitionOwnership> claimOwnership(List<com.azure.messaging.eventhubs.models.PartitionOwnership> requestedPartitionOwnerships)
      Specified by:
      claimOwnership in interface com.azure.messaging.eventhubs.CheckpointStore

    • listCheckpoints

      public reactor.core.publisher.Flux<com.azure.messaging.eventhubs.models.Checkpoint> listCheckpoints(String fullyQualifiedNamespace, String eventHubName, String consumerGroup)
      Specified by:
      listCheckpoints in interface com.azure.messaging.eventhubs.CheckpointStore

    • updateCheckpoint

      public reactor.core.publisher.Mono<Void> updateCheckpoint(com.azure.messaging.eventhubs.models.Checkpoint checkpoint)
      Specified by:
      updateCheckpoint in interface com.azure.messaging.eventhubs.CheckpointStore

    • createRetrySpec

      private reactor.util.retry.Retry createRetrySpec(String methodName)

    • getOwnerships

      private reactor.core.publisher.Flux<com.azure.messaging.eventhubs.models.PartitionOwnership> getOwnerships(org.apache.nifi.components.state.StateMap state)

    • getCheckpoints

      private reactor.core.publisher.Flux<com.azure.messaging.eventhubs.models.Checkpoint> getCheckpoints(org.apache.nifi.components.state.StateMap state)

    • getEntries

      private <T> reactor.core.publisher.Flux<T> getEntries(org.apache.nifi.components.state.StateMap state, String kind, BiFunction<String,String,T> converter) throws org.apache.nifi.processor.exception.ProcessException
      Throws:
      org.apache.nifi.processor.exception.ProcessException

    • checkDisconnectedNode

      private void checkDisconnectedNode(org.apache.nifi.components.state.StateMap state)

    • debug

      private void debug(String message, Object... arguments)

    • getState

      private reactor.core.publisher.Mono<org.apache.nifi.components.state.StateMap> getState()

    • updateState

      private reactor.core.publisher.Mono<Void> updateState(org.apache.nifi.components.state.StateMap oldState, Map<String,String> newMap)