cz.o2.proxima.direct.commitlog

Class LogObservers.AbstractSortedLogObserver

java.lang.Object

cz.o2.proxima.direct.commitlog.LogObservers.AbstractSortedLogObserver

All Implemented Interfaces:

LogObserver, java.io.Serializable

Direct Known Subclasses:

LogObservers.SinglePartitionSortedLogObserver, LogObservers.SortedLogObserver

Enclosing class:

LogObservers

private abstract static class LogObservers.AbstractSortedLogObserver
extends java.lang.Object
implements LogObserver

Nested Class Summary

Nested classes/interfaces inherited from interface cz.o2.proxima.direct.commitlog.LogObserver

LogObserver.OffsetCommitter, LogObserver.OnIdleContext, LogObserver.OnNextContext, LogObserver.OnRepartitionContext

Field Summary

Fields

Modifier and Type	Field and Description
(package private) long	allowedLatenessMs
(package private) BiConsumer<StreamElement,LogObserver.OnNextContext>	latecomerConsumer
(package private) LogObserver	upstream

Constructor Summary

Constructors

Constructor and Description
AbstractSortedLogObserver(LogObserver upstream, java.time.Duration allowedLateness, BiConsumer<StreamElement,LogObserver.OnNextContext> latecomerConsumer)

Method Summary

Modifier and Type	Method and Description
(package private) boolean	drainQueue(java.util.PriorityQueue<Pair<StreamElement,LogObserver.OnNextContext>> queue, long maxTimestamp)
(package private) abstract void	enqueue(StreamElement ingest, LogObserver.OnNextContext context)
(package private) abstract long	getWatermark(LogObserver.OnNextContext context)
void	onCancelled() Notify that the processing has been canceled.
void	onCompleted() Notify that the processing has gracefully ended.
(package private) abstract void	onCompletedDrainQueue()
boolean	onError(java.lang.Throwable error) Called to notify there was an {@link Throwable error) in the commit reader.
void	onIdle(LogObserver.OnIdleContext context) Called when the observer is idle.
(package private) abstract void	onIdleDrainQueue(LogObserver.OnIdleContext context)
boolean	onNext(StreamElement ingest, LogObserver.OnNextContext context) Process next record in the commit log.
(package private) abstract boolean	onNextDrainQueue(LogObserver.OnNextContext context)
void	onRepartition(LogObserver.OnRepartitionContext context) Callback to notify of automatic repartitioning.
protected abstract void	reassignPartitions(java.util.Collection<Partition> partitions)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface cz.o2.proxima.direct.commitlog.LogObserver

onException, onFatalError

Field Detail

upstream

final LogObserver upstream

allowedLatenessMs

final long allowedLatenessMs

latecomerConsumer

final BiConsumer<StreamElement,LogObserver.OnNextContext> latecomerConsumer

Constructor Detail

AbstractSortedLogObserver

public AbstractSortedLogObserver(LogObserver upstream,
                                 java.time.Duration allowedLateness,
                                 BiConsumer<StreamElement,LogObserver.OnNextContext> latecomerConsumer)

Method Detail

onCompleted

public void onCompleted()

Description copied from interface: LogObserver

Notify that the processing has gracefully ended.

Specified by:

onCompleted in interface LogObserver

onCancelled

public void onCancelled()

Description copied from interface: LogObserver

Notify that the processing has been canceled.

Specified by:

onCancelled in interface LogObserver

onError

public boolean onError(java.lang.Throwable error)

Description copied from interface: LogObserver

Called to notify there was an {@link Throwable error) in the commit reader.

Specified by:

onError in interface LogObserver

Parameters:

error - error caught during processing

Returns:

true to restart processing from last committed position, false to stop processing

onNext

public boolean onNext(StreamElement ingest,
                      LogObserver.OnNextContext context)

Description copied from interface: LogObserver

Process next record in the commit log.

Specified by:

onNext in interface LogObserver

Parameters:

ingest - the ingested data written to the commit log

context - a context that the application must use to confirm processing of the ingest. If the application fails to do so, the result is undefined.

Returns:

true if the processing should continue, false otherwise

onRepartition

public void onRepartition(LogObserver.OnRepartitionContext context)

Description copied from interface: LogObserver

Callback to notify of automatic repartitioning. This method is always called first before any LogObserver.onNext(cz.o2.proxima.storage.StreamElement, cz.o2.proxima.direct.commitlog.LogObserver.OnNextContext) call happens.

Specified by:

onRepartition in interface LogObserver

Parameters:

context - context of the repartition

onIdle

public void onIdle(LogObserver.OnIdleContext context)

Description copied from interface: LogObserver

Called when the observer is idle. Note that the definition of idle is commit-log dependent and it even might NOT be called at all, if the commit log guarantees that as long as there are *any* data flowing in, then LogObserver.onNext(cz.o2.proxima.storage.StreamElement, cz.o2.proxima.direct.commitlog.LogObserver.OnNextContext) will be called eventually.

Typical example of commit log with no need to call LogObserver.onIdle(cz.o2.proxima.direct.commitlog.LogObserver.OnIdleContext) is google PubSub, having virtually single shared partition which loads balances incoming data.

Specified by:

onIdle in interface LogObserver

Parameters:

context - the context for on idle processing

enqueue

abstract void enqueue(StreamElement ingest,
                      LogObserver.OnNextContext context)

onCompletedDrainQueue

abstract void onCompletedDrainQueue()

onNextDrainQueue

abstract boolean onNextDrainQueue(LogObserver.OnNextContext context)

onIdleDrainQueue

abstract void onIdleDrainQueue(LogObserver.OnIdleContext context)

reassignPartitions

protected abstract void reassignPartitions(java.util.Collection<Partition> partitions)

getWatermark

abstract long getWatermark(LogObserver.OnNextContext context)

drainQueue

boolean drainQueue(java.util.PriorityQueue<Pair<StreamElement,LogObserver.OnNextContext>> queue,
                   long maxTimestamp)