Package org.apache.pulsar.compaction

Class AbstractTwoPhaseCompactor<T>

java.lang.Object

org.apache.pulsar.compaction.Compactor

org.apache.pulsar.compaction.AbstractTwoPhaseCompactor<T>

Direct Known Subclasses:

EventTimeOrderCompactor, PublishingOrderCompactor

public abstract class AbstractTwoPhaseCompactor<T>
extends Compactor

Compaction will go through the topic in two passes. The first pass selects latest offset for each key in the topic. Then the second pass writes these values to a ledger.

The two passes are required to avoid holding the payloads of each of the latest values in memory, as the payload can be many orders of magnitude larger than a message id.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	AbstractTwoPhaseCompactor.PhaseOneResult<T>

Field Summary

Fields

Modifier and Type	Field	Description
protected static final int	MAX_OUTSTANDING
protected final Duration	phaseOneLoopReadTimeout
protected final boolean	topicCompactionRetainNullKey

Fields inherited from class org.apache.pulsar.compaction.Compactor

bk, COMPACTED_TOPIC_LEDGER_PROPERTY, COMPACTION_SUBSCRIPTION, conf, mxBean, pulsar, scheduler

Constructor Summary

Constructors

Constructor	Description
AbstractTwoPhaseCompactor(ServiceConfiguration conf, org.apache.pulsar.client.api.PulsarClient pulsar, BookKeeper bk, ScheduledExecutorService scheduler)

Method Summary

Modifier and Type	Method	Description
protected CompletableFuture<Void>	closeLedger(LedgerHandle lh)
protected abstract boolean	compactBatchMessage(String topic, Map<String,T> latestForKey, RawMessage m, org.apache.pulsar.common.api.proto.MessageMetadata metadata, org.apache.pulsar.client.api.MessageId id)
protected abstract boolean	compactMessage(String topic, Map<String,T> latestForKey, RawMessage m, org.apache.pulsar.common.api.proto.MessageMetadata metadata, org.apache.pulsar.client.api.MessageId id)
protected CompletableFuture<LedgerHandle>	createLedger(BookKeeper bk, Map<String,byte[]> metadata)
protected CompletableFuture<Void>	deleteLedger(BookKeeper bk, LedgerHandle lh)
protected CompletableFuture<Long>	doCompaction(RawReader reader, BookKeeper bk)
protected org.apache.commons.lang3.tuple.Pair<String,Integer>	extractKeyAndSize(RawMessage m, org.apache.pulsar.common.api.proto.MessageMetadata msgMetadata)
long	getPhaseOneLoopReadTimeoutInSeconds()
protected Optional<RawMessage>	rebatchMessage(String topic, RawMessage msg, org.apache.pulsar.common.api.proto.MessageMetadata metadata, BiPredicate<String,org.apache.pulsar.client.api.MessageId> filter, boolean retainNullKey)
protected abstract Map<String,org.apache.pulsar.client.api.MessageId>	toLatestMessageIdForKey(Map<String,T> latestForKey)

Methods inherited from class org.apache.pulsar.compaction.Compactor

compact, getStats

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

MAX_OUTSTANDING

protected static final int MAX_OUTSTANDING

See Also:

• Constant Field Values

phaseOneLoopReadTimeout

protected final Duration phaseOneLoopReadTimeout

topicCompactionRetainNullKey

protected final boolean topicCompactionRetainNullKey

Constructor Details

AbstractTwoPhaseCompactor

public AbstractTwoPhaseCompactor(ServiceConfiguration conf,
 org.apache.pulsar.client.api.PulsarClient pulsar,
 BookKeeper bk,
 ScheduledExecutorService scheduler)

Method Details

toLatestMessageIdForKey

protected abstract Map<String,org.apache.pulsar.client.api.MessageId> toLatestMessageIdForKey(Map<String,T> latestForKey)

compactMessage

protected abstract boolean compactMessage(String topic,
 Map<String,T> latestForKey,
 RawMessage m,
 org.apache.pulsar.common.api.proto.MessageMetadata metadata,
 org.apache.pulsar.client.api.MessageId id)

compactBatchMessage

protected abstract boolean compactBatchMessage(String topic,
 Map<String,T> latestForKey,
 RawMessage m,
 org.apache.pulsar.common.api.proto.MessageMetadata metadata,
 org.apache.pulsar.client.api.MessageId id)

doCompaction

protected CompletableFuture<Long> doCompaction(RawReader reader,
 BookKeeper bk)

Specified by:

doCompaction in class Compactor

createLedger

protected CompletableFuture<LedgerHandle> createLedger(BookKeeper bk,
 Map<String,byte[]> metadata)

deleteLedger

protected CompletableFuture<Void> deleteLedger(BookKeeper bk,
 LedgerHandle lh)

closeLedger

protected CompletableFuture<Void> closeLedger(LedgerHandle lh)

extractKeyAndSize

protected org.apache.commons.lang3.tuple.Pair<String,Integer> extractKeyAndSize(RawMessage m,
 org.apache.pulsar.common.api.proto.MessageMetadata msgMetadata)

rebatchMessage

protected Optional<RawMessage> rebatchMessage(String topic,
 RawMessage msg,
 org.apache.pulsar.common.api.proto.MessageMetadata metadata,
 BiPredicate<String,org.apache.pulsar.client.api.MessageId> filter,
 boolean retainNullKey)
                                       throws IOException

Throws:

IOException

getPhaseOneLoopReadTimeoutInSeconds

public long getPhaseOneLoopReadTimeoutInSeconds()