Package org.apache.pulsar.broker.loadbalance.extensions.scheduler

Class TransferShedder

java.lang.Object

org.apache.pulsar.broker.loadbalance.extensions.scheduler.TransferShedder

All Implemented Interfaces:

NamespaceUnloadStrategy

public class TransferShedder
extends Object
implements NamespaceUnloadStrategy

Load shedding strategy that unloads bundles from the highest loaded brokers. This strategy is only configurable in the broker load balancer extensions introduced by PIP-192[https://github.com/apache/pulsar/issues/16691]. This load shedding strategy has the following goals: 1. Distribute bundle load across brokers in order to make the standard deviation of the avg resource usage, std(exponential-moving-avg(max(cpu, memory, network, throughput)) for each broker) below the target, configurable by loadBalancerBrokerLoadTargetStd. 2. Use the transfer protocol to transfer bundle load from the highest loaded to the lowest loaded brokers, if configured by loadBalancerTransferEnabled=true. 3. Avoid repeated bundle unloading by recomputing historical broker resource usage after unloading and also skipping the bundles that are recently unloaded. 4. Prioritize unloading bundles to underloaded brokers when their message throughput is zero(new brokers). 5. Do not use outdated broker load data (configurable by loadBalancerBrokerLoadDataTTLInSeconds). 6. Give enough time for each broker to recompute its load after unloading (configurable by loadBalanceUnloadDelayInSeconds) 7. Do not transfer bundles with namespace isolation policies or anti-affinity group policies. 8. Limit the max number of brokers to transfer bundle load for each cycle, (loadBalancerMaxNumberOfBrokerTransfersPerCycle). 9. Print more logs with a debug option(loadBalancerDebugModeEnabled=true).

Constructor Summary

Constructors

Constructor	Description
TransferShedder(UnloadCounter counter)
TransferShedder(PulsarService pulsar, UnloadCounter counter, List<BrokerFilter> brokerFilterPipeline, IsolationPoliciesHelper isolationPoliciesHelper, AntiAffinityGroupPolicyHelper antiAffinityGroupPolicyHelper)

Method Summary

Modifier and Type	Method	Description
Set<UnloadDecision>	findBundlesForUnloading(LoadManagerContext context, Map<String,Long> recentlyUnloadedBundles, Map<String,Long> recentlyUnloadedBrokers)	Recommend that all the returned bundles be unloaded.
void	initialize(PulsarService pulsar)	Initializes the internals.
protected boolean	isLoadBalancerSheddingBundlesWithPoliciesEnabled(LoadManagerContext context, NamespaceBundle namespaceBundle)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

TransferShedder

public TransferShedder(UnloadCounter counter)

TransferShedder

public TransferShedder(PulsarService pulsar,
 UnloadCounter counter,
 List<BrokerFilter> brokerFilterPipeline,
 IsolationPoliciesHelper isolationPoliciesHelper,
 AntiAffinityGroupPolicyHelper antiAffinityGroupPolicyHelper)

Method Details

initialize

public void initialize(PulsarService pulsar)

Description copied from interface: NamespaceUnloadStrategy

Initializes the internals.

Specified by:

initialize in interface NamespaceUnloadStrategy

Parameters:

pulsar - The pulsar service instance.

findBundlesForUnloading

public Set<UnloadDecision> findBundlesForUnloading(LoadManagerContext context,
 Map<String,Long> recentlyUnloadedBundles,
 Map<String,Long> recentlyUnloadedBrokers)

Description copied from interface: NamespaceUnloadStrategy

Recommend that all the returned bundles be unloaded.

Specified by:

findBundlesForUnloading in interface NamespaceUnloadStrategy

Parameters:

context - The context used for decisions.

recentlyUnloadedBundles - The recently unloaded bundles.

recentlyUnloadedBrokers - The recently unloaded brokers.

Returns:

unloadDecision containing a list of the bundles that should be unloaded.

isLoadBalancerSheddingBundlesWithPoliciesEnabled

protected boolean isLoadBalancerSheddingBundlesWithPoliciesEnabled(LoadManagerContext context,
 NamespaceBundle namespaceBundle)