class TransportConf extends AnyRef

A central location that tracks all the settings we expose to users.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. TransportConf
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new TransportConf(module: String, conf: ConfigProvider)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def authEngineVersion(): Int

    Version number to be used by the AuthEngine key agreement protocol.

    Version number to be used by the AuthEngine key agreement protocol. Valid values are 1 or 2. The default version is 1 for backward compatibility. Version 2 is recommended for stronger security properties.

  6. def authRTTimeoutMs(): Int

    Timeout for a single round trip of auth message exchange, in milliseconds.

  7. def backLog(): Int

    Requested maximum length of the queue of incoming connections.

    Requested maximum length of the queue of incoming connections. If < 1, the default Netty value of io.netty.util.NetUtil#SOMAXCONN will be used. Default to -1.

  8. def chunkFetchHandlerThreads(): Int

    Percentage of io.serverThreads used by netty to process ChunkFetchRequest.

    Percentage of io.serverThreads used by netty to process ChunkFetchRequest. When the config spark.shuffle.server.chunkFetchHandlerThreadsPercent is set, shuffle server will use a separate EventLoopGroup to process ChunkFetchRequest messages. Although when calling the async writeAndFlush on the underlying channel to send response back to client, the I/O on the channel is still being handled by org.apache.spark.network.server.TransportServer's default EventLoopGroup that's registered with the Channel, by waiting inside the ChunkFetchRequest handler threads for the completion of sending back responses, we are able to put a limit on the max number of threads from TransportServer's default EventLoopGroup that are going to be consumed by writing response to ChunkFetchRequest, which are I/O intensive and could take long time to process due to disk contentions. By configuring a slightly higher number of shuffler server threads, we are able to reserve some threads for handling other RPC messages, thus making the Client less likely to experience timeout when sending RPC messages to the shuffle server. The number of threads used for handling chunked fetch requests are percentage of io.serverThreads (if defined) else it is a percentage of 2 * #cores. However, a percentage of 0 means netty default number of threads which is 2 * #cores ignoring io.serverThreads. The percentage here is configured via spark.shuffle.server.chunkFetchHandlerThreadsPercent. The returned value is rounded off to ceiling of the nearest integer.

  9. def cipherTransformation(): String

    The cipher transformation to use for encrypting session data.

  10. def clientThreads(): Int

    Number of threads used in the client thread pool.

    Number of threads used in the client thread pool. Default to 0, which is 2x#cores.

  11. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  12. def connectionCreationTimeoutMs(): Int

    Connect creation timeout in milliseconds.

    Connect creation timeout in milliseconds. Default 120 secs.

  13. def connectionTimeoutMs(): Int

    Connection idle timeout in milliseconds.

    Connection idle timeout in milliseconds. Default 120 secs.

  14. def cryptoConf(): Properties

    The commons-crypto configuration for the module.

  15. def enableSaslRetries(): Boolean

    Whether to enable sasl retries or not.

    Whether to enable sasl retries or not. The number of retries is dictated by the config spark.shuffle.io.maxRetries.

  16. def enableTcpKeepAlive(): Boolean

    Whether to enable TCP keep-alive.

    Whether to enable TCP keep-alive. If true, the TCP keep-alives are enabled, which removes connections that are idle for too long.

  17. def encryptionEnabled(): Boolean

    Enables strong encryption.

    Enables strong encryption. Also enables the new auth protocol, used to negotiate keys.

  18. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  19. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  20. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  21. def get(name: String, defaultValue: String): String
  22. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  23. def getInt(name: String, defaultValue: Int): Int
  24. def getModuleName(): String
  25. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  26. def ioExceptionsThresholdDuringMerge(): Int

    The threshold for number of IOExceptions while merging shuffle blocks to a shuffle partition.

    The threshold for number of IOExceptions while merging shuffle blocks to a shuffle partition. When the number of IOExceptions while writing to merged shuffle data/index/meta file exceed this threshold then the shuffle server will respond back to client to stop pushing shuffle blocks for this shuffle partition.

  27. def ioMode(): String

    IO mode: nio or epoll

  28. def ioRetryWaitTimeMs(): Int

    Time (in milliseconds) that we will wait in order to perform a retry after an IOException.

    Time (in milliseconds) that we will wait in order to perform a retry after an IOException. Only relevant if maxIORetries > 0.

  29. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  30. def lazyFileDescriptor(): Boolean

    Whether to initialize FileDescriptor lazily or not.

    Whether to initialize FileDescriptor lazily or not. If true, file descriptors are created only when data is going to be transferred. This can reduce the number of open files.

  31. def maxChunksBeingTransferred(): Long

    The max number of chunks allowed to be transferred at the same time on shuffle service.

    The max number of chunks allowed to be transferred at the same time on shuffle service. Note that new incoming connections will be closed when the max number is hit. The client will retry according to the shuffle retry configs (see spark.shuffle.io.maxRetries and spark.shuffle.io.retryWait), if those limits are reached the task will fail with fetch failure.

  32. def maxIORetries(): Int

    Max number of times we will try IO exceptions (such as connection timeouts) per request.

    Max number of times we will try IO exceptions (such as connection timeouts) per request. If set to 0, we will not do any retries.

  33. def maxSaslEncryptedBlockSize(): Int

    Maximum number of bytes to be encrypted at a time when SASL encryption is used.

  34. def memoryMapBytes(): Int

    Minimum size of a block that we should start using memory map rather than reading in through normal IO operations.

    Minimum size of a block that we should start using memory map rather than reading in through normal IO operations. This prevents Spark from memory mapping very small blocks. In general, memory mapping has high overhead for blocks close to or below the page size of the OS.

  35. def mergedIndexCacheSize(): Long

    The maximum size of cache in memory which is used in push-based shuffle for storing merged index files.

    The maximum size of cache in memory which is used in push-based shuffle for storing merged index files. This cache is in addition to the one configured via spark.shuffle.service.index.cache.size.

  36. def mergedShuffleCleanerShutdownTimeout(): Long

    The RemoteBlockPushResolver#mergedShuffleCleanermergedShuffleCleaner shutdown timeout, in seconds.

  37. def mergedShuffleFileManagerImpl(): String

    Class name of the implementation of MergedShuffleFileManager that merges the blocks pushed to it when push-based shuffle is enabled.

    Class name of the implementation of MergedShuffleFileManager that merges the blocks pushed to it when push-based shuffle is enabled. By default, push-based shuffle is disabled at a cluster level because this configuration is set to 'org.apache.spark.network.shuffle.NoOpMergedShuffleFileManager'. To turn on push-based shuffle at a cluster level, set the configuration to 'org.apache.spark.network.shuffle.RemoteBlockPushResolver'.

  38. def minChunkSizeInMergedShuffleFile(): Int

    The minimum size of a chunk when dividing a merged shuffle file into multiple chunks during push-based shuffle.

    The minimum size of a chunk when dividing a merged shuffle file into multiple chunks during push-based shuffle. A merged shuffle file consists of multiple small shuffle blocks. Fetching the complete merged shuffle file in a single disk I/O increases the memory requirements for both the clients and the external shuffle service. Instead, the external shuffle service serves the merged file in MB-sized chunks. This configuration controls how big a chunk can get. A corresponding index file for each merged shuffle file will be generated indicating chunk boundaries.

    Setting this too high would increase the memory requirements on both the clients and the external shuffle service.

    Setting this too low would increase the overall number of RPC requests to external shuffle service unnecessarily.

  39. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  40. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  41. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  42. def numConnectionsPerPeer(): Int

    Number of concurrent connections between two nodes for fetching data.

  43. def portMaxRetries(): Int

    Maximum number of retries when binding to a port before giving up.

  44. def preferDirectBufs(): Boolean

    If true, we will prefer allocating off-heap byte buffers within Netty.

  45. def preferDirectBufsForSharedByteBufAllocators(): Boolean

    If enabled then off-heap byte buffers will be preferred for the shared ByteBuf allocators.

  46. def receiveBuf(): Int

    Receive buffer size (SO_RCVBUF).

    Receive buffer size (SO_RCVBUF). Note: the optimal size for receive buffer and send buffer should be latency * network_bandwidth. Assuming latency = 1ms, network_bandwidth = 10Gbps buffer size should be ~ 1.25MB

  47. def saslEncryption(): Boolean

    Whether to enable SASL-based encryption when authenticating using SASL.

  48. def saslFallback(): Boolean

    Whether to fall back to SASL if the new auth protocol fails.

    Whether to fall back to SASL if the new auth protocol fails. Enabled by default for backwards compatibility.

  49. def saslServerAlwaysEncrypt(): Boolean

    Whether the server should enforce encryption on SASL-authenticated connections.

  50. def sendBuf(): Int

    Send buffer size (SO_SNDBUF).

  51. def separateChunkFetchRequest(): Boolean

    Whether to use a separate EventLoopGroup to process ChunkFetchRequest messages, it is decided by the config spark.shuffle.server.chunkFetchHandlerThreadsPercent is set or not.

  52. def serverThreads(): Int

    Number of threads used in the server thread pool.

    Number of threads used in the server thread pool. Default to 0, which is 2x#cores.

  53. def sharedByteBufAllocators(): Boolean

    Flag indicating whether to share the pooled ByteBuf allocators between the different Netty channels.

    Flag indicating whether to share the pooled ByteBuf allocators between the different Netty channels. If enabled then only two pooled ByteBuf allocators are created: one where caching is allowed (for transport servers) and one where not (for transport clients). When disabled a new allocator is created for each transport servers and clients.

  54. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  55. def toString(): String
    Definition Classes
    AnyRef → Any
  56. def useOldFetchProtocol(): Boolean

    Whether to use the old protocol while doing the shuffle block fetching.

    Whether to use the old protocol while doing the shuffle block fetching. It is only enabled while we need the compatibility in the scenario of new spark version job fetching blocks from old version external shuffle service.

  57. def verboseMetrics(): Boolean

    Whether to track Netty memory detailed metrics.

    Whether to track Netty memory detailed metrics. If true, the detailed metrics of Netty PoolByteBufAllocator will be gotten, otherwise only general memory usage will be tracked.

  58. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  59. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  60. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped