c

org.apache.spark.network.client

TransportClient

class TransportClient extends Closeable

Client for fetching consecutive chunks of a pre-negotiated stream. This API is intended to allow efficient transfer of a large amount of data, broken up into chunks with size ranging from hundreds of KB to a few MB.

Note that while this client deals with the fetching of chunks from a stream (i.e., data plane), the actual setup of the streams is done outside the scope of the transport layer. The convenience method "sendRPC" is provided to enable control plane communication between the client and server to perform this setup.

For example, a typical workflow might be: client.sendRPC(new OpenFile("/foo")) --> returns StreamId = 100 client.fetchChunk(streamId = 100, chunkIndex = 0, callback) client.fetchChunk(streamId = 100, chunkIndex = 1, callback) ... client.sendRPC(new CloseStream(100))

Construct an instance of TransportClient using TransportClientFactory. A single TransportClient may be used for multiple streams, but any given stream must be restricted to a single client, in order to avoid out-of-order responses.

NB: This class is used to make requests to the server, while TransportResponseHandler is responsible for handling responses from the server.

Concurrency: thread safe and can be called from multiple threads.

Linear Supertypes
Closeable, AutoCloseable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. TransportClient
  2. Closeable
  3. AutoCloseable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new TransportClient(channel: Channel, handler: TransportResponseHandler)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. def close(): Unit
    Definition Classes
    TransportClient → Closeable → AutoCloseable
    Annotations
    @Override()
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. def fetchChunk(streamId: Long, chunkIndex: Int, callback: ChunkReceivedCallback): Unit

    Requests a single chunk from the remote side, from the pre-negotiated streamId.

    Requests a single chunk from the remote side, from the pre-negotiated streamId.

    Chunk indices go from 0 onwards. It is valid to request the same chunk multiple times, though some streams may not support this.

    Multiple fetchChunk requests may be outstanding simultaneously, and the chunks are guaranteed to be returned in the same order that they were requested, assuming only a single TransportClient is used to fetch the chunks.

    streamId

    Identifier that refers to a stream in the remote StreamManager. This should be agreed upon by client and server beforehand.

    chunkIndex

    0-based index of the chunk to fetch

    callback

    Callback invoked upon successful receipt of chunk, or upon any failure.

  10. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. def getChannel(): Channel
  12. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. def getClientId(): String

    Returns the ID used by the client to authenticate itself when authentication is enabled.

    Returns the ID used by the client to authenticate itself when authentication is enabled.

    returns

    The client ID, or null if authentication is disabled.

  14. def getHandler(): TransportResponseHandler
    Annotations
    @VisibleForTesting()
  15. def getSocketAddress(): SocketAddress
  16. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  17. def isActive(): Boolean
  18. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  19. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  21. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  22. def removeRpcRequest(requestId: Long): Unit

    Removes any state associated with the given RPC.

    Removes any state associated with the given RPC.

    requestId

    The RPC id returned by RpcResponseCallback).

  23. def send(message: ByteBuffer): Unit

    Sends an opaque message to the RpcHandler on the server-side.

    Sends an opaque message to the RpcHandler on the server-side. No reply is expected for the message, and no delivery guarantees are made.

    message

    The message to send.

  24. def sendMergedBlockMetaReq(appId: String, shuffleId: Int, shuffleMergeId: Int, reduceId: Int, callback: MergedBlockMetaResponseCallback): Unit

    Sends a MergedBlockMetaRequest message to the server.

    Sends a MergedBlockMetaRequest message to the server. The response of this message is either a MergedBlockMetaSuccess or RpcFailure.

    appId

    applicationId.

    shuffleId

    shuffle id.

    shuffleMergeId

    shuffleMergeId is used to uniquely identify merging process of shuffle by an indeterminate stage attempt.

    reduceId

    reduce id.

    callback

    callback the handle the reply.

  25. def sendRpc(message: ByteBuffer, callback: RpcResponseCallback): Long

    Sends an opaque message to the RpcHandler on the server-side.

    Sends an opaque message to the RpcHandler on the server-side. The callback will be invoked with the server's response or upon any failure.

    message

    The message to send.

    callback

    Callback to handle the RPC's reply.

    returns

    The RPC's id.

  26. def sendRpcSync(message: ByteBuffer, timeoutMs: Long): ByteBuffer

    Synchronously sends an opaque message to the RpcHandler on the server-side, waiting for up to a specified timeout for a response.

  27. def setClientId(id: String): Unit

    Sets the authenticated client ID.

    Sets the authenticated client ID. This is meant to be used by the authentication layer.

    Trying to set a different client ID after it's been set will result in an exception.

  28. def stream(streamId: String, callback: StreamCallback): Unit

    Request to stream the data with the given stream ID from the remote end.

    Request to stream the data with the given stream ID from the remote end.

    streamId

    The stream to fetch.

    callback

    Object to call with the stream data.

  29. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  30. def timeOut(): Unit

    Mark this channel as having timed out.

  31. def toString(): String
    Definition Classes
    TransportClient → AnyRef → Any
    Annotations
    @Override()
  32. def uploadStream(meta: ManagedBuffer, data: ManagedBuffer, callback: RpcResponseCallback): Long

    Send data to the remote end as a stream.

    Send data to the remote end as a stream. This differs from stream() in that this is a request to *send* data to the remote end, not to receive it from the remote.

    meta

    meta data associated with the stream, which will be read completely on the receiving end before the stream itself.

    data

    this will be streamed to the remote end to allow for transferring large amounts of data without reading into memory.

    callback

    handles the reply -- onSuccess will only be called when both message and data are received successfully.

  33. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  34. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  35. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Closeable

Inherited from AutoCloseable

Inherited from AnyRef

Inherited from Any

Ungrouped