c

org.apache.spark.sql.delta.commands

ConvertToDeltaCommandBase

abstract class ConvertToDeltaCommandBase extends LogicalPlan with LeafRunnableCommand with DeltaCommand

Convert an existing parquet table to a delta table by creating delta logs based on existing files. Here are the main components:

  • File Listing: Launch a spark job to list files from a given directory in parallel.
  • Schema Inference: Given an iterator on the file list result, we group the iterator into sequential batches and launch a spark job to infer schema for each batch, and finally merge schemas from all batches.
  • Stats collection: Again, we group the iterator on file list results into sequential batches and launch a spark job to collect stats for each batch.
  • Commit the files: We take the iterator of files with stats and write out a delta log file as the first commit. This bypasses the transaction protocol, but it's ok as this would be the very first commit.
Linear Supertypes
DeltaCommand, DeltaLogging, DatabricksLogging, DeltaProgressReporter, LeafRunnableCommand, LeafLike[LogicalPlan], RunnableCommand, Command, LogicalPlan, Logging, QueryPlanConstraints, ConstraintHelper, LogicalPlanStats, AnalysisHelper, QueryPlan[LogicalPlan], SQLConfHelper, TreeNode[LogicalPlan], TreePatternBits, Product, Equals, AnyRef, Any
Known Subclasses
ConvertToDeltaCommand
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ConvertToDeltaCommandBase
  2. DeltaCommand
  3. DeltaLogging
  4. DatabricksLogging
  5. DeltaProgressReporter
  6. LeafRunnableCommand
  7. LeafLike
  8. RunnableCommand
  9. Command
  10. LogicalPlan
  11. Logging
  12. QueryPlanConstraints
  13. ConstraintHelper
  14. LogicalPlanStats
  15. AnalysisHelper
  16. QueryPlan
  17. SQLConfHelper
  18. TreeNode
  19. TreePatternBits
  20. Product
  21. Equals
  22. AnyRef
  23. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new ConvertToDeltaCommandBase(tableIdentifier: TableIdentifier, partitionSchema: Option[StructType], deltaPath: Option[String])

    tableIdentifier

    the target parquet table.

    partitionSchema

    the partition schema of the table, required when table is partitioned.

    deltaPath

    if provided, the delta log will be written to this location.

Type Members

  1. case class ConvertTarget(catalogTable: Option[CatalogTable], provider: Option[String], targetDir: String, properties: Map[String, String]) extends Product with Serializable
    Attributes
    protected

Abstract Value Members

  1. abstract def canEqual(that: Any): Boolean
    Definition Classes
    Equals
  2. abstract def productArity: Int
    Definition Classes
    Product
  3. abstract def productElement(n: Int): Any
    Definition Classes
    Product

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. lazy val allAttributes: AttributeSeq
    Definition Classes
    QueryPlan
  5. def analyzed: Boolean
    Definition Classes
    AnalysisHelper
  6. def apply(number: Int): TreeNode[_]
    Definition Classes
    TreeNode
  7. def argString(maxFields: Int): String
    Definition Classes
    TreeNode
  8. def asCode: String
    Definition Classes
    TreeNode
  9. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  10. def assertNotAnalysisRule(): Unit
    Attributes
    protected
    Definition Classes
    AnalysisHelper
  11. def buildBaseRelation(spark: SparkSession, txn: OptimisticTransaction, actionType: String, rootPath: Path, inputLeafFiles: Seq[String], nameToAddFileMap: Map[String, AddFile]): HadoopFsRelation

    Build a base relation of files that need to be rewritten as part of an update/delete/merge operation.

    Build a base relation of files that need to be rewritten as part of an update/delete/merge operation.

    Attributes
    protected
    Definition Classes
    DeltaCommand
  12. final lazy val canonicalized: LogicalPlan
    Definition Classes
    QueryPlan
    Annotations
    @transient()
  13. final def children: Seq[LogicalPlan]
    Definition Classes
    LeafLike
  14. def childrenResolved: Boolean
    Definition Classes
    LogicalPlan
  15. def clone(): LogicalPlan
    Definition Classes
    AnalysisHelper → TreeNode → AnyRef
  16. def collect[B](pf: PartialFunction[LogicalPlan, B]): Seq[B]
    Definition Classes
    TreeNode
  17. def collectFirst[B](pf: PartialFunction[LogicalPlan, B]): Option[B]
    Definition Classes
    TreeNode
  18. def collectLeaves(): Seq[LogicalPlan]
    Definition Classes
    TreeNode
  19. def collectWithSubqueries[B](f: PartialFunction[LogicalPlan, B]): Seq[B]
    Definition Classes
    QueryPlan
  20. def commitLarge(spark: SparkSession, txn: OptimisticTransaction, actions: Iterator[Action], op: Operation, context: Map[String, String], metrics: Map[String, String]): Long

    Create a large commit on the Delta log by directly writing an iterator of FileActions to the LogStore.

    Create a large commit on the Delta log by directly writing an iterator of FileActions to the LogStore. This function only commits the next possible version and will not check whether the commit is retry-able. If the next version has already been committed, then this function will fail. This bypasses all optimistic concurrency checks. We assume that transaction conflicts should be rare because this method is typically used to create new tables (e.g. CONVERT TO DELTA) or apply some commands which rarely receive other transactions (e.g. CLONE/RESTORE).

    Attributes
    protected
    Definition Classes
    DeltaCommand
  21. def conf: SQLConf
    Definition Classes
    SQLConfHelper
  22. lazy val constraints: ExpressionSet
    Definition Classes
    QueryPlanConstraints
  23. def constructIsNotNullConstraints(constraints: ExpressionSet, output: Seq[Attribute]): ExpressionSet
    Definition Classes
    ConstraintHelper
  24. final def containsAllPatterns(patterns: TreePattern*): Boolean
    Definition Classes
    TreePatternBits
  25. final def containsAnyPattern(patterns: TreePattern*): Boolean
    Definition Classes
    TreePatternBits
  26. lazy val containsChild: Set[TreeNode[_]]
    Definition Classes
    TreeNode
  27. final def containsPattern(t: TreePattern): Boolean
    Definition Classes
    TreePatternBits
    Annotations
    @inline()
  28. def copyTagsFrom(other: LogicalPlan): Unit
    Definition Classes
    TreeNode
  29. def createDeltaActions(manifest: ConvertTargetFileManifest, partitionSchema: StructType, txn: OptimisticTransaction, fs: FileSystem): Iterator[AddFile]

    Given the file manifest, create corresponding AddFile actions for the entire list of files.

    Given the file manifest, create corresponding AddFile actions for the entire list of files.

    Attributes
    protected
  30. def doCanonicalize(): LogicalPlan
    Attributes
    protected
    Definition Classes
    QueryPlan
  31. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  32. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  33. final def expressions: Seq[Expression]
    Definition Classes
    QueryPlan
  34. def fastEquals(other: TreeNode[_]): Boolean
    Definition Classes
    TreeNode
  35. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  36. def find(f: (LogicalPlan) => Boolean): Option[LogicalPlan]
    Definition Classes
    TreeNode
  37. def flatMap[A](f: (LogicalPlan) => TraversableOnce[A]): Seq[A]
    Definition Classes
    TreeNode
  38. def foreach(f: (LogicalPlan) => Unit): Unit
    Definition Classes
    TreeNode
  39. def foreachUp(f: (LogicalPlan) => Unit): Unit
    Definition Classes
    TreeNode
  40. def formattedNodeName: String
    Attributes
    protected
    Definition Classes
    QueryPlan
  41. def generateCandidateFileMap(basePath: Path, candidateFiles: Seq[AddFile]): Map[String, AddFile]

    Generates a map of file names to add file entries for operations where we will need to rewrite files such as delete, merge, update.

    Generates a map of file names to add file entries for operations where we will need to rewrite files such as delete, merge, update. We expect file names to be unique, because each file contains a UUID.

    Attributes
    protected
    Definition Classes
    DeltaCommand
  42. def generateTreeString(depth: Int, lastChildren: Seq[Boolean], append: (String) => Unit, verbose: Boolean, prefix: String, addSuffix: Boolean, maxFields: Int, printNodeId: Boolean, indent: Int): Unit
    Definition Classes
    TreeNode
  43. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  44. def getContext: Map[String, String]
    Attributes
    protected
  45. def getDefaultTreePatternBits: BitSet
    Attributes
    protected
    Definition Classes
    TreeNode
  46. def getDeltaLog(spark: SparkSession, path: Option[String], tableIdentifier: Option[TableIdentifier], operationName: String): DeltaLog

    Utility method to return the DeltaLog of an existing Delta table referred by either the given path or

    Utility method to return the DeltaLog of an existing Delta table referred by either the given path or

    spark

    SparkSession reference to use.

    path

    Table location. Expects a non-empty tableIdentifier or path.

    tableIdentifier

    Table identifier. Expects a non-empty tableIdentifier or path.

    operationName

    Operation that is getting the DeltaLog, used in error messages.

    returns

    DeltaLog of the table

    Attributes
    protected
    Definition Classes
    DeltaCommand
    Exceptions thrown

    AnalysisException If either no Delta table exists at the given path/identifier or there is neither path nor tableIdentifier is provided.

  47. def getOperation(numFilesConverted: Long, convertProperties: ConvertTarget): Operation

    Get the operation to store in the commit message.

    Get the operation to store in the commit message.

    Attributes
    protected
  48. def getTagValue[T](tag: TreeNodeTag[T]): Option[T]
    Definition Classes
    TreeNode
  49. def getTargetTable(spark: SparkSession, target: ConvertTarget): ConvertTargetTable

    Get the instance of the convert target table, which provides file manifest and schema

    Get the instance of the convert target table, which provides file manifest and schema

    Attributes
    protected
  50. def getTouchedFile(basePath: Path, filePath: String, nameToAddFileMap: Map[String, AddFile]): AddFile

    Find the AddFile record corresponding to the file that was read as part of a delete/update/merge operation.

    Find the AddFile record corresponding to the file that was read as part of a delete/update/merge operation.

    filePath

    The path to a file. Can be either absolute or relative

    nameToAddFileMap

    Map generated through generateCandidateFileMap()

    Attributes
    protected
    Definition Classes
    DeltaCommand
  51. def hashCode(): Int
    Definition Classes
    TreeNode → AnyRef → Any
  52. def inferAdditionalConstraints(constraints: ExpressionSet): ExpressionSet
    Definition Classes
    ConstraintHelper
  53. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  54. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  55. def innerChildren: Seq[QueryPlan[_]]
    Definition Classes
    QueryPlan → TreeNode
  56. def inputSet: AttributeSet
    Definition Classes
    QueryPlan
  57. final def invalidateStatsCache(): Unit
    Definition Classes
    LogicalPlanStats
  58. def isCanonicalizedPlan: Boolean
    Attributes
    protected
    Definition Classes
    QueryPlan
  59. def isCatalogTable(analyzer: Analyzer, tableIdent: TableIdentifier): Boolean

    Calls DeltaCommand.isCatalogTable.

    Calls DeltaCommand.isCatalogTable. With Convert, we may get a format check error in cases where the metastore and the underlying table don't align, e.g. external table where the underlying files are converted to delta but the metadata has not been converted yet. In these cases, catch the error and return based on whether the provided Table Identifier could reasonably be a path

    analyzer

    The session state analyzer to call

    tableIdent

    Table Identifier to determine whether is path based or not

    returns

    Boolean where true means that the table is a table in a metastore and false means the table is a path based table

    Definition Classes
    ConvertToDeltaCommandBase → DeltaCommand
  60. def isHiveStyleParquetTable(catalogTable: CatalogTable): Boolean
    Attributes
    protected
  61. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  62. def isPathIdentifier(tableIdent: TableIdentifier): Boolean

    Override this method since parquet paths are valid for Convert

    Override this method since parquet paths are valid for Convert

    tableIdent

    the provided table or path

    returns

    Whether or not the ident provided can refer to a table by path

    Definition Classes
    ConvertToDeltaCommandBase → DeltaCommand
  63. def isRuleIneffective(ruleId: RuleId): Boolean
    Attributes
    protected
    Definition Classes
    TreeNode
  64. def isStreaming: Boolean
    Definition Classes
    LogicalPlan
  65. def isSupportedProvider(lowerCaseProvider: String): Boolean
    Attributes
    protected
  66. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  67. def jsonFields: List[JField]
    Attributes
    protected
    Definition Classes
    TreeNode
  68. final def legacyWithNewChildren(newChildren: Seq[LogicalPlan]): LogicalPlan
    Attributes
    protected
    Definition Classes
    TreeNode
  69. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  70. def logConsole(line: String): Unit
    Definition Classes
    DatabricksLogging
  71. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  72. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  73. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  74. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  75. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  76. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  77. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  78. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  79. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  80. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  81. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  82. def makeCopy(newArgs: Array[AnyRef]): LogicalPlan
    Definition Classes
    TreeNode
  83. def map[A](f: (LogicalPlan) => A): Seq[A]
    Definition Classes
    TreeNode
  84. final def mapChildren(f: (LogicalPlan) => LogicalPlan): LogicalPlan
    Definition Classes
    LeafLike
  85. def mapExpressions(f: (Expression) => Expression): ConvertToDeltaCommandBase.this.type
    Definition Classes
    QueryPlan
  86. def mapProductIterator[B](f: (Any) => B)(implicit arg0: ClassTag[B]): Array[B]
    Attributes
    protected
    Definition Classes
    TreeNode
  87. def markRuleAsIneffective(ruleId: RuleId): Unit
    Attributes
    protected
    Definition Classes
    TreeNode
  88. def maxRows: Option[Long]
    Definition Classes
    LogicalPlan
  89. def maxRowsPerPartition: Option[Long]
    Definition Classes
    LogicalPlan
  90. def metadataOutput: Seq[Attribute]
    Definition Classes
    LogicalPlan
  91. lazy val metrics: Map[String, SQLMetric]
    Definition Classes
    RunnableCommand
  92. final def missingInput: AttributeSet
    Definition Classes
    QueryPlan
  93. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  94. def nodeName: String
    Definition Classes
    TreeNode
  95. final val nodePatterns: Seq[TreePattern]
    Definition Classes
    Command → TreeNode
  96. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  97. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  98. def numberedTreeString: String
    Definition Classes
    TreeNode
  99. val origin: Origin
    Definition Classes
    TreeNode
  100. def otherCopyArgs: Seq[AnyRef]
    Attributes
    protected
    Definition Classes
    TreeNode
  101. def output: Seq[Attribute]
    Definition Classes
    Command → QueryPlan
  102. def outputOrdering: Seq[SortOrder]
    Definition Classes
    LogicalPlan
  103. lazy val outputSet: AttributeSet
    Definition Classes
    QueryPlan
    Annotations
    @transient()
  104. def p(number: Int): LogicalPlan
    Definition Classes
    TreeNode
  105. def parsePredicates(spark: SparkSession, predicate: String): Seq[Expression]

    Converts string predicates into Expressions relative to a transaction.

    Converts string predicates into Expressions relative to a transaction.

    Attributes
    protected
    Definition Classes
    DeltaCommand
    Exceptions thrown

    AnalysisException if a non-partition column is referenced.

  106. def prettyJson: String
    Definition Classes
    TreeNode
  107. def printSchema(): Unit
    Definition Classes
    QueryPlan
  108. def producedAttributes: AttributeSet
    Definition Classes
    Command → QueryPlan
  109. def productElementName(n: Int): String
    Definition Classes
    Product
  110. def productElementNames: Iterator[String]
    Definition Classes
    Product
  111. def productIterator: Iterator[Any]
    Definition Classes
    Product
  112. def productPrefix: String
    Definition Classes
    Product
  113. def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null, path: Option[Path] = None): Unit

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    path

    Used to log the path of the delta table when deltaLog is null.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  114. def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  115. def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  116. def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
    Definition Classes
    DatabricksLogging
  117. def recordFrameProfile[T](group: String, name: String)(thunk: => T): T
    Attributes
    protected
    Definition Classes
    DeltaLogging
  118. def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = null, silent: Boolean = true)(thunk: => S): S
    Definition Classes
    DatabricksLogging
  119. def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
    Definition Classes
    DatabricksLogging
  120. def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
    Definition Classes
    DatabricksLogging
  121. def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
    Definition Classes
    DatabricksLogging
  122. lazy val references: AttributeSet
    Definition Classes
    QueryPlan
    Annotations
    @transient()
  123. def refresh(): Unit
    Definition Classes
    LogicalPlan
  124. def removeFilesFromPaths(deltaLog: DeltaLog, nameToAddFileMap: Map[String, AddFile], filesToRewrite: Seq[String], operationTimestamp: Long): Seq[RemoveFile]

    This method provides the RemoveFile actions that are necessary for files that are touched and need to be rewritten in methods like Delete, Update, and Merge.

    This method provides the RemoveFile actions that are necessary for files that are touched and need to be rewritten in methods like Delete, Update, and Merge.

    deltaLog

    The DeltaLog of the table that is being operated on

    nameToAddFileMap

    A map generated using generateCandidateFileMap.

    filesToRewrite

    Absolute paths of the files that were touched. We will search for these in candidateFiles. Obtained as the output of the input_file_name function.

    operationTimestamp

    The timestamp of the operation

    Attributes
    protected
    Definition Classes
    DeltaCommand
  125. def resolve(nameParts: Seq[String], resolver: Resolver): Option[NamedExpression]
    Definition Classes
    LogicalPlan
  126. def resolve(schema: StructType, resolver: Resolver): Seq[Attribute]
    Definition Classes
    LogicalPlan
  127. def resolveChildren(nameParts: Seq[String], resolver: Resolver): Option[NamedExpression]
    Definition Classes
    LogicalPlan
  128. def resolveExpressions(r: PartialFunction[Expression, Expression]): LogicalPlan
    Definition Classes
    AnalysisHelper
  129. def resolveExpressionsWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): LogicalPlan
    Definition Classes
    AnalysisHelper
  130. def resolveIdentifier(analyzer: Analyzer, identifier: TableIdentifier): LogicalPlan

    Use the analyzer to resolve the identifier provided

    Use the analyzer to resolve the identifier provided

    analyzer

    The session state analyzer to call

    identifier

    Table Identifier to determine whether is path based or not

    Attributes
    protected
    Definition Classes
    DeltaCommand
  131. def resolveOperators(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    AnalysisHelper
  132. def resolveOperatorsDown(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    AnalysisHelper
  133. def resolveOperatorsDownWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    AnalysisHelper
  134. def resolveOperatorsUp(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    AnalysisHelper
  135. def resolveOperatorsUpWithNewOutput(rule: PartialFunction[LogicalPlan, (LogicalPlan, Seq[(Attribute, Attribute)])]): LogicalPlan
    Definition Classes
    AnalysisHelper
  136. def resolveOperatorsUpWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    AnalysisHelper
  137. def resolveOperatorsWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    AnalysisHelper
  138. def resolveQuoted(name: String, resolver: Resolver): Option[NamedExpression]
    Definition Classes
    LogicalPlan
  139. lazy val resolved: Boolean
    Definition Classes
    LogicalPlan
  140. def rewriteAttrs(attrMap: AttributeMap[Attribute]): LogicalPlan
    Definition Classes
    QueryPlan
  141. def run(spark: SparkSession): Seq[Row]
    Definition Classes
    ConvertToDeltaCommandBase → RunnableCommand
  142. def sameOutput(other: LogicalPlan): Boolean
    Definition Classes
    LogicalPlan
  143. final def sameResult(other: LogicalPlan): Boolean
    Definition Classes
    QueryPlan
  144. lazy val schema: StructType
    Definition Classes
    QueryPlan
  145. def schemaString: String
    Definition Classes
    QueryPlan
  146. final def semanticHash(): Int
    Definition Classes
    QueryPlan
  147. def setTagValue[T](tag: TreeNodeTag[T], value: T): Unit
    Definition Classes
    TreeNode
  148. def simpleString(maxFields: Int): String
    Definition Classes
    QueryPlan → TreeNode
  149. def simpleStringWithNodeId(): String
    Definition Classes
    QueryPlan → TreeNode
  150. def statePrefix: String
    Attributes
    protected
    Definition Classes
    LogicalPlan → QueryPlan
  151. def stats: Statistics
    Definition Classes
    Command → LogicalPlanStats
  152. val statsCache: Option[Statistics]
    Attributes
    protected
    Definition Classes
    LogicalPlanStats
  153. def stringArgs: Iterator[Any]
    Attributes
    protected
    Definition Classes
    TreeNode
  154. def subqueries: Seq[LogicalPlan]
    Definition Classes
    QueryPlan
  155. def subqueriesAll: Seq[LogicalPlan]
    Definition Classes
    QueryPlan
  156. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  157. def toJSON: String
    Definition Classes
    TreeNode
  158. def toString(): String
    Definition Classes
    TreeNode → AnyRef → Any
  159. def transform(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    TreeNode
  160. def transformAllExpressions(rule: PartialFunction[Expression, Expression]): ConvertToDeltaCommandBase.this.type
    Definition Classes
    QueryPlan
  161. def transformAllExpressionsWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): ConvertToDeltaCommandBase.this.type
    Definition Classes
    AnalysisHelper → QueryPlan
  162. def transformDown(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    TreeNode
  163. def transformDownWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    AnalysisHelper → TreeNode
  164. def transformExpressions(rule: PartialFunction[Expression, Expression]): ConvertToDeltaCommandBase.this.type
    Definition Classes
    QueryPlan
  165. def transformExpressionsDown(rule: PartialFunction[Expression, Expression]): ConvertToDeltaCommandBase.this.type
    Definition Classes
    QueryPlan
  166. def transformExpressionsDownWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): ConvertToDeltaCommandBase.this.type
    Definition Classes
    QueryPlan
  167. def transformExpressionsUp(rule: PartialFunction[Expression, Expression]): ConvertToDeltaCommandBase.this.type
    Definition Classes
    QueryPlan
  168. def transformExpressionsUpWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): ConvertToDeltaCommandBase.this.type
    Definition Classes
    QueryPlan
  169. def transformExpressionsWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): ConvertToDeltaCommandBase.this.type
    Definition Classes
    QueryPlan
  170. def transformUp(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    TreeNode
  171. def transformUpWithBeforeAndAfterRuleOnChildren(cond: (LogicalPlan) => Boolean, ruleId: RuleId)(rule: PartialFunction[(LogicalPlan, LogicalPlan), LogicalPlan]): LogicalPlan
    Definition Classes
    TreeNode
  172. def transformUpWithNewOutput(rule: PartialFunction[LogicalPlan, (LogicalPlan, Seq[(Attribute, Attribute)])], skipCond: (LogicalPlan) => Boolean, canGetOutput: (LogicalPlan) => Boolean): LogicalPlan
    Definition Classes
    AnalysisHelper → QueryPlan
  173. def transformUpWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    AnalysisHelper → TreeNode
  174. def transformUpWithSubqueries(f: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    QueryPlan
  175. def transformWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    TreeNode
  176. lazy val treePatternBits: BitSet
    Definition Classes
    QueryPlan → TreeNode → TreePatternBits
  177. def treeString(append: (String) => Unit, verbose: Boolean, addSuffix: Boolean, maxFields: Int, printOperatorId: Boolean): Unit
    Definition Classes
    TreeNode
  178. final def treeString(verbose: Boolean, addSuffix: Boolean, maxFields: Int, printOperatorId: Boolean): String
    Definition Classes
    TreeNode
  179. final def treeString: String
    Definition Classes
    TreeNode
  180. def unsetTagValue[T](tag: TreeNodeTag[T]): Unit
    Definition Classes
    TreeNode
  181. def updateAndCheckpoint(spark: SparkSession, deltaLog: DeltaLog, commitSize: Int, attemptVersion: Long, txnId: String): Snapshot

    Update the table now that the commit has been made, and write a checkpoint.

    Update the table now that the commit has been made, and write a checkpoint.

    Attributes
    protected
    Definition Classes
    DeltaCommand
  182. def updateOuterReferencesInSubquery(plan: LogicalPlan, attrMap: AttributeMap[Attribute]): LogicalPlan
    Definition Classes
    AnalysisHelper → QueryPlan
  183. lazy val validConstraints: ExpressionSet
    Attributes
    protected
    Definition Classes
    QueryPlanConstraints
  184. def verboseString(maxFields: Int): String
    Definition Classes
    QueryPlan → TreeNode
  185. def verboseStringWithOperatorId(): String
    Definition Classes
    QueryPlan
  186. def verboseStringWithSuffix(maxFields: Int): String
    Definition Classes
    LogicalPlan → TreeNode
  187. def verifyPartitionPredicates(spark: SparkSession, partitionColumns: Seq[String], predicates: Seq[Expression]): Unit
    Definition Classes
    DeltaCommand
  188. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  189. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  190. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  191. def withDmqTag[T](thunk: => T): T
    Attributes
    protected
    Definition Classes
    DeltaLogging
  192. final def withNewChildren(newChildren: Seq[LogicalPlan]): LogicalPlan
    Definition Classes
    TreeNode
  193. final def withNewChildrenInternal(newChildren: IndexedSeq[LogicalPlan]): LogicalPlan
    Definition Classes
    LeafLike
  194. def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: => T): T

    Report a log to indicate some command is running.

    Report a log to indicate some command is running.

    Definition Classes
    DeltaProgressReporter

Inherited from DeltaCommand

Inherited from DeltaLogging

Inherited from DatabricksLogging

Inherited from DeltaProgressReporter

Inherited from LeafRunnableCommand

Inherited from LeafLike[LogicalPlan]

Inherited from RunnableCommand

Inherited from Command

Inherited from LogicalPlan

Inherited from Logging

Inherited from QueryPlanConstraints

Inherited from ConstraintHelper

Inherited from LogicalPlanStats

Inherited from AnalysisHelper

Inherited from QueryPlan[LogicalPlan]

Inherited from SQLConfHelper

Inherited from TreeNode[LogicalPlan]

Inherited from TreePatternBits

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped