class ParquetTable extends ConvertTargetTable with DeltaLogging

Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ParquetTable
  2. DeltaLogging
  3. DatabricksLogging
  4. DeltaProgressReporter
  5. Logging
  6. ConvertTargetTable
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new ParquetTable(spark: SparkSession, basePath: String, partitionSchema: Option[StructType])

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. val fileManifest: ConvertTargetFileManifest

    The file manifest of the target table

    The file manifest of the target table

    Definition Classes
    ParquetTable → ConvertTargetTable
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  10. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def getSchemaForBatch(spark: SparkSession, batch: Seq[SerializableFileStatus], serializedConf: SerializableConfiguration): StructType
    Attributes
    protected
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  14. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  15. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  16. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  17. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  18. def logConsole(line: String): Unit
    Definition Classes
    DatabricksLogging
  19. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  20. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  21. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  22. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  23. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  24. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  25. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  26. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. def mergeSchemasInParallel(sparkSession: SparkSession, filesToTouch: Seq[FileStatus], serializedConf: SerializableConfiguration): Option[StructType]

    This method is forked from ParquetFileFormat.

    This method is forked from ParquetFileFormat. The only change here is that we use our SchemaMergingUtils.mergeSchemas() instead of StructType.merge(), where we allow upcast between ByteType, ShortType and IntegerType.

    Figures out a merged Parquet schema with a distributed Spark job.

    Note that locality is not taken into consideration here because:

    1. For a single Parquet part-file, in most cases the footer only resides in the last block of that file. Thus we only need to retrieve the location of the last block. However, Hadoop FileSystem only provides API to retrieve locations of all blocks, which can be potentially expensive.

    2. This optimization is mainly useful for S3, where file metadata operations can be pretty slow. And basically locality is not available when using S3 (you can't run computation on S3 nodes).

    Attributes
    protected
  31. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  32. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  33. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  34. def numFiles: Long

    The number of files from the target table

    The number of files from the target table

    Definition Classes
    ParquetTable → ConvertTargetTable
  35. val partitionSchema: Option[StructType]

    The partition schema of the target table, if known

    The partition schema of the target table, if known

    Definition Classes
    ParquetTable → ConvertTargetTable
  36. def properties: Map[String, String]

    The table properties of the target table

    The table properties of the target table

    Definition Classes
    ConvertTargetTable
  37. def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null, path: Option[Path] = None): Unit

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    path

    Used to log the path of the delta table when deltaLog is null.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  38. def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  39. def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  40. def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
    Definition Classes
    DatabricksLogging
  41. def recordFrameProfile[T](group: String, name: String)(thunk: => T): T
    Attributes
    protected
    Definition Classes
    DeltaLogging
  42. def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = null, silent: Boolean = true)(thunk: => S): S
    Definition Classes
    DatabricksLogging
  43. def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
    Definition Classes
    DatabricksLogging
  44. def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
    Definition Classes
    DatabricksLogging
  45. def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
    Definition Classes
    DatabricksLogging
  46. def requiredColumnMappingMode: DeltaColumnMappingMode

    Whether this table requires column mapping to be converted

    Whether this table requires column mapping to be converted

    Definition Classes
    ConvertTargetTable
  47. lazy val serializableConf: SerializableConfiguration
    Attributes
    protected
  48. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  49. def tableSchema: StructType

    The table schema of the target table

    The table schema of the target table

    Definition Classes
    ParquetTable → ConvertTargetTable
  50. def toString(): String
    Definition Classes
    AnyRef → Any
  51. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  52. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  53. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  54. def withDmqTag[T](thunk: => T): T
    Attributes
    protected
    Definition Classes
    DeltaLogging
  55. def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: => T): T

    Report a log to indicate some command is running.

    Report a log to indicate some command is running.

    Definition Classes
    DeltaProgressReporter

Inherited from DeltaLogging

Inherited from DatabricksLogging

Inherited from DeltaProgressReporter

Inherited from Logging

Inherited from ConvertTargetTable

Inherited from AnyRef

Inherited from Any

Ungrouped