engine

Provides the tech.sourced.engine.Engine class, which is the main entry point of all the analysis you might do using this library as well as some implicits to make it easier to use. In particular, it adds some methods to be able to join with other "tables" directly from any org.apache.spark.sql.DataFrame.

import tech.sourced.engine._

val engine = Engine(sparkSession, "/path/to/repositories")

If you don't want to import everything in the engine, even though it only exposes what's truly needed to not pollute the user namespace, you can do it by just importing the tech.sourced.engine.Engine class and the tech.sourced.engine.EngineDataFrame implicit class.

import tech.sourced.engine.{Engine, EngineDataFrame}

val engine = Engine(sparkSession, "/path/to/repositories")

Linear Supertypes

AnyRef, Any

Type Members

class DefaultSource extends RelationProvider with DataSourceRegister

Default source to provide new git relations.
class Engine extends Logging

Engine is the main entry point to all usage of the source{d} spark-engine.
Engine is the main entry point to all usage of the source{d} spark-engine. It has methods to configure all possible configurable options as well as the available methods to start analysing repositories of code.
```
import tech.sourced.engine._

val engine = Engine(sparkSession, "/path/to/repositories")
```
NOTE: Keep in mind that you will need to register the UDFs in the session manually if you choose to instantiate this class directly instead of using the companion object.
```
import tech.sourced.engine.{Engine, SessionFunctions}

engine = new Engine(sparkSession)
sparkSession.registerUDFs()
```
The only method available as of now is getRepositories, which will generate a DataFrame of repositories, which is the very first thing you need to analyse repositories of code.
implicit class EngineDataFrame extends AnyRef

Adds some utility methods to the org.apache.spark.sql.DataFrame class so you can, for example, get the references, commits, etc from a data frame containing repositories.
case class GitRelation(session: SparkSession, schema: StructType, joinConditions: Option[Expression] = None, tableSource: Option[String] = None) extends BaseRelation with CatalystScan with Product with Serializable

A relation based on git data from rooted repositories in siva files.
A relation based on git data from rooted repositories in siva files. The data this relation will offer depends on the given tableSource, which controls the table that will be accessed. Also, the tech.sourced.engine.rule.GitOptimizer might merge some table sources into one by squashing joins, so the result will be the resultant table chained with the previous one using chained iterators.
session
Spark session
schema
schema of the relation
joinConditions
join conditions, if any
tableSource
source table if any
case class Join(left: String, right: String, conditions: Seq[JoinCondition]) extends Product with Serializable
case class JoinCondition(leftTable: String, leftCol: String, rightTable: String, rightCol: String) extends Product with Serializable
case class MetadataRelation(session: SparkSession, schema: StructType, dbPath: String, joinConditions: Option[Expression] = None, tableSource: Option[String] = None) extends BaseRelation with CatalystScan with Product with Serializable
class MetadataSource extends RelationProvider with DataSourceRegister

Data source to provide new metadata relations.
implicit class SessionFunctions extends AnyRef

Implicit class that adds some functions to the org.apache.spark.sql.SparkSession.

Value Members

val BlobsTable: String
val CommitsTable: String
object DefaultSource

Just contains some useful constants for the DefaultSource class to use.
val DefaultSourceName: String
object Engine

Factory for tech.sourced.engine.Engine instances.
object MetadataSource
val MetadataSourceName: String
val ReferencesTable: String
val RepositoriesTable: String
val RepositoryHasCommitsTable: String
object Sources

Defines the hierarchy between data sources.
object Tables
val TreeEntriesTable: String
package iterator
def parseUASTNode(data: Array[Byte]): Node

Create a new Node from a binary-encoded node as a byte array.
Create a new Node from a binary-encoded node as a byte array.
data
binary-encoded node as byte array
returns
parsed Node
package provider
package rule
package udf
package util

package engine

Type Members

class DefaultSource extends RelationProvider with DataSourceRegister

class Engine extends Logging

implicit class EngineDataFrame extends AnyRef

case class GitRelation(session: SparkSession, schema: StructType, joinConditions: Option[Expression] = None, tableSource: Option[String] = None) extends BaseRelation with CatalystScan with Product with Serializable

case class Join(left: String, right: String, conditions: Seq[JoinCondition]) extends Product with Serializable

case class JoinCondition(leftTable: String, leftCol: String, rightTable: String, rightCol: String) extends Product with Serializable

case class MetadataRelation(session: SparkSession, schema: StructType, dbPath: String, joinConditions: Option[Expression] = None, tableSource: Option[String] = None) extends BaseRelation with CatalystScan with Product with Serializable

class MetadataSource extends RelationProvider with DataSourceRegister

implicit class SessionFunctions extends AnyRef

Value Members

val BlobsTable: String

val CommitsTable: String

object DefaultSource

val DefaultSourceName: String

object Engine

object MetadataSource

val MetadataSourceName: String

val ReferencesTable: String

val RepositoriesTable: String

val RepositoryHasCommitsTable: String

object Sources

object Tables

val TreeEntriesTable: String

package iterator

def parseUASTNode(data: Array[Byte]): Node

package provider

package rule

package udf

package util

Inherited from AnyRef

Inherited from Any

Ungrouped