sql

Value Members

1. val ArrayType: sql.catalyst.types.ArrayType.type

   :: DeveloperApi ::

   :: DeveloperApi ::

   An ArrayType object can be constructed with two ways,

   ArrayType(elementType: DataType, containsNull: Boolean)

   and

   ArrayType(elementType: DataType)

   For ArrayType(elementType), the field of containsNull is set to false.

2. val BinaryType: sql.catalyst.types.BinaryType.type

   :: DeveloperApi ::

   :: DeveloperApi ::

   The data type representing Array[Byte] values.

3. val BooleanType: sql.catalyst.types.BooleanType.type

   :: DeveloperApi ::

   :: DeveloperApi ::

   The data type representing Boolean values.

4. val ByteType: sql.catalyst.types.ByteType.type

   :: DeveloperApi ::

   :: DeveloperApi ::

   The data type representing Byte values.

5. val DecimalType: sql.catalyst.types.DecimalType.type

   :: DeveloperApi ::

   :: DeveloperApi ::

   The data type representing scala.math.BigDecimal values.

6. val DoubleType: sql.catalyst.types.DoubleType.type

   :: DeveloperApi ::

   :: DeveloperApi ::

   The data type representing Double values.

7. val FloatType: sql.catalyst.types.FloatType.type

   :: DeveloperApi ::

   :: DeveloperApi ::

   The data type representing Float values.

8. val IntegerType: sql.catalyst.types.IntegerType.type

   :: DeveloperApi ::

   :: DeveloperApi ::

   The data type representing Int values.

9. val LongType: sql.catalyst.types.LongType.type

   :: DeveloperApi ::

   :: DeveloperApi ::

   The data type representing Long values.

10. val MapType: sql.catalyst.types.MapType.type

    :: DeveloperApi ::

    :: DeveloperApi ::

    A MapType object can be constructed with two ways,

    MapType(keyType: DataType, valueType: DataType, valueContainsNull: Boolean)

    and

    MapType(keyType: DataType, valueType: DataType)

    For MapType(keyType: DataType, valueType: DataType), the field of valueContainsNull is set to true.

11. val Row: sql.catalyst.expressions.Row.type

    :: DeveloperApi ::

    :: DeveloperApi ::

    A Row object can be constructed by providing field values. Example:

    import org.apache.spark.sql._
    
    // Create a Row from values.
    Row(value1, value2, value3, ...)
    // Create a Row from a Seq of values.
    Row.fromSeq(Seq(value1, value2, ...))

    A value of a row can be accessed through both generic access by ordinal, which will incur boxing overhead for primitives, as well as native primitive access. An example of generic access by ordinal:

    import org.apache.spark.sql._
    
    val row = Row(1, true, "a string", null)
    // row: Row = [1,true,a string,null]
    val firstValue = row(0)
    // firstValue: Any = 1
    val fourthValue = row(3)
    // fourthValue: Any = null

    For native primitive access, it is invalid to use the native primitive interface to retrieve a value that is null, instead a user must check isNullAt before attempting to retrieve a value that might be null. An example of native primitive access:

    // using the row from the previous example.
    val firstValue = row.getInt(0)
    // firstValue: Int = 1
    val isNull = row.isNullAt(3)
    // isNull: Boolean = true

    Interfaces related to native primitive access are:

    isNullAt(i: Int): Boolean

    getInt(i: Int): Int

    getLong(i: Int): Long

    getDouble(i: Int): Double

    getFloat(i: Int): Float

    getBoolean(i: Int): Boolean

    getShort(i: Int): Short

    getByte(i: Int): Byte

    getString(i: Int): String

    Fields in a Row object can be extracted in a pattern match. Example:

    import org.apache.spark.sql._
    
    val pairs = sql("SELECT key, value FROM src").rdd.map {
      case Row(key: Int, value: String) =>
        key -> value
    }

12. val ShortType: sql.catalyst.types.ShortType.type

    :: DeveloperApi ::

    :: DeveloperApi ::

    The data type representing Short values.

13. val StringType: sql.catalyst.types.StringType.type

    :: DeveloperApi ::

    :: DeveloperApi ::

    The data type representing String values

14. val StructField: sql.catalyst.types.StructField.type

    :: DeveloperApi ::

    :: DeveloperApi ::

    A StructField object can be constructed by

    StructField(name: String, dataType: DataType, nullable: Boolean)

15. val StructType: sql.catalyst.types.StructType.type

    :: DeveloperApi ::

    :: DeveloperApi ::

    A StructType object can be constructed by

    StructType(fields: Seq[StructField])

    For a StructType object, one or multiple StructFields can be extracted by names. If multiple StructFields are extracted, a StructType object will be returned. If a provided name does not have a matching field, it will be ignored. For the case of extracting a single StructField, a null will be returned. Example:

    import org.apache.spark.sql._
    
    val struct =
      StructType(
        StructField("a", IntegerType, true) ::
        StructField("b", LongType, false) ::
        StructField("c", BooleanType, false) :: Nil)
    
    // Extract a single StructField.
    val singleField = struct("b")
    // singleField: StructField = StructField(b,LongType,false)
    
    // This struct does not have a field called "d". null will be returned.
    val nonExisting = struct("d")
    // nonExisting: StructField = null
    
    // Extract multiple StructFields. Field names are provided in a set.
    // A StructType object will be returned.
    val twoFields = struct(Set("b", "c"))
    // twoFields: StructType =
    //   StructType(List(StructField(b,LongType,false), StructField(c,BooleanType,false)))
    
    // Those names do not have matching fields will be ignored.
    // For the case shown below, "d" will be ignored and
    // it is treated as struct(Set("b", "c")).
    val ignoreNonExisting = struct(Set("b", "c", "d"))
    // ignoreNonExisting: StructType =
    //   StructType(List(StructField(b,LongType,false), StructField(c,BooleanType,false)))

    A Row object is used as a value of the StructType. Example:

    import org.apache.spark.sql._
    
    val innerStruct =
      StructType(
        StructField("f1", IntegerType, true) ::
        StructField("f2", LongType, false) ::
        StructField("f3", BooleanType, false) :: Nil)
    
    val struct = StructType(
      StructField("a", innerStruct, true) :: Nil)
    
    // Create a Row with the schema defined by struct
    val row = Row(Row(1, 2, true))
    // row: Row = [[1,2,true]]

16. val TimestampType: sql.catalyst.types.TimestampType.type

    :: DeveloperApi ::

    :: DeveloperApi ::

    The data type representing java.sql.Timestamp values.

17. package api

18. package execution

    :: DeveloperApi :: An execution engine for relational query plans that runs on top Spark and returns RDDs.

19. package hive

20. package parquet

21. package test

22. package types