org.apache.spark.ml.util

Class SchemaUtils

Object

org.apache.spark.ml.util.SchemaUtils

public class SchemaUtils
extends Object

Utils for handling schemas.

Constructor Summary

Constructors

Constructor and Description
SchemaUtils()

Method Summary

Modifier and Type	Method and Description
static StructType	appendColumn(StructType schema, String colName, DataType dataType, boolean nullable) Appends a new column to the input schema.
static StructType	appendColumn(StructType schema, StructField col) Appends a new column to the input schema.
static void	checkColumnType(StructType schema, String colName, DataType dataType, String msg) Check whether the given schema contains a column of the required data type.
static void	checkColumnTypes(StructType schema, String colName, scala.collection.Seq<DataType> dataTypes, String msg) Check whether the given schema contains a column of one of the require data types.
static void	checkNumericType(StructType schema, String colName, String msg) Check whether the given schema contains a column of the numeric data type.
static void	validateVectorCompatibleColumn(StructType schema, String colName) Check whether the given column in the schema is one of the supporting vector type: Vector, Array[Float].

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SchemaUtils

public SchemaUtils()

Method Detail

checkColumnType

public static void checkColumnType(StructType schema,
                                   String colName,
                                   DataType dataType,
                                   String msg)

Check whether the given schema contains a column of the required data type.

Parameters:

colName - column name

dataType - required column data type

schema - (undocumented)

msg - (undocumented)

checkColumnTypes

public static void checkColumnTypes(StructType schema,
                                    String colName,
                                    scala.collection.Seq<DataType> dataTypes,
                                    String msg)

Check whether the given schema contains a column of one of the require data types.

Parameters:

colName - column name

dataTypes - required column data types

schema - (undocumented)

msg - (undocumented)

checkNumericType

public static void checkNumericType(StructType schema,
                                    String colName,
                                    String msg)

Check whether the given schema contains a column of the numeric data type.

Parameters:

colName - column name

schema - (undocumented)

msg - (undocumented)

appendColumn

public static StructType appendColumn(StructType schema,
                                      String colName,
                                      DataType dataType,
                                      boolean nullable)

Appends a new column to the input schema. This fails if the given output column already exists.

Parameters:

schema - input schema

colName - new column name. If this column name is an empty string "", this method returns the input schema unchanged. This allows users to disable output columns.

dataType - new column data type

nullable - (undocumented)

Returns:

new schema with the input column appended

appendColumn

public static StructType appendColumn(StructType schema,
                                      StructField col)

Appends a new column to the input schema. This fails if the given output column already exists.

Parameters:

schema - input schema

col - New column schema

Returns:

new schema with the input column appended

validateVectorCompatibleColumn

public static void validateVectorCompatibleColumn(StructType schema,
                                                  String colName)

Check whether the given column in the schema is one of the supporting vector type: Vector, Array[Float]. Array[Double]

Parameters:

schema - input schema

colName - column name