QuantileDiscretizer (Spark 2.4.0 JavaDoc)

Object
- org.apache.spark.ml.PipelineStage
- - org.apache.spark.ml.Estimator<Bucketizer>
  - - org.apache.spark.ml.feature.QuantileDiscretizer

All Implemented Interfaces:

java.io.Serializable, Logging, Params, HasInputCols, HasOutputCols, DefaultParamsWritable, Identifiable, MLWritable
```
public final class QuantileDiscretizer
extends Estimator<Bucketizer>
implements DefaultParamsWritable, HasInputCols, HasOutputCols
```
QuantileDiscretizer takes a column with continuous features and outputs a column with binned categorical features. The number of bins can be set using the numBuckets parameter. It is possible that the number of buckets used will be smaller than this value, for example, if there are too few distinct values of the input to create enough distinct quantiles. Since 2.3.0, QuantileDiscretizer can map multiple columns at once by setting the inputCols parameter. If both of the inputCol and inputCols parameters are set, an Exception will be thrown. To specify the number of buckets for each column, the numBucketsArray parameter can be set, or if the number of buckets should be the same across columns, numBuckets can be set as a convenience.
NaN handling: null and NaN values will be ignored from the column during QuantileDiscretizer fitting. This will produce a Bucketizer model for making predictions. During the transformation, Bucketizer will raise an error when it finds NaN values in the dataset, but the user can also choose to either keep or remove NaN values within the dataset by setting handleInvalid. If the user chooses to keep NaN values, they will be handled specially and placed into their own bucket, for example, if 4 buckets are used, then non-NaN data will be put into buckets[0-3], but NaNs will be counted in a special bucket[4].
Algorithm: The bin ranges are chosen using an approximate algorithm (see the documentation for org.apache.spark.sql.DataFrameStatFunctions.approxQuantile for a detailed description). The precision of the approximation can be controlled with the relativeError parameter. The lower and upper bin bounds will be -Infinity and +Infinity, covering all real values.

See Also:

Serialized Form

Constructor Summary

Constructors
Constructor and Description

QuantileDiscretizer()

QuantileDiscretizer(String uid)

Constructors
Constructor and Description
`QuantileDiscretizer()`
`QuantileDiscretizer(String uid)`

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`static Params`	`clear(Param<?> param)`
`QuantileDiscretizer`	`copy(ParamMap extra)` Creates a copy of this instance with the same UID and some extra params.
`static String`	`explainParam(Param<?> param)`
`static String`	`explainParams()`
`static ParamMap`	`extractParamMap()`
`static ParamMap`	`extractParamMap(ParamMap extra)`
`Bucketizer`	`fit(Dataset<?> dataset)` Fits a model to the input data.
`static <T> scala.Option<T>`	`get(Param<T> param)`
`static <T> scala.Option<T>`	`getDefault(Param<T> param)`
`static String`	`getHandleInvalid()`
`static String`	`getInputCol()`
`static String[]`	`getInputCols()`
`static int`	`getNumBuckets()`
`int`	`getNumBuckets()`
`static int[]`	`getNumBucketsArray()`
`int[]`	`getNumBucketsArray()`
`static <T> T`	`getOrDefault(Param<T> param)`
`static String`	`getOutputCol()`
`static String[]`	`getOutputCols()`
`static Param<Object>`	`getParam(String paramName)`
`static double`	`getRelativeError()`
`double`	`getRelativeError()`
`static Param<String>`	`handleInvalid()`
`Param<String>`	`handleInvalid()` Param for how to handle invalid entries.
`static <T> boolean`	`hasDefault(Param<T> param)`
`static boolean`	`hasParam(String paramName)`
`static Param<String>`	`inputCol()`
`static StringArrayParam`	`inputCols()`
`static boolean`	`isDefined(Param<?> param)`
`static boolean`	`isSet(Param<?> param)`
`static QuantileDiscretizer`	`load(String path)`
`static IntParam`	`numBuckets()`
`IntParam`	`numBuckets()` Number of buckets (quantiles, or categories) into which data points are grouped.
`static IntArrayParam`	`numBucketsArray()`
`IntArrayParam`	`numBucketsArray()` Array of number of buckets (quantiles, or categories) into which data points are grouped.
`static Param<String>`	`outputCol()`
`static StringArrayParam`	`outputCols()`
`static Param<?>[]`	`params()`
`static DoubleParam`	`relativeError()`
`DoubleParam`	`relativeError()` Relative error (see documentation for `org.apache.spark.sql.DataFrameStatFunctions.approxQuantile` for description) Must be in the range [0, 1].
`static void`	`save(String path)`
`static <T> Params`	`set(Param<T> param, T value)`
`QuantileDiscretizer`	`setHandleInvalid(String value)`
`QuantileDiscretizer`	`setInputCol(String value)`
`QuantileDiscretizer`	`setInputCols(String[] value)`
`QuantileDiscretizer`	`setNumBuckets(int value)`
`QuantileDiscretizer`	`setNumBucketsArray(int[] value)`
`QuantileDiscretizer`	`setOutputCol(String value)`
`QuantileDiscretizer`	`setOutputCols(String[] value)`
`QuantileDiscretizer`	`setRelativeError(double value)`
`static String`	`toString()`
`StructType`	`transformSchema(StructType schema)` :: DeveloperApi ::
`String`	`uid()` An immutable unique ID for the object and its derivatives.
`static MLWriter`	`write()`

Methods inherited from class org.apache.spark.ml.Estimator
fit, fit, fit, fit

Methods inherited from class Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.param.shared.HasHandleInvalid
getHandleInvalid

Methods inherited from interface org.apache.spark.ml.param.shared.HasInputCol
getInputCol, inputCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasOutputCol
getOutputCol, outputCol

Methods inherited from interface org.apache.spark.ml.param.Params
clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn

Methods inherited from interface org.apache.spark.ml.util.Identifiable
toString

Methods inherited from interface org.apache.spark.ml.util.DefaultParamsWritable
write

Methods inherited from interface org.apache.spark.ml.util.MLWritable
save

Methods inherited from interface org.apache.spark.ml.param.shared.HasInputCols
getInputCols, inputCols

Methods inherited from interface org.apache.spark.ml.param.shared.HasOutputCols
getOutputCols, outputCols

Methods inherited from interface org.apache.spark.internal.Logging
initializeLogging, initializeLogIfNecessary, initializeLogIfNecessary, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

- Constructor Detail
  - QuantileDiscretizer
```
public QuantileDiscretizer(String uid)
```
  - QuantileDiscretizer
```
public QuantileDiscretizer()
```
- Method Detail
  - load
```
public static QuantileDiscretizer load(String path)
```
  - toString
```
public static String toString()
```
  - params
```
public static Param<?>[] params()
```
  - explainParam
```
public static String explainParam(Param<?> param)
```
  - explainParams
```
public static String explainParams()
```
  - isSet
```
public static final boolean isSet(Param<?> param)
```
  - isDefined
```
public static final boolean isDefined(Param<?> param)
```
  - hasParam
```
public static boolean hasParam(String paramName)
```
  - getParam
```
public static Param<Object> getParam(String paramName)
```
  - set
```
public static final <T> Params set(Param<T> param,
                                   T value)
```
  - get
```
public static final <T> scala.Option<T> get(Param<T> param)
```
  - clear
```
public static final Params clear(Param<?> param)
```
  - getOrDefault
```
public static final <T> T getOrDefault(Param<T> param)
```
  - getDefault
```
public static final <T> scala.Option<T> getDefault(Param<T> param)
```
  - hasDefault
```
public static final <T> boolean hasDefault(Param<T> param)
```
  - extractParamMap
```
public static final ParamMap extractParamMap(ParamMap extra)
```
  - extractParamMap
```
public static final ParamMap extractParamMap()
```
  - getHandleInvalid
```
public static final String getHandleInvalid()
```
  - inputCol
```
public static final Param<String> inputCol()
```
  - getInputCol
```
public static final String getInputCol()
```
  - outputCol
```
public static final Param<String> outputCol()
```
  - getOutputCol
```
public static final String getOutputCol()
```
  - numBuckets
```
public static IntParam numBuckets()
```
  - getNumBuckets
```
public static int getNumBuckets()
```
  - numBucketsArray
```
public static IntArrayParam numBucketsArray()
```
  - getNumBucketsArray
```
public static int[] getNumBucketsArray()
```
  - relativeError
```
public static DoubleParam relativeError()
```
  - getRelativeError
```
public static double getRelativeError()
```
  - handleInvalid
```
public static Param<String> handleInvalid()
```
  - save
```
public static void save(String path)
                 throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - write
```
public static MLWriter write()
```
  - inputCols
```
public static final StringArrayParam inputCols()
```
  - getInputCols
```
public static final String[] getInputCols()
```
  - outputCols
```
public static final StringArrayParam outputCols()
```
  - getOutputCols
```
public static final String[] getOutputCols()
```
  - uid
```
public String uid()
```
    Description copied from interface: Identifiable
    
    An immutable unique ID for the object and its derivatives.
    
    Specified by:
    
    uid in interface Identifiable
    
    Returns:
    
    (undocumented)
  - setRelativeError
```
public QuantileDiscretizer setRelativeError(double value)
```
  - setNumBuckets
```
public QuantileDiscretizer setNumBuckets(int value)
```
  - setInputCol
```
public QuantileDiscretizer setInputCol(String value)
```
  - setOutputCol
```
public QuantileDiscretizer setOutputCol(String value)
```
  - setHandleInvalid
```
public QuantileDiscretizer setHandleInvalid(String value)
```
  - setNumBucketsArray
```
public QuantileDiscretizer setNumBucketsArray(int[] value)
```
  - setInputCols
```
public QuantileDiscretizer setInputCols(String[] value)
```
  - setOutputCols
```
public QuantileDiscretizer setOutputCols(String[] value)
```
  - transformSchema
```
public StructType transformSchema(StructType schema)
```
    Description copied from class: PipelineStage
    
    :: DeveloperApi ::
    Check transform validity and derive the output schema from the input schema.
    We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().
    Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
    
    Specified by:
    
    transformSchema in class PipelineStage
    
    Parameters:
    
    schema - (undocumented)
    
    Returns:
    
    (undocumented)
  - fit
```
public Bucketizer fit(Dataset<?> dataset)
```
    Description copied from class: Estimator
    
    Fits a model to the input data.
    
    Specified by:
    
    fit in class Estimator<Bucketizer>
    
    Parameters:
    
    dataset - (undocumented)
    
    Returns:
    
    (undocumented)
  - copy
```
public QuantileDiscretizer copy(ParamMap extra)
```
    Description copied from interface: Params
    
    Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().
    
    Specified by:
    
    copy in interface Params
    
    Specified by:
    
    copy in class Estimator<Bucketizer>
    
    Parameters:
    
    extra - (undocumented)
    
    Returns:
    
    (undocumented)
  - getNumBuckets
```
public int getNumBuckets()
```
  - getNumBucketsArray
```
public int[] getNumBucketsArray()
```
  - getRelativeError
```
public double getRelativeError()
```
  - handleInvalid
```
public Param<String> handleInvalid()
```
    Param for how to handle invalid entries. Options are 'skip' (filter out rows with invalid values), 'error' (throw an error), or 'keep' (keep invalid values in a special additional bucket). Note that in the multiple columns case, the invalid handling is applied to all columns. That said for 'error' it will throw an error if any invalids are found in any column, for 'skip' it will skip rows with any invalids in any columns, etc. Default: "error"
    
    Specified by:
    
    handleInvalid in interface HasHandleInvalid
    
    Returns:
    
    (undocumented)
  - numBuckets
```
public IntParam numBuckets()
```
    Number of buckets (quantiles, or categories) into which data points are grouped. Must be greater than or equal to 2.
    See also handleInvalid, which can optionally create an additional bucket for NaN values.
    default: 2
    
    Returns:
    
    (undocumented)
  - numBucketsArray
```
public IntArrayParam numBucketsArray()
```
    Array of number of buckets (quantiles, or categories) into which data points are grouped. Each value must be greater than or equal to 2
    See also handleInvalid, which can optionally create an additional bucket for NaN values.
    
    Returns:
    
    (undocumented)
  - relativeError
```
public DoubleParam relativeError()
```
    Relative error (see documentation for org.apache.spark.sql.DataFrameStatFunctions.approxQuantile for description) Must be in the range [0, 1]. Note that in multiple columns case, relative error is applied to all columns. default: 0.001
    
    Returns:
    
    (undocumented)

Class QuantileDiscretizer

Constructor Summary

Method Summary

Methods inherited from class org.apache.spark.ml.Estimator

Methods inherited from class Object

Methods inherited from interface org.apache.spark.ml.param.shared.HasHandleInvalid

Methods inherited from interface org.apache.spark.ml.param.shared.HasInputCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasOutputCol

Methods inherited from interface org.apache.spark.ml.param.Params

Methods inherited from interface org.apache.spark.ml.util.Identifiable

Methods inherited from interface org.apache.spark.ml.util.DefaultParamsWritable

Methods inherited from interface org.apache.spark.ml.util.MLWritable

Methods inherited from interface org.apache.spark.ml.param.shared.HasInputCols

Methods inherited from interface org.apache.spark.ml.param.shared.HasOutputCols

Methods inherited from interface org.apache.spark.internal.Logging

Constructor Detail

QuantileDiscretizer

QuantileDiscretizer

Method Detail

load

toString

params

explainParam

explainParams

isSet

isDefined

hasParam

getParam

set

get

clear

getOrDefault

getDefault

hasDefault

extractParamMap

extractParamMap

getHandleInvalid

inputCol

getInputCol

outputCol

getOutputCol

numBuckets

getNumBuckets

numBucketsArray

getNumBucketsArray

relativeError

getRelativeError

handleInvalid

save

write

inputCols

getInputCols

outputCols

getOutputCols

uid

setRelativeError

setNumBuckets

setInputCol

setOutputCol

setHandleInvalid

setNumBucketsArray

setInputCols

setOutputCols

transformSchema

fit

copy

getNumBuckets

getNumBucketsArray

getRelativeError

handleInvalid

numBuckets

numBucketsArray

relativeError