org.apache.spark.ml.clustering

Class BisectingKMeans

Object

org.apache.spark.ml.PipelineStage

org.apache.spark.ml.Estimator<BisectingKMeansModel>

org.apache.spark.ml.clustering.BisectingKMeans

All Implemented Interfaces:

java.io.Serializable, Logging, Params, DefaultParamsWritable, Identifiable, MLWritable

public class BisectingKMeans
extends Estimator<BisectingKMeansModel>
implements DefaultParamsWritable

A bisecting k-means algorithm based on the paper "A comparison of document clustering techniques" by Steinbach, Karypis, and Kumar, with modification to fit Spark. The algorithm starts from a single cluster that contains all points. Iteratively it finds divisible clusters on the bottom level and bisects each of them using k-means, until there are k leaf clusters in total or no leaf clusters are divisible. The bisecting steps of clusters on the same level are grouped together to increase parallelism. If bisecting all divisible clusters on the bottom level would result more than k leaf clusters, larger clusters get higher priority.

See Also:

Steinbach, Karypis, and Kumar, A comparison of document clustering techniques, KDD Workshop on Text Mining, 2000., Serialized Form

Constructor Summary

Constructors

Constructor and Description
BisectingKMeans()
BisectingKMeans(String uid)

Method Summary

Modifier and Type	Method and Description
static Params	clear(Param<?> param)
BisectingKMeans	copy(ParamMap extra) Creates a copy of this instance with the same UID and some extra params.
static Param<String>	distanceMeasure()
static String	explainParam(Param<?> param)
static String	explainParams()
static ParamMap	extractParamMap()
static ParamMap	extractParamMap(ParamMap extra)
static Param<String>	featuresCol()
BisectingKMeansModel	fit(Dataset<?> dataset) Fits a model to the input data.
static <T> scala.Option<T>	get(Param<T> param)
static <T> scala.Option<T>	getDefault(Param<T> param)
static String	getDistanceMeasure()
static String	getFeaturesCol()
static int	getK()
int	getK()
static int	getMaxIter()
static double	getMinDivisibleClusterSize()
double	getMinDivisibleClusterSize()
static <T> T	getOrDefault(Param<T> param)
static Param<Object>	getParam(String paramName)
static String	getPredictionCol()
static long	getSeed()
static <T> boolean	hasDefault(Param<T> param)
static boolean	hasParam(String paramName)
static boolean	isDefined(Param<?> param)
static boolean	isSet(Param<?> param)
static IntParam	k()
IntParam	k() The desired number of leaf clusters.
static BisectingKMeans	load(String path)
static IntParam	maxIter()
static DoubleParam	minDivisibleClusterSize()
DoubleParam	minDivisibleClusterSize() The minimum number of points (if greater than or equal to 1.0) or the minimum proportion of points (if less than 1.0) of a divisible cluster (default: 1.0).
static Param<?>[]	params()
static Param<String>	predictionCol()
static void	save(String path)
static LongParam	seed()
static <T> Params	set(Param<T> param, T value)
BisectingKMeans	setDistanceMeasure(String value)
BisectingKMeans	setFeaturesCol(String value)
BisectingKMeans	setK(int value)
BisectingKMeans	setMaxIter(int value)
BisectingKMeans	setMinDivisibleClusterSize(double value)
BisectingKMeans	setPredictionCol(String value)
BisectingKMeans	setSeed(long value)
static String	toString()
StructType	transformSchema(StructType schema) :: DeveloperApi ::
String	uid() An immutable unique ID for the object and its derivatives.
StructType	validateAndTransformSchema(StructType schema) Validates and transforms the input schema.
static MLWriter	write()

Methods inherited from class org.apache.spark.ml.Estimator

fit, fit, fit, fit

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.param.shared.HasMaxIter

getMaxIter, maxIter

Methods inherited from interface org.apache.spark.ml.param.shared.HasFeaturesCol

featuresCol, getFeaturesCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasSeed

getSeed, seed

Methods inherited from interface org.apache.spark.ml.param.shared.HasPredictionCol

getPredictionCol, predictionCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasDistanceMeasure

distanceMeasure, getDistanceMeasure

Methods inherited from interface org.apache.spark.ml.param.Params

clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn

Methods inherited from interface org.apache.spark.ml.util.Identifiable

toString

Methods inherited from interface org.apache.spark.ml.util.DefaultParamsWritable

write

Methods inherited from interface org.apache.spark.ml.util.MLWritable

save

Methods inherited from interface org.apache.spark.internal.Logging

initializeLogging, initializeLogIfNecessary, initializeLogIfNecessary, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

BisectingKMeans

public BisectingKMeans(String uid)

BisectingKMeans

public BisectingKMeans()

Method Detail

load

public static BisectingKMeans load(String path)

toString

public static String toString()

params

public static Param<?>[] params()

explainParam

public static String explainParam(Param<?> param)

explainParams

public static String explainParams()

isSet

public static final boolean isSet(Param<?> param)

isDefined

public static final boolean isDefined(Param<?> param)

hasParam

public static boolean hasParam(String paramName)

getParam

public static Param<Object> getParam(String paramName)

set

public static final <T> Params set(Param<T> param,
                                   T value)

get

public static final <T> scala.Option<T> get(Param<T> param)

clear

public static final Params clear(Param<?> param)

getOrDefault

public static final <T> T getOrDefault(Param<T> param)

getDefault

public static final <T> scala.Option<T> getDefault(Param<T> param)

hasDefault

public static final <T> boolean hasDefault(Param<T> param)

extractParamMap

public static final ParamMap extractParamMap(ParamMap extra)

extractParamMap

public static final ParamMap extractParamMap()

maxIter

public static final IntParam maxIter()

getMaxIter

public static final int getMaxIter()

featuresCol

public static final Param<String> featuresCol()

getFeaturesCol

public static final String getFeaturesCol()

seed

public static final LongParam seed()

getSeed

public static final long getSeed()

predictionCol

public static final Param<String> predictionCol()

getPredictionCol

public static final String getPredictionCol()

distanceMeasure

public static final Param<String> distanceMeasure()

getDistanceMeasure

public static final String getDistanceMeasure()

k

public static final IntParam k()

getK

public static int getK()

minDivisibleClusterSize

public static final DoubleParam minDivisibleClusterSize()

getMinDivisibleClusterSize

public static double getMinDivisibleClusterSize()

save

public static void save(String path)
                 throws java.io.IOException

Throws:

java.io.IOException

write

public static MLWriter write()

uid

public String uid()

Description copied from interface: Identifiable

An immutable unique ID for the object and its derivatives.

Specified by:

uid in interface Identifiable

Returns:

(undocumented)

copy

public BisectingKMeans copy(ParamMap extra)

Description copied from interface: Params

Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().

Specified by:

copy in interface Params

Specified by:

copy in class Estimator<BisectingKMeansModel>

Parameters:

extra - (undocumented)

Returns:

(undocumented)

setFeaturesCol

public BisectingKMeans setFeaturesCol(String value)

setPredictionCol

public BisectingKMeans setPredictionCol(String value)

setK

public BisectingKMeans setK(int value)

setMaxIter

public BisectingKMeans setMaxIter(int value)

setSeed

public BisectingKMeans setSeed(long value)

setMinDivisibleClusterSize

public BisectingKMeans setMinDivisibleClusterSize(double value)

setDistanceMeasure

public BisectingKMeans setDistanceMeasure(String value)

fit

public BisectingKMeansModel fit(Dataset<?> dataset)

Description copied from class: Estimator

Fits a model to the input data.

Specified by:

fit in class Estimator<BisectingKMeansModel>

Parameters:

dataset - (undocumented)

Returns:

(undocumented)

transformSchema

public StructType transformSchema(StructType schema)

Description copied from class: PipelineStage

:: DeveloperApi ::

Check transform validity and derive the output schema from the input schema.

We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().

Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.

Specified by:

transformSchema in class PipelineStage

Parameters:

schema - (undocumented)

Returns:

(undocumented)

getK

public int getK()

getMinDivisibleClusterSize

public double getMinDivisibleClusterSize()

k

public IntParam k()

The desired number of leaf clusters. Must be &gt; 1. Default: 4. The actual number could be smaller if there are no divisible leaf clusters.

Returns:

(undocumented)

minDivisibleClusterSize

public DoubleParam minDivisibleClusterSize()

The minimum number of points (if greater than or equal to 1.0) or the minimum proportion of points (if less than 1.0) of a divisible cluster (default: 1.0).

Returns:

(undocumented)

validateAndTransformSchema

public StructType validateAndTransformSchema(StructType schema)

Validates and transforms the input schema.

Parameters:

schema - input schema

Returns:

output schema