public abstract class LDAModel extends Model<LDAModel> implements Logging, MLWritable
LDA.
param: vocabSize Vocabulary size (number of terms or words in the vocabulary) param: sparkSession Used to construct local DataFrames for returning query results
| Modifier and Type | Method and Description |
|---|---|
Dataset<Row> |
describeTopics() |
Dataset<Row> |
describeTopics(int maxTermsPerTopic)
Return the topics described by their top-weighted terms.
|
DoubleArrayParam |
docConcentration()
Concentration parameter (commonly named "alpha") for the prior placed on documents'
distributions over topics ("theta").
|
Vector |
estimatedDocConcentration()
Value for
docConcentration estimated from data. |
double[] |
getDocConcentration() |
int |
getK() |
boolean |
getKeepLastCheckpoint() |
double |
getLearningDecay() |
double |
getLearningOffset() |
Vector |
getOldDocConcentration()
Get docConcentration used by spark.mllib LDA
|
LDAOptimizer |
getOldOptimizer() |
double |
getOldTopicConcentration()
Get topicConcentration used by spark.mllib LDA
|
boolean |
getOptimizeDocConcentration() |
String |
getOptimizer() |
double |
getSubsamplingRate() |
double |
getTopicConcentration() |
String |
getTopicDistributionCol() |
abstract boolean |
isDistributed()
Indicates whether this instance is of type
DistributedLDAModel |
IntParam |
k()
Param for the number of topics (clusters) to infer.
|
BooleanParam |
keepLastCheckpoint()
For EM optimizer only:
optimizer = "em". |
DoubleParam |
learningDecay()
For Online optimizer only:
optimizer = "online". |
DoubleParam |
learningOffset()
For Online optimizer only:
optimizer = "online". |
double |
logLikelihood(Dataset<?> dataset)
Calculates a lower bound on the log likelihood of the entire corpus.
|
double |
logPerplexity(Dataset<?> dataset)
Calculate an upper bound on perplexity.
|
BooleanParam |
optimizeDocConcentration()
For Online optimizer only (currently):
optimizer = "online". |
Param<String> |
optimizer()
Optimizer or inference algorithm used to estimate the LDA model.
|
LDAModel |
setFeaturesCol(String value)
The features for LDA should be a
Vector representing the word counts in a document. |
LDAModel |
setSeed(long value) |
LDAModel |
setTopicDistributionCol(String value) |
DoubleParam |
subsamplingRate()
For Online optimizer only:
optimizer = "online". |
String[] |
supportedOptimizers()
Supported values for Param
optimizer. |
DoubleParam |
topicConcentration()
Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics'
distributions over terms.
|
Param<String> |
topicDistributionCol()
Output column with estimates of the topic mixture distribution for each document (often called
"theta" in the literature).
|
Matrix |
topicsMatrix()
Inferred topics, where each topic is represented by a distribution over terms.
|
Dataset<Row> |
transform(Dataset<?> dataset)
Transforms the input dataset.
|
StructType |
transformSchema(StructType schema)
:: DeveloperApi ::
|
String |
uid()
An immutable unique ID for the object and its derivatives.
|
StructType |
validateAndTransformSchema(StructType schema)
Validates and transforms the input schema.
|
int |
vocabSize() |
transform, transform, transformequals, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitfeaturesCol, getFeaturesColgetMaxIter, maxItercheckpointInterval, getCheckpointIntervalclear, copy, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwntoStringinitializeLogging, initializeLogIfNecessary, initializeLogIfNecessary, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarningsave, writepublic Dataset<Row> describeTopics(int maxTermsPerTopic)
maxTermsPerTopic - Maximum number of terms to collect for each topic.
Default value of 10.public Vector estimatedDocConcentration()
docConcentration estimated from data.
If Online LDA was used and optimizeDocConcentration was set to false,
then this returns the fixed (given) value for the docConcentration parameter.public abstract boolean isDistributed()
DistributedLDAModelpublic double logLikelihood(Dataset<?> dataset)
See Equation (16) in the Online LDA paper (Hoffman et al., 2010).
WARNING: If this model is an instance of DistributedLDAModel (produced when optimizer
is set to "em"), this involves collecting a large topicsMatrix to the driver.
This implementation may be changed in the future.
dataset - test corpus to use for calculating log likelihoodpublic double logPerplexity(Dataset<?> dataset)
WARNING: If this model is an instance of DistributedLDAModel (produced when optimizer
is set to "em"), this involves collecting a large topicsMatrix to the driver.
This implementation may be changed in the future.
dataset - test corpus to use for calculating perplexitypublic LDAModel setFeaturesCol(String value)
Vector representing the word counts in a document.
The vector should be of length vocabSize, with counts for each term (word).
value - (undocumented)public LDAModel setSeed(long value)
public LDAModel setTopicDistributionCol(String value)
public Matrix topicsMatrix()
WARNING: If this model is actually a DistributedLDAModel instance produced by
the Expectation-Maximization ("em") optimizer, then this method could involve
collecting a large amount of data to the driver (on the order of vocabSize x k).
public Dataset<Row> transform(Dataset<?> dataset)
WARNING: If this model is an instance of DistributedLDAModel (produced when optimizer
is set to "em"), this involves collecting a large topicsMatrix to the driver.
This implementation may be changed in the future.
transform in class Transformerdataset - (undocumented)public StructType transformSchema(StructType schema)
PipelineStageCheck transform validity and derive the output schema from the input schema.
We check validity for interactions between parameters during transformSchema and
raise an exception if any parameter value is invalid. Parameter value checks which
do not depend on other parameters are handled by Param.validate().
Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
transformSchema in class PipelineStageschema - (undocumented)public String uid()
Identifiableuid in interface Identifiablepublic int vocabSize()
public IntParam k()
public int getK()
public DoubleArrayParam docConcentration()
This is the parameter to a Dirichlet distribution, where larger values mean more smoothing (more regularization).
If not set by the user, then docConcentration is set automatically. If set to
singleton vector [alpha], then alpha is replicated to a vector of length k in fitting.
Otherwise, the docConcentration vector must be length k.
(default = automatic)
Optimizer-specific parameter settings: - EM - Currently only supports symmetric distributions, so all values in the vector should be the same. - Values should be greater than 1.0 - default = uniformly (50 / k) + 1, where 50/k is common in LDA libraries and +1 follows from Asuncion et al. (2009), who recommend a +1 adjustment for EM. - Online - Values should be greater than or equal to 0 - default = uniformly (1.0 / k), following the implementation from here.
public double[] getDocConcentration()
public Vector getOldDocConcentration()
public DoubleParam topicConcentration()
This is the parameter to a symmetric Dirichlet distribution.
Note: The topics' distributions over terms are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many later papers such as Asuncion et al., 2009.
If not set by the user, then topicConcentration is set automatically. (default = automatic)
Optimizer-specific parameter settings: - EM - Value should be greater than 1.0 - default = 0.1 + 1, where 0.1 gives a small amount of smoothing and +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM. - Online - Value should be greater than or equal to 0 - default = (1.0 / k), following the implementation from here.
public double getTopicConcentration()
public double getOldTopicConcentration()
public String[] supportedOptimizers()
optimizer.public Param<String> optimizer()
For details, see the following papers: - Online LDA: Hoffman, Blei and Bach. "Online Learning for Latent Dirichlet Allocation." Neural Information Processing Systems, 2010. See here - EM: Asuncion et al. "On Smoothing and Inference for Topic Models." Uncertainty in Artificial Intelligence, 2009. See here
public String getOptimizer()
public Param<String> topicDistributionCol()
This uses a variational approximation following Hoffman et al. (2010), where the approximate distribution is called "gamma." Technically, this method returns this approximation "gamma" for each document.
public String getTopicDistributionCol()
public DoubleParam learningOffset()
optimizer = "online".
A (positive) learning parameter that downweights early iterations. Larger values make early iterations count less. This is called "tau0" in the Online LDA paper (Hoffman et al., 2010) Default: 1024, following Hoffman et al.
public double getLearningOffset()
public DoubleParam learningDecay()
optimizer = "online".
Learning rate, set as an exponential decay rate. This should be between (0.5, 1.0] to guarantee asymptotic convergence. This is called "kappa" in the Online LDA paper (Hoffman et al., 2010). Default: 0.51, based on Hoffman et al.
public double getLearningDecay()
public DoubleParam subsamplingRate()
optimizer = "online".
Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1].
Note that this should be adjusted in synch with LDA.maxIter
so the entire corpus is used. Specifically, set both so that
maxIterations * miniBatchFraction greater than or equal to 1.
Note: This is the same as the miniBatchFraction parameter in
OnlineLDAOptimizer.
Default: 0.05, i.e., 5% of total documents.
public double getSubsamplingRate()
public BooleanParam optimizeDocConcentration()
optimizer = "online".
Indicates whether the docConcentration (Dirichlet parameter for document-topic distribution) will be optimized during training. Setting this to true will make the model more expressive and fit the training data better. Default: false
public boolean getOptimizeDocConcentration()
public BooleanParam keepLastCheckpoint()
optimizer = "em".
If using checkpointing, this indicates whether to keep the last checkpoint. If false, then the checkpoint will be deleted. Deleting the checkpoint can cause failures if a data partition is lost, so set this bit with care. Note that checkpoints will be cleaned up via reference counting, regardless.
See DistributedLDAModel.getCheckpointFiles for getting remaining checkpoints and
DistributedLDAModel.deleteCheckpointFiles for removing remaining checkpoints.
Default: true
public boolean getKeepLastCheckpoint()
public StructType validateAndTransformSchema(StructType schema)
schema - input schemapublic LDAOptimizer getOldOptimizer()