public abstract class LDAModel extends Model<LDAModel> implements Logging, MLWritable
LDA
.
param: vocabSize Vocabulary size (number of terms or terms in the vocabulary) param: sqlContext Used to construct local DataFrames for returning query results
Modifier and Type | Method and Description |
---|---|
IntParam |
checkpointInterval()
Param for set checkpoint interval (>= 1) or disable checkpoint (-1).
|
DataFrame |
describeTopics() |
DataFrame |
describeTopics(int maxTermsPerTopic)
Return the topics described by their top-weighted terms.
|
DoubleArrayParam |
docConcentration()
Concentration parameter (commonly named "alpha") for the prior placed on documents'
distributions over topics ("theta").
|
Vector |
estimatedDocConcentration()
Value for
docConcentration estimated from data. |
Param<java.lang.String> |
featuresCol()
Param for features column name.
|
int |
getCheckpointInterval() |
double[] |
getDocConcentration() |
java.lang.String |
getFeaturesCol() |
int |
getK() |
double |
getLearningDecay() |
double |
getLearningOffset() |
int |
getMaxIter() |
protected abstract LDAModel |
getModel()
Returns underlying spark.mllib model, which may be local or distributed
|
Vector |
getOldDocConcentration()
Get docConcentration used by spark.mllib LDA
|
LDAOptimizer |
getOldOptimizer() |
double |
getOldTopicConcentration()
Get topicConcentration used by spark.mllib LDA
|
boolean |
getOptimizeDocConcentration() |
java.lang.String |
getOptimizer() |
long |
getSeed() |
double |
getSubsamplingRate() |
double |
getTopicConcentration() |
java.lang.String |
getTopicDistributionCol() |
abstract boolean |
isDistributed()
Indicates whether this instance is of type
DistributedLDAModel |
IntParam |
k()
Param for the number of topics (clusters) to infer.
|
DoubleParam |
learningDecay()
Learning rate, set as an exponential decay rate.
|
DoubleParam |
learningOffset()
A (positive) learning parameter that downweights early iterations.
|
double |
logLikelihood(DataFrame dataset)
Calculates a lower bound on the log likelihood of the entire corpus.
|
double |
logPerplexity(DataFrame dataset)
Calculate an upper bound bound on perplexity.
|
IntParam |
maxIter()
Param for maximum number of iterations (>= 0).
|
protected abstract LocalLDAModel |
oldLocalModel()
Underlying spark.mllib model.
|
BooleanParam |
optimizeDocConcentration()
Indicates whether the docConcentration (Dirichlet parameter for
document-topic distribution) will be optimized during training.
|
Param<java.lang.String> |
optimizer()
Optimizer or inference algorithm used to estimate the LDA model.
|
LongParam |
seed()
Param for random seed.
|
LDAModel |
setFeaturesCol(java.lang.String value)
The features for LDA should be a
Vector representing the word counts in a document. |
LDAModel |
setSeed(long value) |
protected SQLContext |
sqlContext() |
DoubleParam |
subsamplingRate()
Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent,
in range (0, 1].
|
java.lang.String[] |
supportedOptimizers()
Supported values for Param
optimizer . |
DoubleParam |
topicConcentration()
Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics'
distributions over terms.
|
Param<java.lang.String> |
topicDistributionCol()
Output column with estimates of the topic mixture distribution for each document (often called
"theta" in the literature).
|
Matrix |
topicsMatrix()
Inferred topics, where each topic is represented by a distribution over terms.
|
DataFrame |
transform(DataFrame dataset)
Transforms the input dataset.
|
StructType |
transformSchema(StructType schema)
:: DeveloperApi ::
|
java.lang.String |
uid()
An immutable unique ID for the object and its derivatives.
|
StructType |
validateAndTransformSchema(StructType schema)
Validates and transforms the input schema.
|
void |
validateParams() |
int |
vocabSize() |
transform, transform, transform
transformSchema
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
clear, copy, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn
toString
initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning
save, write
public java.lang.String uid()
Identifiable
uid
in interface Identifiable
public int vocabSize()
protected SQLContext sqlContext()
protected abstract LocalLDAModel oldLocalModel()
protected abstract LDAModel getModel()
public LDAModel setFeaturesCol(java.lang.String value)
Vector
representing the word counts in a document.
The vector should be of length vocabSize, with counts for each term (word).value
- (undocumented)public LDAModel setSeed(long value)
public DataFrame transform(DataFrame dataset)
WARNING: If this model is an instance of DistributedLDAModel
(produced when optimizer
is set to "em"), this involves collecting a large topicsMatrix
to the driver.
This implementation may be changed in the future.
transform
in class Transformer
dataset
- (undocumented)public StructType transformSchema(StructType schema)
PipelineStage
Derives the output schema from the input schema.
transformSchema
in class PipelineStage
schema
- (undocumented)public Vector estimatedDocConcentration()
docConcentration
estimated from data.
If Online LDA was used and optimizeDocConcentration
was set to false,
then this returns the fixed (given) value for the docConcentration
parameter.public Matrix topicsMatrix()
WARNING: If this model is actually a DistributedLDAModel
instance produced by
the Expectation-Maximization ("em") optimizer
, then this method could involve
collecting a large amount of data to the driver (on the order of vocabSize x k).
public abstract boolean isDistributed()
DistributedLDAModel
public double logLikelihood(DataFrame dataset)
See Equation (16) in the Online LDA paper (Hoffman et al., 2010).
WARNING: If this model is an instance of DistributedLDAModel
(produced when optimizer
is set to "em"), this involves collecting a large topicsMatrix
to the driver.
This implementation may be changed in the future.
dataset
- test corpus to use for calculating log likelihoodpublic double logPerplexity(DataFrame dataset)
WARNING: If this model is an instance of DistributedLDAModel
(produced when optimizer
is set to "em"), this involves collecting a large topicsMatrix
to the driver.
This implementation may be changed in the future.
dataset
- test corpus to use for calculating perplexitypublic DataFrame describeTopics(int maxTermsPerTopic)
maxTermsPerTopic
- Maximum number of terms to collect for each topic.
Default value of 10.public DataFrame describeTopics()
public IntParam k()
public int getK()
public DoubleArrayParam docConcentration()
This is the parameter to a Dirichlet distribution, where larger values mean more smoothing (more regularization).
If not set by the user, then docConcentration is set automatically. If set to
singleton vector [alpha], then alpha is replicated to a vector of length k in fitting.
Otherwise, the docConcentration
vector must be length k.
(default = automatic)
Optimizer-specific parameter settings:
- EM
- Currently only supports symmetric distributions, so all values in the vector should be
the same.
- Values should be > 1.0
- default = uniformly (50 / k) + 1, where 50/k is common in LDA libraries and +1 follows
from Asuncion et al. (2009), who recommend a +1 adjustment for EM.
- Online
- Values should be >= 0
- default = uniformly (1.0 / k), following the implementation from
https://github.com/Blei-Lab/onlineldavb
.
public double[] getDocConcentration()
public Vector getOldDocConcentration()
public DoubleParam topicConcentration()
This is the parameter to a symmetric Dirichlet distribution.
Note: The topics' distributions over terms are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many later papers such as Asuncion et al., 2009.
If not set by the user, then topicConcentration is set automatically. (default = automatic)
Optimizer-specific parameter settings:
- EM
- Value should be > 1.0
- default = 0.1 + 1, where 0.1 gives a small amount of smoothing and +1 follows
Asuncion et al. (2009), who recommend a +1 adjustment for EM.
- Online
- Value should be >= 0
- default = (1.0 / k), following the implementation from
https://github.com/Blei-Lab/onlineldavb
.
public double getTopicConcentration()
public double getOldTopicConcentration()
public java.lang.String[] supportedOptimizers()
optimizer
.public Param<java.lang.String> optimizer()
For details, see the following papers:
- Online LDA:
Hoffman, Blei and Bach. "Online Learning for Latent Dirichlet Allocation."
Neural Information Processing Systems, 2010.
http://www.cs.columbia.edu/~blei/papers/HoffmanBleiBach2010b.pdf
- EM:
Asuncion et al. "On Smoothing and Inference for Topic Models."
Uncertainty in Artificial Intelligence, 2009.
http://arxiv.org/pdf/1205.2662.pdf
public java.lang.String getOptimizer()
public Param<java.lang.String> topicDistributionCol()
This uses a variational approximation following Hoffman et al. (2010), where the approximate distribution is called "gamma." Technically, this method returns this approximation "gamma" for each document.
public java.lang.String getTopicDistributionCol()
public DoubleParam learningOffset()
public double getLearningOffset()
public DoubleParam learningDecay()
public double getLearningDecay()
public DoubleParam subsamplingRate()
Note that this should be adjusted in synch with LDA.maxIter
so the entire corpus is used. Specifically, set both so that
maxIterations * miniBatchFraction >= 1.
Note: This is the same as the miniBatchFraction
parameter in
OnlineLDAOptimizer
.
Default: 0.05, i.e., 5% of total documents.
public double getSubsamplingRate()
public BooleanParam optimizeDocConcentration()
public boolean getOptimizeDocConcentration()
public StructType validateAndTransformSchema(StructType schema)
schema
- input schemapublic void validateParams()
validateParams
in interface Params
public LDAOptimizer getOldOptimizer()
public Param<java.lang.String> featuresCol()
public java.lang.String getFeaturesCol()
public IntParam maxIter()
public int getMaxIter()
public LongParam seed()
public long getSeed()
public IntParam checkpointInterval()
public int getCheckpointInterval()