org.apache.ctakes.assertion.eval
Class AnnotationStatisticsCompact<OUTCOME_TYPE extends Comparable<? super OUTCOME_TYPE>>

java.lang.Object
  org.apache.ctakes.assertion.eval.AnnotationStatisticsCompact<OUTCOME_TYPE>

All Implemented Interfaces:

Serializable

public class AnnotationStatisticsCompact<OUTCOME_TYPE extends Comparable<? super OUTCOME_TYPE>>
extends Object
implements Serializable

Stores statistics for comparing Annotations extracted by a system to gold Annotations.
Copyright (c) 2012, Regents of the University of Colorado
All rights reserved.

Author:

Steven Bethard

See Also:

Serialized Form

Constructor Summary

AnnotationStatisticsCompact()
Create an AnnotationStatisticsCompact that compares Annotations based on their begin and end offsets, plus a Feature of the Annotation that represents the outcome or label.

Method Summary

<ANNOTATION_TYPE extends org.apache.uima.jcas.tcas.Annotation> void	add(Collection<? extends ANNOTATION_TYPE> referenceAnnotations, Collection<? extends ANNOTATION_TYPE> predictedAnnotations) Update the statistics, comparing the reference annotations to the predicted annotations.
<ANNOTATION_TYPE,SPAN_TYPE> void	add(Collection<? extends ANNOTATION_TYPE> referenceAnnotations, Collection<? extends ANNOTATION_TYPE> predictedAnnotations, com.google.common.base.Function<ANNOTATION_TYPE,SPAN_TYPE> annotationToSpan, com.google.common.base.Function<ANNOTATION_TYPE,OUTCOME_TYPE> annotationToOutcome) Update the statistics, comparing the reference annotations to the predicted annotations.
void	addAll(AnnotationStatisticsCompact<OUTCOME_TYPE> that) Adds all the statistics collected by another AnnotationStatisticsCompact to this one.
static <OUTCOME_TYPE extends Comparable<? super OUTCOME_TYPE>> AnnotationStatisticsCompact<OUTCOME_TYPE>	addAll(Iterable<AnnotationStatisticsCompact<OUTCOME_TYPE>> statistics) Add all statistics together.
static <ANNOTATION_TYPE extends org.apache.uima.jcas.cas.TOP> com.google.common.base.Function<ANNOTATION_TYPE,String>	annotationToFeatureValue(String featureName) Creates a Function that extracts a feature value from a TOP.
static <ANNOTATION_TYPE,OUTCOME_TYPE> com.google.common.base.Function<ANNOTATION_TYPE,OUTCOME_TYPE>	annotationToNull() Creates a Function that always returns null.
static <ANNOTATION_TYPE extends org.apache.uima.jcas.tcas.Annotation> com.google.common.base.Function<ANNOTATION_TYPE,org.apache.ctakes.assertion.eval.AnnotationStatisticsCompact.Span>	annotationToSpan() Creates a Function that converts an Annotation into a hashable representation of its begin and end offsets.
org.cleartk.eval.util.ConfusionMatrix<OUTCOME_TYPE>	confusions() Returns the ConfusionMatrix tabulating reference outcomes matched to predicted outcomes.
int	countCorrectOutcomes()
int	countCorrectOutcomes(OUTCOME_TYPE outcome)
int	countFalseNegatives(OUTCOME_TYPE... positiveOutcomes)
int	countFalsePositives(OUTCOME_TYPE... positiveOutcomes)
int	countPredictedOutcomes()
int	countPredictedOutcomes(OUTCOME_TYPE outcome)
int	countReferenceOutcomes()
int	countReferenceOutcomes(OUTCOME_TYPE outcome)
int	countTrueNegatives(OUTCOME_TYPE... positiveOutcomes)
int	countTruePositives(OUTCOME_TYPE... positiveOutcomes)
double	f(double beta)
double	f(double beta, OUTCOME_TYPE outcome)
double	f1()
double	f1(OUTCOME_TYPE outcome)
double	precision()
double	precision(OUTCOME_TYPE outcome)
double	recall()
double	recall(OUTCOME_TYPE outcome)
String	toString()
String	toTsv()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

AnnotationStatisticsCompact

public AnnotationStatisticsCompact()

Create an AnnotationStatisticsCompact that compares Annotations based on their begin and end offsets, plus a Feature of the Annotation that represents the outcome or label.

Method Detail

annotationToSpan

public static <ANNOTATION_TYPE extends org.apache.uima.jcas.tcas.Annotation> com.google.common.base.Function<ANNOTATION_TYPE,org.apache.ctakes.assertion.eval.AnnotationStatisticsCompact.Span> annotationToSpan()

Creates a Function that converts an Annotation into a hashable representation of its begin and end offsets. The Function created by this method is suitable for passing to the first Function argument of add(Collection, Collection, Function, Function).

annotationToFeatureValue

public static <ANNOTATION_TYPE extends org.apache.uima.jcas.cas.TOP> com.google.common.base.Function<ANNOTATION_TYPE,String> annotationToFeatureValue(String featureName)

Creates a Function that extracts a feature value from a TOP. The Function created by this method is suitable for passing to the second Function argument of add(Collection, Collection, Function, Function).

Parameters:

featureName - The name of the feature whose value is to be extracted.

annotationToNull

public static <ANNOTATION_TYPE,OUTCOME_TYPE> com.google.common.base.Function<ANNOTATION_TYPE,OUTCOME_TYPE> annotationToNull()

Creates a Function that always returns null. This may be useful when only the span of the offset is important, but you still need to pass in the final argument of add(Collection, Collection, Function, Function).

addAll

public static <OUTCOME_TYPE extends Comparable<? super OUTCOME_TYPE>> AnnotationStatisticsCompact<OUTCOME_TYPE> addAll(Iterable<AnnotationStatisticsCompact<OUTCOME_TYPE>> statistics)

Add all statistics together. This is often useful for combining individual fold statistics that result from methods like Evaluation_ImplBase#crossValidation(List, int).

Parameters:

statistics - The sequence of statistics that should be combined.

Returns:

The combination of all the individual statistics.

add

public <ANNOTATION_TYPE extends org.apache.uima.jcas.tcas.Annotation> void add(Collection<? extends ANNOTATION_TYPE> referenceAnnotations,
                                                                               Collection<? extends ANNOTATION_TYPE> predictedAnnotations)

Update the statistics, comparing the reference annotations to the predicted annotations. Annotations are considered to match if they have the same character offsets in the text. All outcomes (e.g. as returned in confusions()) will be null.

Parameters:

referenceAnnotations - The reference annotations, typically identified by humans.

predictedAnnotations - The predicted annotations, typically identified by a model.

add

public <ANNOTATION_TYPE,SPAN_TYPE> void add(Collection<? extends ANNOTATION_TYPE> referenceAnnotations,
                                            Collection<? extends ANNOTATION_TYPE> predictedAnnotations,
                                            com.google.common.base.Function<ANNOTATION_TYPE,SPAN_TYPE> annotationToSpan,
                                            com.google.common.base.Function<ANNOTATION_TYPE,OUTCOME_TYPE> annotationToOutcome)

Update the statistics, comparing the reference annotations to the predicted annotations. Annotations are considered to match if they have the same span (according to annotationToSpan) and if they have the same outcome (according to annotationToOutcome).

Parameters:

referenceAnnotations - The reference annotations, typically identified by humans.

predictedAnnotations - The predicted annotations, typically identified by a model.

annotationToSpan - A function that defines how to convert an annotation into a hashable object that represents the span of that annotation. The annotationToSpan() method provides an example function that could be used here.

annotationToOutcome - A function that defines how to convert an annotation into an object that represents the outcome (or "label") assigned to that annotation. The annotationToFeatureValue(String) method provides a sample function that could be used here.

addAll

public void addAll(AnnotationStatisticsCompact<OUTCOME_TYPE> that)

Adds all the statistics collected by another AnnotationStatisticsCompact to this one.

Parameters:

that - The other statistics that should be added to this one.

countCorrectOutcomes

public int countCorrectOutcomes()

countCorrectOutcomes

public int countCorrectOutcomes(OUTCOME_TYPE outcome)

countPredictedOutcomes

public int countPredictedOutcomes()

countPredictedOutcomes

public int countPredictedOutcomes(OUTCOME_TYPE outcome)

countReferenceOutcomes

public int countReferenceOutcomes()

countReferenceOutcomes

public int countReferenceOutcomes(OUTCOME_TYPE outcome)

countFalseNegatives

public int countFalseNegatives(OUTCOME_TYPE... positiveOutcomes)

countFalsePositives

public int countFalsePositives(OUTCOME_TYPE... positiveOutcomes)

countTrueNegatives

public int countTrueNegatives(OUTCOME_TYPE... positiveOutcomes)

countTruePositives

public int countTruePositives(OUTCOME_TYPE... positiveOutcomes)

confusions

public org.cleartk.eval.util.ConfusionMatrix<OUTCOME_TYPE> confusions()

Returns the ConfusionMatrix tabulating reference outcomes matched to predicted outcomes.

Returns:

The confusion matrix.

precision

public double precision()

precision

public double precision(OUTCOME_TYPE outcome)

recall

public double recall()

recall

public double recall(OUTCOME_TYPE outcome)

f

public double f(double beta)

f

public double f(double beta,
                OUTCOME_TYPE outcome)

f1

public double f1()

f1

public double f1(OUTCOME_TYPE outcome)

toString

public String toString()

Overrides:

toString in class Object

toTsv

public String toTsv()