Fast, weighted ROC curves

tests
coverage

Receiver Operating Characteristic (ROC) curve analysis is one way to evaluate an algorithm for binary classification. R packages ROCR/pROC/AUC/PerfMeas/PRROC implement ROC curve computation. However, if the observations have weights (non-uniform loss, see Definition vignette) then these packages can not be used. The WeightedROC package implements ROC and Area Under the Curve (AUC) computation for weighted binary classification problems.

From CRAN:

install.packages("WeightedROC")

From GitHub:

if(!require(devtools))install.packages("devtools")
devtools::install_github("tdhock/WeightedROC")

library(WeightedROC)
example(WeightedROC)
example(WeightedAUC)

• weights shows which packages allow weights (non-uniform loss for each observation).
• tests shows which R packages implement unit tests to check that the ROC/AUC is computed correctly.
• lines of R code shows how many lines of code were used in the pkg/R/* files. Note that WeightedROC has the simplest implementation other than glmnet::auc.
• cumsum shows whether or not the cumsum function is used to compute the ROC curve. Using the cumsum function is simple to code and fast – see the Speed vignette.

For “soft” real-valued labels (not “hard” labels $∈ \{-1, 1\}$), and accurate interpolation of Precision-Recall curves, use PRROC. Note that PRROC uses the word “Weighted” to mean something completely different (soft labels) than the weights in this package (non-uniform loss), as explained in their PLOS ONE paper.

To compute other evaluation metrics (e.g. lift) use the ROCR package. WeightedROC does not implement evaluation metrics other than ROC/AUC.

To compute the partial AUC and compare curves using statistical tests use the pROC package. WeightedROC does not implement these features.

The glmnet package includes an auc function for computing AUC, but does not include a function for computing the ROC curve. So it actually can compute the AUC faster than WeightedROC, for both equal or unequal weights. WARNINGS:

• make sure the class labels are either 0 or 1 (not factors, not -1 or 1 – these will give the incorrect result, with no warning/error).
• if the data set has tied scores AND weights, glmnet::auc computes something different, see example(WeightedAUC).