Confusion Matrix and Metrics

Adopting two perspectives on a population

Essentially, the confusion matrix views a population of N individuals in different ways by adopting different perspectives. “Adopting a perspective” means that we can distinguish between individuals on the basis of some criterion. The two primary criteria used here are:

(a) each individual’s condition, which can either be present (TRUE) or absent (FALSE), and
(b) each individual’s decision, which can either be positive (TRUE) or negative (FALSE).

Numerically, the adoption of each of these two perspectives splits the population into two subgroups.¹ Applying two different splits of a population into two subgroups results in 2 × 2 = 4 cases, which form the core of the confusion matrix:

1. hi represents hits (or true positives): condition present (TRUE) & decision positive (TRUE).
2. mi represents misses (or false negatives): condition present (TRUE) & decision negative (FALSE).
3. fa represents false alarms (or false positives): condition absent (FALSE) & decision positive (TRUE).
4. cr represents correct rejections (or true negatives): condition absent (FALSE) & decision negative (FALSE).

Importantly, all frequencies required to understand and compute various metrics are combinations of these four frequencies — which is why we refer to them as the four essential frequencies (see the vignette on Data formats). For instance, adding up the columns and rows of the matrix yields the frequencies of the two subgroups that result from adopting our two perspectives on the population N (or splitting N into subgroups by applying two binary criteria):

(a) by condition (cd) (corresponding to the two columns of the confusion matrix):

$$ \begin{aligned} \texttt{N} \ &= \ \texttt{cond_true} & +\ \ \ \ \ &\texttt{cond_false} & \textrm{(a)} \\ \ &= \ (\texttt{hi} + \texttt{mi}) & +\ \ \ \ \ &(\texttt{fa} + \texttt{cr}) \\ \end{aligned} $$

(b) by decision (dc) (corresponding to the two rows of the confusion matrix):

$$ \begin{aligned} \texttt{N} \ &= \ \texttt{dec_pos} & +\ \ \ \ \ &\texttt{dec_neg} & \ \ \ \ \textrm{(b)} \\ \ &= \ (\texttt{hi} + \texttt{fa}) & +\ \ \ \ \ &(\texttt{mi} + \texttt{cr}) \\ \end{aligned} $$

To reflect these two perspectives in the confusion matrix, we only need to add the sums of columns (i.e., by condition) and rows (by decision):

An third perspective is provided by considering the diagonals of the 2x2 matrix. In many semantic domains, the diagonals denote the accuracy of the classification, or the correspondence between dimensions (see below).

Example

To view a 2x2 matrix (or confusion table) in riskyr, we can use the plot_tab() function (i.e., plot an existing scenario as type = "tab"):

## (1) Plot table from basic input parameters: ----- 
plot_tab(prev = .05, sens = .75, spec = .66, N = 1000,
         p_lbl = "def") # show condition probabilies (by default)

Example of a 2x2 confusion table in riskyr.

Accuracy as a third perspective

A third way of grouping the four essential frequencies results from asking the question: Which of the four essential frequencies are correct decisions and which are erroneous decisions? Crucially, this question about decision accuracy can neither be answered by only considering each individual’s condition (i.e., the columns of the matrix), nor can it be answered by only considering each individual’s decision (i.e., the rows of the matrix). Instead, answering the question about accuracy requires that the other dimensions have been determined and then considering the correspondence between condition and decision. Checking the correspondence between rows and columns for the four essential frequencies yields an important insight: The confusion matrix contains two types of correct decisions and two types of errors:

• A decision is correct, when it corresponds to the condition. This is the case for two cells in (or the “" diagonal of) the confusion matrix:

  • hi: condition present (TRUE) & decision positive (TRUE)
  • cr: condition absent (FALSE) & decision negative (FALSE)

• A decision is incorrect or erroneous, when it does not correspond to the condition. This also is the case for two cells in (or the “/” diagonal of) the confusion matrix:

  • mi: condition present (TRUE) & decision negative (FALSE)
  • fa: condition absent (FALSE) & decision positive (TRUE)

Splitting all N individuals into two subgroups of those with correct vs. those with erroneous decisions yields a third perspective on the population:

(c) by accuracy (ac) or the correspondence between decisions and conditions (corresponding to the two diagonals of the confusion matrix):

$$ \begin{aligned} \texttt{N} \ &= \ \texttt{dec_cor} & +\ \ \ \ \ &\texttt{dec_err} & \ \ \textrm{(c)} \\ \ &= \ (\texttt{hi} + \texttt{cr}) & +\ \ \ \ \ &(\texttt{mi} + \texttt{fa}) \\ \end{aligned} $$

Example

Re-arranging the cells of the 2x2 matrix allows illustrating accuracy as a third perspective (e.g., by specifying the perspective by = "cdac"):

plot_tab(prev = .05, sens = .75, spec = .66, N = 1000,
         by = "cdac", p_split = "h", 
         p_lbl = "def", title_lbl = "Scenario 2")
#> Argument 'title_lbl' is deprecated. Please use 'main' instead.

Arranging a 2x2 confusion table by condition and by accuracy.

A. Specific metrics: Conditional probabilities

The goal of a specific accuracy metric is to quantify some particular aspect of decision performance. For instance, how accurate is our decision or diagnostic test in correctly detecting cond_true cases? How accurate is it in detecting cond_false cases?

As we are dealing with two types of correct decisions (hi and cr) and two perspectives (by columns vs. by rows), we can provide 4 answers to these questions. To obtain a numeric quantity, we divide the frequency of correct cases (either hi or cr) by

(a) column sums (cond_true vs. cond_false): This yields the decision’s sensitivity (sens) and specificity (spec):

$$ \begin{aligned} \texttt{sens} \ &= \frac{\texttt{hi}}{\texttt{cond_true}} & \ \ \textrm{(a1)} \\ \ \\ \texttt{spec} \ &= \frac{\texttt{cr}}{\texttt{cond_false}} & \ \ \textrm{(a2)} \\ \end{aligned} $$

(b) row sums (dec_pos vs. dec_neg): This yields the decision’s positive predictive value (PPV) and negative predictive value (NPV):

$$ \begin{aligned} \texttt{PPV} \ &= \frac{\texttt{hi}}{\texttt{dec_pos}} & \ \ \ \textrm{(b1)} \\ \ \\ \texttt{NPV} \ &= \frac{\texttt{cr}}{\texttt{dec_neg}} & \ \ \ \textrm{(b2)} \\ \end{aligned} $$

B. General metrics: Measures of accuracy

In contrast to these specific metrics, general metrics of accuracy aim to capture overall performance (i.e., summarize the four essential frequencies of the confusion matrix) in a single quantity. riskyr currently computes four general metrics (which are contained in accu):

Resources

The following resources and versions are currently available: