Metrics
Brier Skill Score
CRPSS
Reliability plot
Relative Economic Value
Relative Mean Error
Relative Operating Characteristic score
References
References
Verification software
Miscellaneous
LateX users
Metrics
Brier Skill Score
CRPSS
Reliability plot
Relative Economic Value
Relative Mean Error
Relative Operating Characteristic score
References
References
Verification software
Miscellaneous
LateX users
The relative operating characteristic (ROC; [green_signal_1966]) plots the hit rate versus the false alarm rate. These are calculated using the elements of a contingency table, which is valid for a single probabilistic decision rule,
Event observed | Event NOT observed | $\sum$ | |
---|---|---|---|
Warning issued | hits $h \left( C + L_u \right)$ | false alarms $f \left( C \right) $ | $w$ |
Warning NOT issued | missed events $m \left( L_a + L_u \right)$ | quiets/correct negatives $q \left( - \right) $ | $w'$ |
$\sum$ | $o$ | $o'$ | $N$ |
Contingency table. The consequences of the items listed are in brackets. Taken from [verkade_estimating_2011].
and are defined as follows
\begin{align} \textrm{hit rate} & = \frac{\textrm{# hits}}{\textrm{# observed events}} = \frac{h}{o}\\ \textrm{false alarm rate} & = \frac{\textrm{# false alarms}}{\textrm{# events not observed}} = \frac{f}{o'} \textrm{,} \nonumber \end{align}
The ROC score measures the area under the ROC curve (AUC) after adjusting for randomness, i.e.
\begin{equation} \mathrm{ROCS} = 2 \times \left( \mathrm{AUC} - 0.5 \right) \textrm{.} \end{equation}