How To Evaluate an Early Warning System ? Towards a unified Statistical Framework for Assessing Financial Crises
Forecasting Methods
Early Warning System (EWS) Evaluation Toolbox
==========================================================================
INPUT:
- observed crises series for a certain country (1 - crisis; 0 - calm)
- probability sequence (obtained from any type of EWS model)
OPTIONAL: a second probability sequence
and the option nested=1 % 1 if the 2 models are nested (m2 nestes m1 !!!)
-------------------------------------------------------------------------OUTPUT:
1. For each model (series of probabilities)
Identification fo the optimal cut-off
- Optimal cutoff (sensitivity-specificity criteria)
- Confusion matrix for this optimal cut-off
- Optimal cutoff (accuracy measures - Younden index)
- Confusion matrix for this optimal cut-off
- Optimal cutoff (misclassification error measures-Total error)
- Confusion matrix for this optimal cut-off
Evaluation criteria:
-QPS;
-LPS;
-AUC;
-Pietra Index;
Plots:
- Sensitivity-Specificity
- Accuracy and Misclassification Error Measures
- ROC curve
- Crisis Probabilities vs. Observed Crises
2. Comparison Tests (only if 2 series of probabilities are specified)
Clark-West and ROC tests: ( test-statistic and p-value)
H0: the 2 models have equal forecasting abilities
H1: the models have different forecasting abilities (identify the higher
AUC (res. the lower QPS) to find the outperforming model)
Results for model 1
-----------------------------------------------------------------------------I) Optimal cut-off Identification
=====================================================
a. Sensitivity-Specificity criteria
-----------------------------------------------------
page 1
cut-off = 0.091
Observed: no crisis
0.956
crisis
Predicted:
no crisis
crisis
0.044
0.077
0.923
b. Accuracy Measures - Younden Index
----------------------------------------------------cut-off = 0.069
Observed: no crisis
0.956
crisis
Predicted:
no crisis
crisis
0.044
0.000
1.000
c. Misclassification Error Measures - Total Error
----------------------------------------------------cut-off = 0.069
Observed: no crisis
0.956
crisis
Predicted:
no crisis
crisis
0.044
0.000
1.000
II) Evaluation criteria
=====================================================
QPS
0.038
LPS
0.059
AUC
0.989
Pietra
0.338
=====================================================
Results for model 2
-----------------------------------------------------------------------------I) Optimal cut-off Identification
=====================================================
a. Sensitivity-Specificity criteria
----------------------------------------------------cut-off = 0.241
page 2
Observed: no crisis
0.956
crisis
Predicted:
no crisis
crisis
0.044
0.077
0.923
b. Accuracy Measures - Younden Index
----------------------------------------------------cut-off = 0.219
Observed: no crisis
0.956
crisis
Predicted:
no crisis
crisis
0.044
0.000
1.000
c. Misclassification Error Measures - Total Error
----------------------------------------------------cut-off = 0.219
Observed: no crisis
0.956
crisis
Predicted:
no crisis
crisis
0.044
0.000
1.000
II) Evaluation criteria
=====================================================
QPS
0.081
LPS
0.209
AUC
0.988
Pietra
0.338
=====================================================
Results for the comparison tests
-----------------------------------------------------------------------------II) Comparison tests
=====================================================
ROC
Clark-West
Test statistic 0.842
0.252
P-value
0.359
0.801
=====================================================
page 3
Sensitivity−Specificity Plot
Crisis Probability vs. Observed crisis
1
1
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0
0
50
100
150
Time
Observed Crisis
200
250
Optimal cut−off
300
0
0
0.1
0.2
0.3
0.4
0.5
Cut−off
Sensitivity
EWS
0.6
0.7
0.8
0.9
1
0.9
1
Specificity
Accuracy and Misclassification Error Measures Plot
1
Receiving Operating Characteristic (ROC) Curve
1
0.9
0.9
0.8
0.8
0.7
0.7
0.6
Sensitivity
0.6
0.5
0.4
0.5
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0
0
0.1
0.2
0.3
Younden index
0.4
0.5
Cut−off
MCC
0.6
Total error
0.7
0.8
Weighted error
0.9
1
0
0
0.1
0.2
0.3
0.4
0.5
0.6
1−Specificity
EWS
page 4
Random model
0.7
0.8
Sensitivity−Specificity Plot
Crisis Probability vs. Observed crisis
1
1
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0
0
50
100
150
Time
Observed Crisis
200
250
Optimal cut−off
300
0
0
0.1
0.2
0.3
0.4
0.5
Cut−off
Sensitivity
EWS
0.6
0.7
0.8
0.9
1
0.9
1
Specificity
Accuracy and Misclassification Error Measures Plot
1
Receiving Operating Characteristic (ROC) Curve
1
0.9
0.9
0.8
0.8
0.7
0.7
0.6
Sensitivity
0.6
0.5
0.4
0.5
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0
0
0.1
0.2
0.3
Younden index
0.4
0.5
Cut−off
MCC
0.6
Total error
0.7
0.8
Weighted error
0.9
1
0
0
0.1
0.2
0.3
0.4
0.5
0.6
1−Specificity
EWS
page 5
Random model
0.7
0.8
CODER INFORMATION
Bertrand Candelon
Maastricht University
Elena-Ivona Dumitrescu
University of Orleans
Christophe Hurlin
University of Orleans
PLEASE CITE THE PUBLICATION
AND THE COMPANION SITE
Candelon, B., Dumitrescu, E. and Hurlin, C.
(2012), ”How To Evaluate an Early Warning System ? Towards a unified Statistical
Framework for Assessing Financial Crises
Forecasting Methods”
, IMF Economic
Review
Candelon B., Dumitrescu E. and Hurlin C.
(2012), ”How To Evaluate an Early
Warning System ? Towards a unified Statistical Framework for Assessing Financial Crises Forecasting Methods”.
RunShare Companion Website
www.runshare.org/CompanionSite/Site70
Powered by Matlab
page 6