Measuring performance over extreme subsets:

One of the main problems of evaluating models in the context of risk management is that the performance is gauged by looking at the few adverse events beyond the 99% confidence level or associated with the few strongest adverse price movements in relation to some portfolio. Even with a back testing history of 1000 days (as used in this paper) we expect only 10 events beyond the 99% confidence level. Statistics and performance measures based solely on these few events cannot help clearly distinguish good models from bad. Our approach of plotting model performance against confidence level or volatility percentile is motivated by exactly this problem. As discussed in section , the confidence level or the volatility percentile (plotted on the X axis of all figures) parametrize a class of increasingly extreme partial sets of events by differing definitions. By probing model performance as a function of increasingly extreme (but at the same time increasingly depopulated) event classes - we can get a better feel for the behaviour in the limit of extreme behaviour for that class type. We propose that the consistent superiority of one model over another over an entire range of confidence levels or volatility percentiles - approaching the 99% level (say from 75% to 99%) - increases confidence in the model. In addition by looking at the extrapolated approach to the 99% level from the less depopulated classes - we can better appreciate the error in the empirically determined measure at this level.