Performance measures

 The purpose of this paper in not simply the study of the different candidate models itemized in section  but also to present the different subtleties in evaluation of models in the context of risk management. For better visualization of performance, the above measures are plotted for each model as a function of increasingly extreme classes of realizations. To construct the classes of extreme realizations we use two different parameters such that - as they increase - they define a class of more and more extreme observations. These are:

Volatility percentile: The candidate measure is plotted against the percentile level of |x_t^(k,P)| - and for a given level the measure is evaluated as a count or average over those realizations which correspond to a higher percentile level.

Confidence level: The candidate measure is plotted against a confidence level - and for a given level the measure is evaluated as a count or average over those realizations which are outside the confidence level.

There is a subtle difference between the two kinds of plots in terms of how the X-axis refers to an increasingly extreme set of points. The first plot directly evaluates the performance of the model for large price changes. The second evaluates the performance of the model for movements which are regarded as increasingly improbable by the model itself.

From the latter, it should be clear that - when comparing plots for different models against the volatility percentile - a given point on the X-axis will always refer to the same data. On the other hand, for plots against the confidence level - a point on the X-axis will refer to differing data sets since this discrimination depends on the model itself.

The performance measures presented in this paper are:

1.

Observed/Predicted exceedence ratio against confidence level

2.

BIS Red, Yellow and Green Zone frequency against confidence level

3.

Observed/Predicted serial exceedence against confidence level

4.

Mean log likelihood against confidence level

5.

Mean log likelihood against volatility percentile

As already mentioned in section , we have one further variation in the presentation of performance measures - the univariate context based on 10000 prediction-realization pairs and the multivariate context based on 1000 prediction-realization pairs. The two contexts allow us to distinguish between the performance of a model for forecasting a risk factor with the performance of the model for forecasting the portfolio. Knowledge of the relative performance of the model in the univariate and multivariate context highlights an important distinction. It is usually the case that the methodology used for the multivariate computation p_t^(P) as prescribed by a given model can be refined without affecting the univariate computation p_t^(k). (For all the models presented in this paper - barring historical simulation - this means that it is possible to modify the methodology for updating the off diagonal elements of the covariance matrix $\Sigma_t$ without changing the prescription for the diagonal elements.) The relative performance of a given model in the univariate and multivariate context can indicate if the point of weakness for the model lies in the computation of p_t^(P) and can therefore motivate further research in rectifying the point of weakness.