On the Impossibility of Measuring Model Risk
This week The Economist poked a little fun at the quants:
JPMorgan Chase holds $3 billion of “model-uncertainty reserves” to cover mishaps caused by quants who have been too clever by half. If you can make provisions for bad loans, why not bad maths too?
And in response to this revelation, Francine McKenna wondered how the auditors could have signed-off on the models:
If you need $3 billion of “model reserves” how [does] PwC attest to [the] models underlying valuations, estimates and reserves?
It’s worth noting that these model-uncertainty reserves not only comply with GAAP, but are mandated by it. So in response to Ms. McKenna’s concern, there is in fact a “GAAP for that.”
FAS 157 Par. C16: This Statement clarifies that the measurements should be adjusted for risk, that is, the amount market participants would demand because of the risk (uncertainty) inherent in a particular valuation technique used to measure fair value (such as a pricing model) and/or the risk inherent in the inputs to the valuation technique (a risk premium notion). Accordingly, a measurement (for example, a “mark-to-model” measurement) that does not include an adjustment for risk would not represent a fair value measurement if market participants would include one in pricing the related asset or liability. [Emphasis mine.]
OK, so now we understand why banks have to measure model risk, but how do you do it? Well, if you’re being honest, you don’t. Model risk is impossible to measure. Here’s why.
Pricing Models as Interpolation
First, it’s important to understand what a pricing model is and why they are used. Pricing models are used for two purposes: valuation (that is, to come up with fair values for instruments that do not have directly observed prices—e.g. OTC derivatives) and risk management (that is, to measure the sensitivities of instruments to particular risks for the purpose of managing an overall book). Let’s put aside for now the risk management purpose and focus on the valuation.
The majority of OTC positions are not “marked-to-model” in any meaningful sense of that term. Yes, there are pricing models used to value them, but they’re not the scary kind of marks that skeptics rightly call “mark-to-make-believe.” Most of the time, the pricing model is simply a fancy (and sometimes expensive) tool to interpolate between observed market prices.
Let’s say I have an interest rate swap. I can observe the market prices (rates of various maturities) and as long as my swap is within the range of my observations, then my pricing model is calibrated to market. The only modeling I’ve done is to build a rate curve based on observed inputs and used this curve to discount the contractual cash flows of the swap. This is simply a robust way to interpolate the value of my swap from observed quotes on similar instruments (i.e. other swaps). Now this is obviously a very simple example, but this model-as-interpolation view can also be said of more complicated, but traded, instruments like synthetic index CDOs.
What’s this have to do with model risk? When models are calibrated to observed market prices, and hence where the model is used an interpolation tool, the model risk is (pretty much) already captured by the model. This is true even if the model is “wrong”. If the model calibrates to market, it already reflects the market’s view of the model risk—at least with respect to the observed instruments to which it’s calibrated. I should add that even if you’re interpolating between observed prices, you might have residual model risk—how much residual risk (which could be significant) depends on the granularity of observed data, among other things.
True Mark-to-Model Positions and Why Model Risk is Immeasurable
But wait. If most positions are marked to prices interpolated between observed quotes, what about the rest? Here’s where we get into the true mark-to-model issues, and where model risk is most prevalent. Thankfully, these are easy enough to identify on a balance sheet. They are anything noted as a “level 3” fair value measure—i.e. instruments where the value significantly depends on the model itself and on the unobservable inputs or parameters thereto. Think of a CDO-squared or a bespoke synthetic CDO.
I promised I’d get to the point about the impossibility of measuring model risk, so here it is:
- Model risk is the risk that you’re using the wrong model.
- The space of possible models is infinite. That is, there are an infinite number of models to choose from, including those not yet discovered.
- No one knows what the right model is. If you knew which model was the right one, you’d already be using it. Even if most market participants agree on a model today, they might discover a better model tomorrow, or simply decide that no model is sufficient to assess the risks (this has happened).
- Judgments about the amount of model risk are necessarily qualitative. The best I could hope for would be to say that this model feels more certain than that one.
- Model risk is recursive. Even if I could quantify the level of model risk, what model would I use to measure the impact of that model risk on fair value? Where are the models of model risk? Even if they existed, those model risk models have model risk, no?
That $3B Model-Uncertainty Reserve
If model risk is immeasurable, where did JPMorgan’s $3B come from and what does it mean? As to where it came from, I don’t know the specifics, but I suspect they’ve either: (a) shocked the unobservable model inputs by some arbitrary amount and taken the worst of the lot or (b) run some “shadow models” (i.e. run the same positions through multiple known models) and taken the worst of the lot. Either way, the result is arbitrary. So as to what it means: not much. At best, it gives us some insight into the subjective judgments of JPMorgan management with respect to the quality of their models. So yeah, not much at all.
