the justifications for retaining out-of-the-box default settings as
well as the evidence required to support customized settings.
If this sounds technical and abstract, well it is, but that is the
nature of the guidance, and every bank is expected to comply.
It Helps to Be a Bit Philosophical
and as Scientific as Possible
It is worth noting upfront that not all model risk is manageable,
but to see that, one must be a slight bit philosophical. For banks,
models apply economic, financial, mathematical or statistical
theories to uncertain social, economic, and financial phenomena,
conditions, or relationships. At best, these theories and models
are simplified, but they are appropriately representative and
consistent abstractions of real underlying events or conditions.
At worst, they are wrong and misleading, and not much different
than superstition.
In either case, theories are applied because humans don’t
fully understand these social and economic phenomena and
interactions, and there is nothing wrong with admitting it. These
epistemic limitations generally cannot be mitigated and should
not be ignored. Per Donald Rumsfeld’s famous quote during the
Iraq War, “It’s not possible to model the ‘unknown unknowns’
which may or may not exist, but is very useful to consider how
wrong you can be.”
It’s also worth noting that based upon its technical definition
of “measurable” uncertainty, the word, “risk,” in “model risk” is
incorrect. That’s because we cannot reasonably or intelligently
assign probabilities to bad outcomes associated with model usage;
so, there’s no number you can put on it. Given the measurement
and epistemic limits, the guidance defines “model risk” as the
“potential for adverse consequences from model usage”, and not
the probability of adverse consequences or the expected value of
adverse consequences. It’s also why if someone tells you that they
can assign a number or dollar value to your firm’s model risk,
the process that generates that number will likely fail a rigorous
validation because you don’t know all of the ways that you, or
the model, can be wrong. So, managing models will continue to
be uncomfortable and uncertain.
Does this mean that all is hopeless? No, but it does mean that a
lot can go wrong, and whether the model is homegrown or store
bought, someone at your bank should know what is represents and
what it doesn’t, and make sure that it is only used appropriately.
Management
