exist, as well as environmental volatility—both at present and
anticipated through time.
Recall that the definition of model risk is the potential for
adverse consequences from use. Size or importance represents the
magnitude of those consequences, and uncertainty or uneasiness
captures their potential to occur. Combining those factors allows
the inventory to be classified appropriately, and validated in order
from most to least riskiest, with ongoing oversight varying with
class or rank, too.
In our experience, the marginal benefit of an extremely detailed classification scheme, with a large number of criteria,
is small; the rankings won’t change much. Moreover, regardless of a bank’s business mix, we’d argue that fewer than 25%
of its models generate a substantial majority of its model
risk, and knowledgeable executives, familiar with the bank’s
operations would have similar rankings with no more guidance than “sort by a combination of importance and uneasiness.” Effectively validate those models in the highest class or
rank, and you will have mitigated much of the controllable
risk. More precisely, those controllable risks will be mitigated
when the findings from those validations are remedied. 3
Informed Use in Decisions and Reports
In their policies, most banks define the owner to be the model’s
single point of contact, and the person responsible for all of its
aspects, except its actual use. 4 Regardless, users of the output
bear that ultimate responsibility. At most banks, large and small,
executives are users of the output of firm’s critical models—not
owners, developers, processors, or validators. And users are responsible for understanding what they are considering. (That
seems almost silly to write.)
This does not mean that the user should be able to solve
advanced math problems, interpret computer code, or perform
statistical tests to verify assumptions and relationships, but the user
should have a sufficient intuitive understanding of the real world
problem and the model to understand differences between the
two, including design weaknesses and limitations. In addition, the
user should be aware of key assumptions and their implications;
the relationships among factors and variables of interest, including
the sensitivity of results to differences in and across inputs; and
key validation findings and their implications. Awareness and
understanding of alternative theories and methodologies further
strengthens the user’s ability to compare and interpret results.
If this information isn’t presented to the user, he or she should
request the model’s documentation and developmental evidence. If
it is absent, then request the validation report, which should describe
the above items along with related findings noting the absence.
In social and economic settings, like banking, the use of models
to inform decisions and reports will always entail risk. That risk is
exacerbated when the model is conceptually unsound or poorly
implemented, or when it is otherwise sound but misapplied.
The controllable aspects of model risk are most efficiently
managed during development, i.e., during design and construc-
tion, when errors can be avoided through good practices. When
that’s not possible, a strong, knowledgeable validation team can
ensure model quality by finding problems and requiring their
remediation. However, regardless of the strength of development
and validation activities, users who consider model output for
decisions and reports, are responsible for understanding its ef-
ficacy, including its weaknesses and situations in which the model
is likely to be unrepresentative or unreliable. In all three areas,
context, via experience or relevant education, combined with
rigorous thought and calculation, help mitigate model risk. ■
Endnotes
1 In that sense, validation is like getting a second opinion from a different
physician.
2 Despite the sales pitch, vendor models have weaknesses, limitations, and
uncertainties, too. Be sure they are documented.
3 Note that unvalidated models can be partially controlled through ongoing
monitoring and other compensating controls.
4 The owner should know every user and each intended use.
ABOUT THE AUTHOR
ANDY SPERO is Executive Vice President at Regions Bank,
responsible for centralized model development and governance, as
well as significant analytical tools and stress testing governance.
He is the former Head of Model Risk Management at Regions, and
he served in a similar role at another CCAR bank. He has consulted
in market and credit risk at two other CCAR banks, and he has
been a business school professor at Washington University and the
University of Minnesota.
Andy has a PhD from Carnegie-Mellon University (GSIA, now
the Tepper School,) and an MBA from the University of Pittsburgh.
Andy can be reached at andy@spero.co.
Financial model usage generally does not have immediate
and observable life or death consequences, but an unsound or
badly constructed financial model can create billions of dollars of direct
and collateral damage and can severely harm the lives of customers,
employees, shareholders, and their dependents.
