Minimizing Risk Efficiently through
Sound Development or Purchase:
Or, The Best Defense is a Good Offense
To date, it seems that validation has been emphasized through
heavy scrutiny, and this makes sense. Banks, and their supervisors, require an independent and rigorous assessment of where
risks exist, and which of those risks can be mitigated. An effective
validation group can assess and report unavoidable, environmental
uncertainties and limitations, as well as avoidable conceptual errors
and implementation mistakes made during design and construction
(or during customization for a vendor model). Strong validation
by itself, can effectively reduce risks but can’t do it very efficiently.
Effective development recognizes and documents environmental
uncertainties and limitations, and prevents or discovers avoidable
errors through good practices and robust testing. This leads to
faster, cheaper validations as well as a reduction in rework and
other corrective actions by developers, owners, or users. Whether
performed in-house by a dedicated team or peer developers or
by external consultants, validation is expensive, and the time and
expense increase dramatically if the model’s concepts, tests, and
implementation are not fully and transparently documented.
Similarly, if failure-related costs associated with redevelop
and the negative consequences of wrong decisions, are high,
then seek a higher level of prevention activities to avoid errors.
For important models applied to high-volume or large-dollar
transactions, crucial estimates on financial statements, or loss
forecasts for capital planning, banks should have something close
to a zero-defect development policy. Given epistemic issues and
other constraints, that doesn’t mean that the resulting model will
be perfect, or even good, but it does mean that it will be defendable, and that is true regardless of where the model was built.
Models should be conceptually sound and well-implemented.
Would anyone argue the opposite? Both characteristics are determined during development when designs are considered,
selected, and constructed. Development can be divided into the
following steps:
1. Determine the intended use or uses—both short-term and
long-term.
2. Consider the ideal design by carefully describing the environment or phenomenon of interest, and by investigating applicable theories or experiences. For example, to forecast credit
losses, specify all risk factors and how they may differentially
affect segments of a loan portfolio.
3. Determine the environmental and organizational constraints
that prevent the ideal design from being implemented. These
may be epistemic, data, or resource related.
4. Determine the best constrained solution by considering applicable theories and methodologies.
5. Construct the implementation and any alternative specifications that may be used as benchmarks.
6. Perform robust testing and sensitivity analysis. Given the
compromises and trade-offs in the constrained solution,
thoroughly test the selected theory or method’s required
assumptions, to ensure they are sound, as well as the imple-
mentation to identify and document relationships among
variables and weaknesses, limitations, and uncertainties in the
chosen design. In addition, try to estimate the implications of
the shortcomings, including their direction and magnitude.
(This is often difficult given the epistemic risk.)
7. Determine how the model’s output will be reported along with
the weaknesses, limitations, and uncertainties and whether
those shortcomings require overlays or adjustments to the
calculated value.
8. Determine a long-term plan to mitigate (to the extent possible)
those weaknesses, limitations and uncertainties, primarily
by relaxing the constraints identified above (e.g., acquire
additional data).
Bad development practices treat each step less rigorously than
necessary. Really bad development practices compress steps 2
through 4 into one. Search Google for a recipe or algorithm,
and ignore steps 6 through 8, also. Bad model purchasing and
vendor management practices are no different: a vendor says it
has a solution, and it is bought and used without qualification
or scrutiny to determine if it will perform in reliable and representationally faithful manner.
If no weaknesses, limitations, or uncertainties have been identified during development and communicated to users and oversight
areas, then users should be very suspicious. It’s likely that your
bank has feral developers who have gone without challenge or
supervision for too long. Without such challenge, a developer may
act like the ancient Cypriot, Pygmalion, and fall in love with his
or her creation, believing it is beautiful and a perfect description
of the environment, phenomenon, or condition. This mistake
is the Fallacy of Reification, or the belief the real world behaves
just like your math. 1
So, how can model risk be managed in development? By
managing the developers. Require them to follow well specified
procedures and document their efforts. With a few exceptions,
the advice below can be applied to vendor models, too. Developers and vendors should communicate the advantages and
disadvantages of the selected theory, compared to alternatives,
as well as the strengths and weaknesses of the selected theory,
methodology, and implementation. This information should
not be communicated only through model documentation, but
also directly to users so they can assign an appropriate level of
confidence to the results. 2
If developers and vendors can’t communicate these assessments, then there is a good chance they haven’t considered different theories or alternative methodologies. This is common
among inexperienced developers, who aren’t sure how to weigh
or trade-off different characteristics and aren’t considering the
problem they are trying to solve. Procedures that require developers to write models as math problems, rather than the results
of a statistical exercise, tend to focus attention and reduce the
reliance of “data mining,” in which a developer searches for high
statistical measures of relationships among variables or functions
of variables. Given the freedom to specify different transformations and combinations of those variables, it’s not difficult to
find spurious relationships, but spurious results, by definition,
are only coincidental and misleading.
