Some of the more prevalent risks are:
■ ■ ■ Inaccurate/incomplete data. Poor quality inputs can lead to inaccurate and possibly inappropriate outputs or conclusions.
■ ■ ■ Business Unit Input. Creating models without the benefit of business
unit input can lead to erroneous business models and their associated
outputs. Erroneous outputs fall under two general types:
1. A transaction or practice is identified as posing heightened potential risk,
when in fact it does not. An example is when numerous transactions related
to a specific product, service or customer type are identified as potentially
suspicious. With limited resources available to investigate each transaction, the bank may make a decision to avoid the origin of the risk (e.g. the
product, service or customer), thus essentially engaging in “de-risking,”
2. A transaction or practice which poses heightened risk and is not flagged
at all. One example is when an underlying pattern of lending discrimination is not identified because of the model formulation, data
accuracy or missing data.
■ ■ ■ Understand your model so that it reflects how you do business.
Sometimes compliance officers and their staff rely on outside consultants to
build their compliance models, then they neglect to understand the meaning
of the inputs and outputs of the model in relation to their business unit.
■ ■ ■ Model results should be consistent with policies, procedures,
and actual practices. Another risk associated with models is not incorporating the actions indicated by the output of the models in the bank’s
policies and procedures. The model results may mean that policies and
procedures need to be changed or remuneration may be required.
Build vs. Buy Decision
Whether you decide to build your own models or buy them from a vendor,
there are some key variables to consider. If you are considering buying your
models, think about the following:
■ ■ ■ Costs: Costs will vary but include annual fees, costs for training, special
systems add-ons or conversions if any as well as the expense for new
hires if needed.
■ ■ ■ Software: There may be a charge for software updates. In addition,
there could be hourly charges to assist bank personnel in the installation
of the updates.
■ ■ ■ Usage: The model may require special expertise such as statistical knowledge and on-going training for experts. In addition, if the models are
used infrequently there is often a learning curve the model builder will
experience such as re-learning the software syntax for making changes
to the model. Finally, some models may be made available to be shared
by multiple banks, e.g., underwriting and pricing models. Using shared
models often results in the individual bank-unique business models being
overlooked in the generalized model, and as a result, possibly producing
inaccurate and perhaps misleading results.
■ ■ ■ Vendor: Vendor reputation and capacity to support the product is criti-
cal. For example, does the vendor have support resources, and are they
readily available to support use of the model? What is the feedback from
other institutions about the product and vendor support? Waiting one or
two days for a call back from a vendor about your question is very disap-
pointing. Does your staff understand the special methodologies such as
the analysis process and proxies, as well as the meaning of the output?
■ ■ ■ Information Security: Almost certainly personally-identifiable information and sensitive company information will be shared with the vendor as
part of any model. The information must be adequately protected (and
ultimately returned/destroyed) and the vendor should be restricted from
using the information for any other purpose. The institution must also
need to understand the use of other data that may be part of the model.
If, on the other hand, the choice is to build the models in-house, the following
key considerations are relevant:
■ ■ ■ Does the bank possess model building expertise and is bank staff sufficiently
available to execute the build and roll out to bank in a reasonable period?
Model building is part science and part art. The exact amount of time
or resources that it takes to build a model is often difficult to accurately
estimate. The last thing you want to do is to short-change the time and
effort involved in building a model.
■ ■ ■ Are controls in place to ensure the integrity of the data supporting the
model, e.g., availability and accuracy of data?
■ ■ ■ Does the bank have the expertise to make the best decisions to ensure a
successful model building effort, e.g., types of computers and appropriate software?
■ ■ ■ Does the bank staff have relationships with regulators and other external
model builders to assist them with the inevitable questions that arise in
the model building exercise?
Supervisory Guidance: With the increasing use of models, banking agencies have issued guidance to outline expectations for sound risk management
involving these systems. For example, in 2011 the Board of Governors of the
Federal Reserve System (FRB) and Office of the Comptroller of the Currency (OCC) issued Supervisory Guidance on Model Risk Management
(Joint Guidance). 1 More recently in 2016, the FDIC proposed supplemental
Third-Party Lending guidance (FDIC Proposal) that incorporated Model
Risk as a key supervisory risk category. 2
The Joint Guidance encourages model validation to ensure the soundness
of the system and describes it as “…the set of processes and activities intended
to verify that models are performing as expected, in line with their design objectives and business uses.” The Joint Guidance identifies potential limitations
and assumptions to be considered and assesses their possible impact, e.g., “…if
credit risk models do not incorporate underwriting changes in a timely manner, flawed and costly business decisions could be made before deterioration
in model performance becomes apparent.” It also advises that validations “be
performed by staff with appropriate incentives, competence, and influence.”
The FDIC Proposal underscores the importance of independent verification of third-party models both prior to and after implementation. It also
suggests that third party lending models may be particularly subject to fair
lending risk given the limited history of some models in the marketplace.
Validation Timing: There are three phases in model use in which model
validation is important. First, model validation needs to occur before the
model is put into use. This validation is typically performed when the model
is being developed and involves the statistician, the business line, and the