ABA Bank Compliance - July/August 2014

When it’s not possible to apply a theory—and in a lot of cases, it’s not—there should at least be a story or string of consistent examples or anecdotes to justify the methodology or mathematical relationships. “Conceptual soundness” involves thoughts— organized and defendable thoughts; otherwise, you get a lot of math that doesn’t mean anything. We recall a case where an inexperienced developer was attempting to relate operational losses to macroeconomic variables for CCAR. By searching over many such variables (and many creative functions of economic variables), he found a strong correlation between the annual change in the national unemployment rate (it lagged six quarters) to the regional bank’s losses from internal fraud. So, for example, the developer believed that he had compelling evidence to associate the annual change in the national unemployment rate in 2012 (from January 1 to December 31, 2012) with an estimate of what employees would steal in the third quarter of 2014. Given the utter lack of economic theories, stories, or anecdotes to link the two, we took it as compelling evidence that he would have to be watched closely.

Lastly, require developers and vendors to test everything possible, and report the results to users:

1. Adherence to required assumptions;

2. Performance measurement, including back-testing, or comparisons of predictions with actual results; and

3. Sensitivity testing, including the effects of assumptions and the effects of each variable on results, and the interactions between or among variables on results.

Whether built in-house or by a vendor, comprehensive testing, both during development and through time, informs decisions by illustrating the robustness of the approach and its results. It also reduces risk rather than weak performance or degradation in performance over time.

Validation is Model Quality Control

Simply put, validation is a quality control function, and it can be performed cursorily, or earnestly and effectively. If no model within your organization ever fails validation, it’s highly likely that the assessments are cursory and do not the effectively challenge conceptual soundness or implementation.

Many readers, with normal risk appetites, want products like bridges, airplanes, new medications and automobiles. to be tested and proven to be safe before use. So, why not models used to manage activities like lending and trading? Financial model

B IG STO C K

Effective development recognizes and documents environmental uncertainties and limitations, and prevents or discovers avoidable errors through good practices and robust testing.