The Shortcuts Trap – Key Indicators Under Fire
How common key indicator errors can increase risk and reduce resiliency.
This is the second in a series of four blogs about the ways in which common shortcuts can undermine overall operational risk management success within organizations. You can access the first blog here Risk and Control Self Assessments.
Everyone loves metrics, and particularly key indicators such as key performance indicators (KPIs), key risk indicators (KRIs), and key control indicators (KCIs). In a perfect world, you’ll be using risk management software and each data point should provide the organization with importance intelligence. However, most operational risk teams wind up burdened with hundreds of KPIs, KRIs and KCIs, and then find they cannot see the forest for the trees. At some point, most operational risk teams will find themselves needing to sit down and shake up their stock of key indicators. And that’s where the temptation to use shortcuts can arise. Common shortcut mistakes include:
- Deeming the
KPIs to be KRIs – Often the business will have a list of KPIs that they
track for operational and financial purposes. There can be a temptation to just
adopt all of these KPIs as KRIs, and call it a day. However, KRIs are just what
they say they are – and so they need to be identified with a specific risk. For
example, a KPI that everyone looks at is profitability, and this is an
excellent KPI. However, it tells the operational risk team almost nothing specific about
the way the firm is managing any of its operational risks or controls. Operational risk
teams should challenge the use of KPIs as KRIs by asking which specific risk or
control a KPI relates to. - Regarding
KCIs as KRIs – Once again, it’s important to distinguish between what is a
KRI and what is a KCI, and to treat them differently. KCIs are vitally
important for an operational risk program – they measure the strength of the
control environment. Treating them as KRIs leads to failing to consider the
control environment on its own. KCIs should be paired with a relevant KRI, for
maximum operational risk framework robustness. As well, they should be tested regularly
through the control testing that the operational risk team performs. - Using the
data to set the thresholds – Although using the data might seem to be a
straight-forward way to set the thresholds of key indicators, it is deeply
problematic. For example, some operational risk teams will look at the data for a key
indicator and say to themselves, “we’re generally where we want to be, but
sometimes we’re not so that must be amber, and remember that really big one two
years ago, that must be red.” However, this thinking doesn’t really capture
what is happening on the ground within the business. In fact, the process the
key indicator is capturing may be experiencing problems, so what looks like
“green” from the data might actually be amber or red for the team. It’s
important to have a conversation with the business for each KRI and KCI being
used, to set the thresholds according to what is right for the business. - Believing
all KRIs are early warning signals – Certainly some KRIs are early warning
signals. In particular, those KRIs which relate to the likelihood of a risk
happening can fall into this category. However, those KRIs which are indicators
of the size of the impact of a risk event are not early warning indicators –
they simply indicate the possible magnitude of the impact. As a result, impact
KRIs are more of a lagging indicator. An example of a good operationa risk early
warning signal is a staff turnover metric. Staff leaving a settlements
processing team leads to a loss of operational history as well as the need to
train new staff. Existing staff can become demoralized and not train new team
members properly – and so risk can rise. Therefore, indicators of likelihood
are good early warning signals because the right ones can provide an insight
that a risk is more likely to happen. However, in this example, the number of
settlement fails would be an impact indicator – it is an indicator of the
impact of losing key staff – and not providing an early warning of anything. - Tossing
out indicators that aren’t predictive – Most financial services
organizations have far too many key indicators – they tend to accumulate over
time. However, because there is a commonly-held belief that all key indicators
are early warning signals, some operationa risk teams have simply put other types of
key indicators in the bin. This is a big mistake. For example, KRIs that
suggest the potential impact of a risk event, as well as KCIs are important for
understanding operational resilience. Instead, operationa risk teams seeking to reduce
the number of key indicators should look towards industry best practice and indicator
effectiveness as benchmarks.
In short, it’s important to take a more thoughtful approach
to restructuring an ecosystem of key indicators. Taking shortcuts can reduce
the ability of certain KRIs to be predictive, and damage the firms’
understanding of its operational resiliency, through other KRIs and KCIs, as
well.