Ian Harragan
Director at i-confidential
We all understand the importance of measurement for anything we do in business. And with all the clever IT systems we use today it must be easy to measure everything we need.
Or is it?
If only the world we work in was so accommodating! At i-confidential, we have helped a variety of organisations over the past decade with the challenge of effectively measuring key areas of their cyber security. The improvements they made allowed them to be more in control and take the right actions when required.
Across these engagements, four common pitfalls emerged:
Knowing Where the Information Comes From
We have often found that the security packs CISOs use to provide the board and other senior stakeholders with updates are at best ‘flaky’. It has not been possible to consistently reproduce the desired information.
We believe that strong governance, supported by business areas that understand the role they play, is a key foundation in establishing effective metrics that are consistent and repeatable. The source feeds required to deliver this can vary enormously, and even if you know exactly what you require, getting hold of it in the right format, at the right time, can itself be a problem.
Helping those responsible for providing the information to understand both the security context, together with when it will be required, are also important steps.
Telling Half a Story
It is easy to take whatever measurement data is readily available and replay this as the basis of what is happening more generally. Organisations commonly measure what they can, but not necessarily what they should.
We have seen instances where the data captured only represents a small percentage of an organisation’s asset estate, but this is not made clear when being presented in security updates, suggesting to the reader that this is the ‘full story’.
Coverage is a key dimension in the reporting of security metrics. It should be clear to anyone reviewing the information provided what proportion of the estate is included. Sometimes you will not know the entire population of a particular asset, and you might always struggle to. By making clear the percentage you are measuring everyone can understand what they are being told. Full disclosure will also aid any discussion about what action is required to improve existing coverage.
Thinking the Answer Starts with a Tool
Having bought a software tool, normally in the GRC arena (governance, risk, and compliance), organisations quickly realise they face the challenge of what to populate it with. Where are the data sources? What information do they require to see from the tool?
Our experience suggests there is a natural journey to play out, which we refer to as climbing metrics mountain. As your metrics approach matures, you will learn what is right for you, and when the time comes to improve your tooling and automation you should have developed a clear set of requirements that enable you to move forward successfully.
Thinking That Everyone Understands
For those of us that work in IT and security it is often easy to forget that not everyone in the business will be familiar with some of the language we use. Asking them to understand a large volume of security information that they also have to partially translate is a big ask.
For security metrics to be relevant and do their job they need to be presented using a format and language that can be readily absorbed across an organisation. We also believe in combining or ‘aggregating’ measures to aid in effectively summarising large, disparate data sets. Both of these strategies enable security leaders to tell more effective stories and gain the strong support they need to take action on improvements.
Unfortunately, we live in a world where cyber security risks continue to increase. Being able to collect and present data in the form of effective security metrics is an essential management practice. It can enable an entire organisation to see what is and is not working, and where improvement is required.
Do not delay either starting your journey or taking the next step.