Katy Fraser
Senior Security Consultant at i-confidential
Security metrics identify where you need to address security control weaknesses. A successful metrics programme will have clear roles and responsibilities, owners assigned, and a set of metrics that are clearly understood by the people viewing the results.
But it doesn’t always start that way. In fact, reaching this utopia involves a carefully crafted engagement plan.
So, let’s explore some tactics that go a long way to helping your programme be successful.
The key is stakeholder engagement.
Dispersion of Views, Dispersion of Requirements
It all starts with data. Knowing why you’re producing metrics and what each stakeholder needs to see from the results is incredibly powerful. There are a lot of things to measure, but just because you can measure, doesn’t mean you should.
Why security metrics are important:
Organisations are made up of multiple asset types – servers, applications, humans, third parties, firewall rules, etc.
Security controls are used to limit the exposure of these assets to information risk.
If these controls are not operating effectively then the organisation's exposure to information risk will be high.
Security metrics are essential for measuring control effectiveness and reducing information risk due to weak controls.
Metrics dashboards are required for a variety of stakeholders. They are powerful when they can provide each one with the data they need to meet their particular responsibilities in maintaining control effectiveness and limiting cyber risk.
When we do any kind of engagement, we will look at the data requirements for each stakeholder. The view you need depends on your role:
If you are a remediation owner you need a list of the assets that need to be addressed.
If you are the business owner you need to be able to prioritise remediation activities across your assets.
If you are a risk owner you need to understand the metrics results that measure the controls relating to your risk.
We have a core set of recommended security metrics which feed views of the effectiveness of your security landscape. These must be measured to ensure end-to-end coverage from a security-risk perspective.
We help you define priorities by setting sensible RAG thresholds. Stakeholder views are then built using RAG aggregation.
A Worked Example: What Success Looks Like over Time
A typical engagement journey following the launch of a new metric, data set, or view:
Pre-Launch Engagement
Pre-launch activity involves working with the key owners to ensure they agree the metrics specifications and they understand their responsibilities:
Control Owners |
Accountable for the performance of the controls they own. |
Accountable for agreeing metric SLAs. |
Responsible for agreeing metric definitions and reviewing with policy owners. |
Risk Owners |
Accountable for agreeing metric SLAs. |
Business (Asset) Owners |
Responsible for ensuring control improvement activities are completed for their assets. |
Day One – The Metric And/or View Is Shared with Stakeholders
Pre-launch, we work with the stakeholders above to agree the appropriate metrics and specification for your organisation.
Post launch, wider stakeholders will start to view the data and the results.
Initial Response
Once reviewed, the initial questions often relate to understanding. These may include:
What data is in this metric?
How is the figure calculated?
Are there any exclusions?
Are there any filters or parameters applied to the data?
This is where having a metrics specification in clear, non-technical language is golden, as it quickly allows wider stakeholders to see how things are working.
At this point, some may try to challenge the validity of the data. Some reasons for this are:
The stakeholder was not involved in the specification process. This may be true for some remediation owners depending on what the metric highlights.
The metric is going to shine a negative light on a new area – especially if the stakeholder has responsibility or ownership for that area.
The metric is calculating a figure which has been used in the past but the figure itself is not exactly the same as it was before.
To help overcome these challenges, a data or metric warranty period is important, especially for measuring anything new. Depending on the frequency of data collection, this period can last from a couple of weeks to a couple of months.
During this phase measures are gathered, and their results are validated and checked for accuracy. Any false positives can then be identified and particular behaviours of systems can be checked, e.g. daily fluctuations in the total number of assets recorded. This is especially true where new remediation processes are going to be introduced and people are expected to do something.
This is a great approach for implementing any kind of change. For example, when policy tuning you might put technology into ‘monitor mode’ before you put it into ‘block mode’. It is also an effective way to get stakeholders used to the new approach before it goes live.
Day Two – Learning Done, Getting on the Bus
As stakeholders build trust in the metrics, their mindset starts to shift, and this is where your team will begin to see requests for data or help with creating some additional views.
If you own metrics production, this is a crucial time for managing priorities. You will want to balance the excitement of connecting with stakeholders with the capacity of your team to handle requests.
Remembering to ask why the data needs to be produced and how it will be used can really help streamline and prioritise the more important requests against something that is simply interesting to look at. This should also help appease stakeholders without ruining their new-found love for security measures!
Day 3 – Pupil Becomes the Teacher, the Requests to Help Business Cases Come In
Over time, as metrics become more precise and people really start to understand the data, increasing numbers of controls may be identified as ineffective. This will create a need to write business cases for more resource or investment to deliver remediation which cannot easily be addressed in business-as-usual operations.
This is where your team may receive data requests for views, some of it historic, to support key decision makers as they assess whether they are ready to invest or not.
Overall Result: Positive Impact
After a year, you can start to show trend analysis. Where higher levels of stakeholder engagement exist, you should see positive trends for risk tolerances, and even jumps in improvements, as initiatives from business cases start to materialise.
Managing security-risk measures can be challenging, and that is where we really enjoy helping clients get set up to harvest maximum value from their measurement programmes.
Why not get in touch to find out how we can help your organisation improve security metrics engagement?