Katy Fraser
Senior Security Consultant at i-confidential
“It’s a dangerous business, going out your door. You step into the road, and if you don’t keep your feet, there is no knowing where you might be swept off to.” --The Lord of the Rings
All businesses, big or small, need to measure control performance. This is fundamental to identifying and reducing security risks.
i-confidential’s goal is to help you understand where you are on your metrics journey and help you reach the next stage.
We have worked with many organisations at different points in their security metrics improvement journey. Our approach is designed to cater for every stage, from those at the beginning, to the more established and mature. No matter where you are currently, there are ways we can help.
Let’s have a look at some key points on the metrics journey. Starting positions will differ, but we help ensure all our clients go in the same direction.
Implementing Improvements: Metrics Mountain
We break the journey into a few core stages:
Organisations can start at any point and may even have a mix of deliverables from different stages.
Before we start climbing, and so we know where we are going, we assess an organisation using our Security Metrics Capability Model. It contains 12 core capabilities needed for robust security metrics. This allows us to identify priority areas to focus on and recommend actions to improve security metrics while aligning with any organisational goals.
Starting Out - Embedding Cyber Metrics
This stage is for an organisation that has either no security risk metrics or only a small degree of existing capability.
Below is a common selection of activities at this stage, but not all of these will be used in every delivery.
Key Metrics
When starting out, it is important to have a small baseline set of metrics to gain insight into the highest-risk items. One of the first things we look at is approximately 50 critical metrics needed to measure priority security controls.
If an organisation has a limited budget or resources, these are the ones it must have in place because weaknesses in these controls will expose it to significant security risks. These metrics cover items which should be measured irrespective of business type or industry.
Metric RAGs
Once data is gathered for a metric, a red, amber, or green (RAG) status should be determined based on agreed thresholds. For each metric, RAG thresholds define which results are acceptable 'green' through to unacceptable 'red'. These are aligned with an organisation’s tolerance for different levels of security risk. We ensure appropriate risk-based tolerance levels are assigned.
Metrics Dashboards
We are tool and software agnostic. We will work with existing dashboarding tools to leverage an organisation's investment in this area and ensure security does not live in a silo.
A key principle is making sure the team providing the metrics also produces the right views to meet stakeholder requirements. This includes defining the ‘right’ metrics for the ‘right’ audiences.
Operating Model
Once new metrics are in place, it is important to establish the right support for managing metrics production and presentation. This involves working with teams to ensure people know what they are supposed to do and when they are supposed to do it.
From here, what generally happens is an initial education phase with wider stakeholders so they understand the new data they are viewing. A successful engagement will see a lot of conversation happening between the team producing the metrics and the business about how to address any RAG challenges (reds and ambers).
Growing up – Business-Aware Metrics and Compelling Dashboards
At this stage, an organisation may have a good initial set of metrics and indicators of where controls need improvement. However, the organisation may generate more results and find it difficult to prioritise these and assign owners to address red and amber issues.
Again, not all of these may feature, but are common intermediate steps:
Enhanced Metrics
We will assess what measures are already being used and how effective they are. Some might warrant being retired while news ones should be introduced.
Asset Alignment
This is the key to being able to allocate remediation of red and amber issues to the right owners. Having an associated business criticality for assets also enables better RAG thresholds and prioritisation.
We will review asset data sources to identify where asset information can be used to enhance existing metrics. This provides better risk views and makes remediation efforts more efficient.
Metric Aggregation
To support conversations with different stakeholders, we have developed ways of rolling up metrics into controls, control families, control areas and, eventually, providing an overall security score.
This is important because organisations need to present the results of potentially hundreds of metrics to answer the “So what?” question from multiple stakeholders. The benefit of metric aggregation is it provides each stakeholder with the most relevant view and avoids confusing them with unnecessary detail.
Performance Culture
Ensuring continuous improvement activities support metrics is key to driving a positive performance culture. If this is not in place, then red and amber metrics can be seen by control owners as just a way of getting beaten up by senior management.
We work with clients to ensure that out-of-tolerance metrics are addressed by appropriate treatment strategies and remediation efforts, supported by any necessary processes, escalations, or reviews.
In the early stages of definition, we set realistic tolerances that are achievable, rather than 'Everything must be 100% for green.' -- this is unrealistic.
Maturity – All controls have metrics with automated dashboards for every audience
An organisation may have had a security measurement programme in place for some time. This leads to a greater demand for metrics with improved data quality and more dashboards.
At this stage metrics production is usually manual, and weaknesses in asset inventories will limit metrics data quality or prevent control coverage from being determined. To address these issues, we typically help organisations with the following:
Automated Metrics Collection
At this level of maturity, automation is critical given the number of metrics that need to be produced, the different control and asset data sources that need to be integrated, and the reliance on accuracy.
We have helped our clients achieve this in multiple ways, from removing manual steps involved in data production to building a cyber-data lake. We understand the data, the technology involved, and how to make automation processes run smoothly.
Automated Dashboards
At this stage, the number of different metrics results that are being collected is often huge. This is combined with multiple stakeholder audiences, from control and business owners, to risk and audit areas, and ultimately the board, who need metrics results at different times each month (or even day!).
Without automation, the production of dashboards is labour-intensive. It is also error-prone, which undermines the senior stakeholders who need to lead action on the results. To address this, automated dashboards are required. We help our clients by defining detailed requirements so their development team can build effective dashboards using their favourite tools. We have also created our own cyber metrics dashboard in ServiceNow.
Asset Inventory Improvement
An organisation's metrics are often limited by its asset inventory capability. Metrics that measure a control’s coverage against the asset population it has been implemented to 'control' are fundamental. Without accurate inventories of people, third parties, servers, internet-facing URLs etc., the coverage metrics will be limited and, in some cases, impossible to generate.
Our Security Metrics Capability Model includes advanced asset inventory requirements that we use to help organisations get past this limitation. From a security perspective, it significantly improves metric effectiveness. It also has benefits outside of security, improving all the IT and business processes that rely on accurate asset information.
The Fourth Dimension - Incident-Based Metrics
At this stage, where an organisation has worked with us, they will have metrics in three dimensions for every control. These provide indicators of control weaknesses that could result in a security incident. However, they do not consider incidents or near misses that may have occurred due to these weaknesses.
Achieving this requires an organisation’s incident management processes to record the controls that failed during each incident. We work with incident management teams to define and capture these metrics.
As a concept, metrics are easy – we have all done mathematics at school.
Doing security metrics effectively is a different story, and that is where our experience can bring real value. We will work hard to help you avoid common pitfalls when you move on to that next phase of your metrics journey.