Vulnerability Management Metrics: It’s Time to Look Past the Metrics Mirage

When it comes to managing security vulnerabilities, it helps to know your enemy.

That’s why businesses rely on a set of vulnerability management metrics to help quantify how resilient they are and better inform their decisions on how to respond.

The logic is clear: the more you know about the vulnerabilities out there and the effectiveness of your response, the better placed you’ll be to reduce your risk and protect your bottom line.

But as we’ll discover, relying too heavily on quantitative data isn’t necessarily the best way to achieve these goals. But first, let’s discuss the most important metrics.  

The Top 10 Most Common Vulnerability Management Metrics

The truth is, few businesses will have the time and resources to resolve each vulnerability immediately. There are simply too many and new ones are created too often.

In practice, that means businesses have to play a careful balancing act, targeting resources where they’re most needed.

To do that, organizations commonly monitor a range of vulnerability management metrics.

Here are 10 of the most common:  

Mean Time To Remediation (MTTR)

MTRR measures the average time taken to resolve and mitigate cybersecurity vulnerabilities, from the moment they are identified until they are successfully remediated.

This gives you an understanding of how responsive your security setup is.

Mean Time To Detection (MTTD)

MTTD represents the average time it takes to detect vulnerabilities or security flaws from the moment they first occur. Similar to MTRR, this is designed to quantify your responsiveness to new threats.

Average Vulnerability Age

This metric calculates the average length of time that vulnerabilities exist within a computing environment before being remediated.

The older a vulnerability, the higher the risk, so companies aim to keep this metric as low as possible.

Scan Coverage

Scan coverage assesses the extent to which a system or network has been examined for vulnerabilities.

This helps organizations understand how complete their data is. Scan coverage can be expressed as the proportion of assets within the company that are being actively monitored for vulnerabilities.

Patching Rate

The patching rate describes how many patches are applied within a specific period.

Again, this aims to quantify your overall resilience, but it doesn’t account for how long specific vulnerabilities have been in the system or how long they took to detect.

Vulnerability Re-Open Rate

Resolved vulnerabilities occasionally need re-opening due to system configuration changes, new information, or inadequate remediation efforts.

The vulnerability re-open rates describe how frequently this happens. A small amount is normal, but higher rates could indicate inadequate remediation processes.

Number of Exceptions Granted

Organizations regularly decide to waive remediation and accept risk, in order to prioritize resources where they’re most needed.

This metric measures the number of exceptions granted so you can quantify how much accepted risk you’re allowing.

Number of Open Critical Vulnerabilities

This focuses on the number of high-risk vulnerabilities that remain unresolved, offering deeper insight into the current threat landscape and your vulnerability backlog. This metric is often preferred due to its focus on higher-risk vulnerabilities.

Total Risks Mitigated

Total risk mitigated is a crucial measure that shows your IT and upper management teams how effective your vulnerability management programme is.

Your business stakeholders may see the value of your security investment if your total risk remediation is consistently trending higher.

Asset Coverage

The amount of assets that need to be patched is determined by this measure. To find new systems on the network, the vulnerability management programme should have auto-discovery capabilities.

By monitoring this measure, you may see how your environment is changing or whether new assets are being added and maintained by an inventory or ticketing system.

Rating Tool: Common Vulnerability Scoring System (CVSS)

CVSS is a standardized scoring system for evaluating and prioritizing vulnerabilities. It assigns a numerical score based on factors such as exploitability and impact, aiding in vulnerability management and mitigation decisions.

These 10 vulnerability management metrics provide vital insights into the strength, resilience, and responsiveness of your cybersecurity defenses.

But crucially, these metrics all have one thing in common: they rely on quantitative, numerical data.

To some extent, that makes a lot of sense. Businesses rightfully want to make important decisions based on clear data, rather than gut feelings, vibes, or hunches.

The problem is, quantifiable metrics don’t tell the whole story.

The Metrics Mirage: The Limitations of Quantifiable Analysis

It might seem on the surface that an effective vulnerability management program is simply an exercise in monitoring the right metrics and keeping them as low as possible.

But to really understand how effective your risk management strategy is, we need to take a step back and consider what we’re really trying to achieve here.

Ultimately, the goal is to identify risk so you can best target the resources you have to mitigating it. But there are many different ways we can define that risk:

1.   Criticality

Not all assets are equally critical. An attack on a customer-facing application or something holding sensitive financial data will be considerably more costly than something targeting non-essential systems.

2.   Scope

A vulnerability that targets multiple assets in your IT environment is likely to be more dangerous than something that targets only a small part of it.

3.   Availability and Popularity

The more well-known a vulnerability is, and the easier it is for hackers to access, the more likely it is to get exploited. Popular and highly available vulnerabilities therefore pose a more immediate and significant threat.

The risk posed by a handful of vulnerabilities is many orders of magnitude greater than most other vulnerabilities.

Walter Haydock, Cybersecurity Expert

Effective vulnerability management programs require carefully prioritizing the resources you have based on the perceived risk.

Metrics Aren’t Everything

Here’s the big issue with basing your whole vulnerability management program on quantifiable data: Most of the commonly used metrics focus only on scale and don’t account for the risk of a specific vulnerability. 

Simply put, the quickest and most responsive security teams in the world could be effectively useless if they’re not focusing their efforts on the highest-risk vulnerabilities.

This, ultimately, is what the metrics mirage is really about.

Vulnerability management metrics are tools to help you understand your risk profile and identify the best response.

But often, businesses see these metrics as the end rather than the means. And if what you’re measuring only has a loose correlation with actual risk, there’s a good chance you’re leading yourself down the garden path.

You might have a pretty good MTTR or MTTD. But organizations with a much worse MTTR that focus on, for example, the 10% of highest-risk vulnerabilities – they’ll get breached much less frequently.

Walter Haydock, Cybersecurity Expert

Some businesses aim to resolve this issue by incorporating risk metrics like the CVSS score or the number of open critical vulnerabilities.

If you want to better target your resources to higher-risk vulnerabilities, this is a good place to start, but it’s still no silver bullet.

Quantifying the Unquantifiable

Even using CVSS data will only get you so far – because it misses the key context of your business, industry, and specific IT setup.

The same attack on two businesses could have drastically different effects, depending on the kind of systems they’re running, the industry they’re in, and how their environment has been architected.

Of the three risk variables we discussed, two (criticality and scope) can vary drastically from organization to organization.

Ultimately, this means quantifiable metrics alone can only be so effective – since they lack the specific context of how it applies to your company.

This limits how useful they can be in demonstrating the effectiveness of your remediation process and showing where to best target resources to improve it.

So how do you break out of the metrics mirage and get a more balanced picture of your risk profile?

How to Get It Right: The Balanced Scorecard Approach

So what do effective vulnerability management programs really look like? In short, it requires a careful balance of both quantitative and qualitative insights:

The Importance of Qualitative Insights

The quantitative metrics outlined above can help you understand how many vulnerabilities you have, the key weaknesses, and how responsive you are in resolving them.

But a qualitative, subjective analysis is the only way to really understand the potential damage these vulnerabilities could do to your specific IT environment.

Done right, this should offer a deeper understanding of the context and nuances that relate to your business and setup.

The Balanced Scorecard: An Effective Vulnerability Management Program

To manage both quantitative and qualitative insights, security teams often turn to the balanced scorecard method.

This is based on a commonly used framework developed by David Norton and Robert Kaplan in 1992. Though not specific to cybersecurity, it’s often adapted for this context.

It seeks to evaluate and improve the effectiveness of a company’s security measures and initiatives, using a more holistic judgment of the risk and consequences of a vulnerability.

This involves four key perspectives:

1. The financial perspective focuses on the financial impact of cybersecurity activities. It considers the cost of implementing security measures, the return on investment (ROI), and the potential financial losses associated with security breaches. Relevant metrics here could include average vulnerability age, scan coverage, and number of open critical vulnerabilities.
2. The customer perspective considers cybersecurity from the viewpoint of the organization’s customers and stakeholders. It involves assessing how well security measures meet customer expectations and what the potential reputational damage of any breaches might be
3. The internal business perspective concentrates on the internal cybersecurity processes and operations within the organization, measuring the efficiency and effectiveness of response. Relevant metrics here could include MTTD, MTTR, as well as the broader effectiveness of your incident response plan when vulnerabilities are detected.
4. The learning and growth perspective: This perspective focuses on building and enhancing the skills and knowledge of security teams. It assesses the training and development programs for security professionals and their ability to adapt to new vulnerabilities and threats.

By considering the real-world context of your business, IT environment, customers, and internal security team – you can start to build a much more targeted picture of your cybersecurity priorities.

From there, you can make better-informed decisions to manage your risk profile and improve your security program.

When you consider the risk reduction, the risk of the mitigation, or the cost of the mitigation – that allows you to make really fine-grained decisions. Then you can say ‘this isn’t a super-high risk patch, and applying the vulnerability isn’t that big a deal. Maybe we do this during our maintenance window next month.

Walter Haydock, Cybersecurity Expert

Of course, this isn’t the only way to manage risk and plan your cybersecurity strategy – but it’s a method that many businesses have found effective over the years.

Moving on From the Metrics Mirage

When it comes to the balanced scorecard, there’s no one-size-fits-all approach.

There’s no formula that will tell you exactly how risky a particular vulnerability is to your specific organization, IT system, and customers. 

That’s why approaches like the balanced scorecard are so important when you’re building and assessing your overall security program. But of course, an effective vulnerability management program should never be all or nothing.

Keeping track of the metrics we discussed at the top of this blog is still important – and will help inform the decisions you make through methods like the balanced scorecard.

The key is to view the metrics as the means rather than the end: they should inform your objective rather than become the objective themselves.

By taking a more holistic view of your cybersecurity challenges, you can make better-informed and more targeted decisions that ultimately improve your overall security posture.

Strengthen Your Patching Practices With Heimdal®

After overcoming the metrics mirage, it’s time to think about how to patch your servers and machines in an efficient manner. Opting for an automated patching solution, such as our Heimdal® Patch & Asset Management Software will save you precious time and resources, and will of course keep your systems safe.

Here are a few things you can do with our solution:

• Patch operating systems such as Windows, Linux, and macOS, third-party, and even proprietary apps, all in one place;
• Generate software and assets inventories;
• Easily achieve compliance with automatically generated detailed reports (GDPR, UK PSN, HIPAA, PCI-DSS, NIST);
• Automatically conduct vulnerability and risk management processes;
• Close vulnerabilities, mitigate exploits, deploy updates both globally and locally, anytime, from anywhere in the world;
• Customize your solution based to perfectly fit the needs of your organization.

Install and Patch Software. Close Vulnerabilities. Achieve Compliance.

Heimdal® Patch & Asset Management

Remotely and automatically install Windows, Linux and 3rd party patches and manage your software inventory.

• Create policies that meet your exact needs;
• Full compliance and CVE/CVSS audit trail;
• Gain extensive vulnerability intelligence;
• And much more than we can fit in here...

Try it for FREE today 30-day Free Trial. Offer valid only for companies.

Vulnerability Management Metrics: FAQs

1.   What Is The Metric Mirage?

The metric mirage occurs when organizations put too much focus on optimizing cybersecurity metrics, and not enough on actually reducing risk and managing costs.

It’s more effective to see these metrics as informing your wider cybersecurity strategy, rather than being the strategy itself.

2.   What Are Vulnerability Management Metrics?

Vulnerability management metrics help your security team understand their cybersecurity risk posture so they can best prioritize their resources to improving it.

These often detail how effective an organization is at patching and resolving vulnerabilities. It’s important not to rely too heavily on them, however, as not every vulnerability is equally dangerous.

3.   What Are the Most Important Vulnerability Management Metrics?

Some of the most popular vulnerability management metrics include mean time to remediation (MTTR), mean time to detection (MTTD), average vulnerability age, scan coverage, and others.

However, it’s important not to overuse these metrics and to look holistically at the specific risk to your organization.

Danny Mitchell