How To Break The Metrics Mirage in Vulnerability Management

Meet Jeff. He’s the CISO of a mid-sized financial services company – and it’s his job to keep the organization safe from security attacks.

Every week, he checks the graphs and dashboards in his SIEM (security information and event management) platform. He has set clear KPIs for where these metrics should be, and his team have worked hard to optimize them.

He’s just gone over the regular reports and the results are good: over the last year, they’ve massively improved both the speed and responsiveness of vulnerability patching.

Jeff sits back and puts his feet up, satisfied with his work.

So he’s pretty surprised to get a call at 2am a few days later to find that their CRM has been hacked and the personal details of thousands of customers have been leaked.

So what happened?

Why You Need to Move Past The Metrics Mirage?

The scenario we just described is every CISO’s worst nightmare.

As far as attacks go, a data breach of sensitive customer data is about as bad as it gets. And despite all the hard work of his team, Jeff still found himself in the worst possible position. But why?

Here’s the reality: Jeff is a victim of the metrics mirage. The team has worked through a long list of known vulnerabilities at breakneck pace.

But the problem is, they’re not all equally as risky. In prioritizing speed and responsiveness above everything else – they’ve missed one key vulnerability that had the potential to bring down the entire IT infrastructure.

The hard work hasn’t paid off for one simple reason: They focused their efforts in the wrong place.

This is what the metrics mirage really is: when CISOs and their security teams rely too heavily on quantifiable vulnerability management metrics.

Common examples of these could include mean time to remediation (MTTR), mean time to detection (MTTD), and average vulnerability age.

These metrics should play a key role in any cybersecurity strategy. But as Jeff discovered, they’re not the whole story. Here’s why:

Quantity vs. quality:

The majority of vulnerability metrics focus on scale over risk. They simply list the number of known vulnerabilities and quantify how quick or responsive you are in patching them.

But some vulnerabilities are more dangerous than others – and effective cybersecurity is an exercise in assigning the resources you have to those issues that pose the most risk.

Quantifiable metrics alone can’t help you make these decisions effectively.

Lack of specificity:

Risk can also vary between organizations and IT environments. The same vulnerability could be relatively harmless to one business and devastating to others – depending on the technology they use and the data they process.

Common quantifiable metrics struggle to clearly communicate the specific harm a vulnerability can do to your organization. This means they can only be so useful in helping you understand which vulnerabilities to prioritize for patching.

You don’t know what you don’t know:

If you focus only on the technology you actively monitor, you could well be missing a back door that you don’t know about.

Often, security managers simply don’t have visibility over apps, IT systems, or servers that aren’t being actively monitored. This is particularly the case when end users start experimenting with shadow IT.

Quantifiable metrics can only give you insights into assets you know about – so it’s important to cast your net wider.

You need to also worry about the unknowns in your vulnerability management program. You can be really good at burning down a [vulnerability] backlog. But if you missed a certain vulnerability, or it’s not public and can’t be found by a scanner – you can still be attacked by that vector.

Walter Haydock, Cybersecurity Expert

Weighing costs against risk:

The truth is, not every vulnerability needs patching.

There are simply too many available and too few resources available to combat them. Plenty of vulnerabilities won’t do you much harm even if they are exploited – so it’s important to pick your battles and prioritize your resources towards the highest-risk vulnerabilities.

This requires an understanding of the resources you have available, as well as the vulnerabilities that need managing. The standard set of quantifiable metrics can’t provide you with this visibility.

In short, quantifiable metrics are helpful. They provide visibility over known vulnerabilities and the responsiveness of your patching process. But there’s a lot that they simply can’t tell you – so it’s important to understand their limitations and avoid putting all your eggs in one basket.

So, what’s the alternative?

How to Break the Metrics Mirage

The goal of any cybersecurity strategy should be to reduce your risk as much as possible with the available resources you have. It’s not about completing the most patches or having the quickest turnaround. If one vulnerability poses 100 times more risk than everything else – the goal of your strategy should be to identify and patch that first.

But here’s the challenge: there’s no single, clear way to do this. The key to moving beyond the metrics mirage is to integrate multiple strategies, ensuring a layered and comprehensive approach. That should combine quantifiable metrics, together with more subjective judgments of the specific risk they pose to your organization.

There are countless different methods, tactics, and strategies this layered approach could consist of. Here are some of the most important:

Incorporate risk data like CVSS

The common vulnerability scoring system (CVSS) is an industry-recognized standard that defines the relative risk of known vulnerabilities, on a scale of 1-10. It provides an easy way for organizations to pick their battles and identify the most critical patches to prioritize.

It’s by no means perfect.

Most importantly, the level of risk isn’t specific to your business – as the CVSS scores are used by companies of all shapes and sizes. But for organizations like Jeff’s, where no risk prioritization happens at all, CVSS data offers the quickest and easiest way to get started.

If all you do is go to the first website, download the massive [CVSS] spreadsheet and compare that to the vulnerabilities in your network, then pick them off from the top – you’re probably going to be doing better than 50% of the organizations out there.

Walter Haydock, Cybersecurity Expert

Categorize assets based on their importance to the business

Using CVSS data is a good place to start. But if you want to properly target your resources, you’ll need to start considering how dangerous specific vulnerabilities are to your IT environment. That starts with understanding how critical specific assets are.

One common way to do this is to categorize the IT assets within your environment, using a tiered system such as critical, high, medium, and low. Once you’ve done this, you can compare your own risk scores with commonly available data like CVSS to weigh up the specific risk of a vulnerability to your business.

From there, it’s much easier to identify the right response for each vulnerability. The most critical should be dealt with immediately. Their lower-risk counterparts can be ignored entirely or queued up for your next regular maintenance window.

Run behavioral analytics

Behavioral analytics can be a key way for security teams to turn an unknown unknown into a known unknown. By studying the patterns and activities of users, employees, and systems, you can start to identify anomalies and potential red flags you wouldn’t otherwise have known about.

Unusual login times, unauthorized access attempts, or unexpected data transfers can all help draw attention to a potential security threat before it’s exploited. Paying attention to signals like this can be a much more effective way of identifying potential risks than going down a list of publically-known vulnerabilities and hoping you get there before the hackers do.

Create a feedback loop

It’s also important to listen to employees, customers, and peers about what’s working and what needs improving. This can play a huge role in helping you understand unseen risks and validate your current strategies.

Frontline users often spot anomalies before the security team. Though they might not immediately recognize these as security risks – they can with the right training. It’s important to have a full, robust, and continuous discussion with non-technical teams about the threat of security vulnerabilities and the role they can play in helping protect against them.

It’s also hugely helpful to track incident response data. When breaches occur, you should take the time to understand why. Even if you encounter a fairly low-risk attack, understanding how this occurred can help you proactively protect yourself against higher-risk breaches in the future.

Get regular penetration tests

Penetration (pen) tests are one of the most powerful tools in a security team’s arsenal. This is a form of ethical hacking, where cybersecurity experts attempt to find potential risks in your organization – using similar methods to real hackers.

There’s one key reason pen tests are so valuable: hackers know how to think outside the box. They understand – and know how to exploit – the risks created by shadow IT, unscanned IT assets, and poor end-user best practices – which traditional vulnerability management metrics struggle to quantify. Pen testers are trained to think like hackers – which means their insights can give you a much more rounded, qualitative picture of your overall security posture.

By far the best way to run a penetration test is to engage qualified ethical hacking professionals. They will know the best ways to diagnose your overall security posture, which is hugely valuable for organizations that process sensitive customer data.

Quantify the cost of vulnerability management

In an ideal world, a security team would have a clear cost/benefit breakdown of every relevant vulnerability and wider security threat. On one hand, the potential financial loss to the company if exploited. On the other, the cost of remediation.

Wishful thinking, you might say. And to some extent, you’d be right: getting an accurate breakdown down to the dollar is always going to be an exercise in educated guesswork. But gaining a more holistic understanding of cybersecurity costs and potential risks can still give you valuable insights into where to target your resources.

To do that, it’s helpful to create and track your own metrics of cybersecurity costs and risks. These can involve:

• Costs: Security team salaries, IT subscriptions, team training costs, consultation fees, etc.
• Risks: Potential regulatory penalties, reputational risks, cost of downtime, potential lawsuits, potential share impact, etc.

Of course, these are always going to be estimates and should be taken with a (very qualitative) pinch of salt. But they can still be really useful in deciding how to improve your cybersecurity defenses. You can also review these estimates regularly, based on changing costs and data from real-life breaches – to see how they line up with reality.

In a perfect world, determining the ROI of every marginal step, decision, or patch is the gold standard of security.

Walter Haydock, Cybersecurity Expert

Automate Your Vulnerability & Patch Management Processes with Heimdal®

Now that you’ll be moving on from the metrics mirage, consider enhancing the efficiency of your server and machine updates. Choosing an automated updating system like our Heimdal® Patch & Asset Management Software not only conserves valuable time and resources but also ensures your systems remain secure.

Here’s what our software offers:

– Centralized patching for Windows, Linux, macOS, third-party apps, and even proprietary software.
– Compile software and asset records.
– Simplify compliance with in-depth, auto-generated reports (e.g., GDPR, UK PSN, HIPAA, PCI-DSS, NIST).
– Streamline vulnerability and risk assessments.
– Address vulnerabilities, counteract threats, and roll out updates both worldwide or locally, at any moment and from any location.
– Tailor the software to align seamlessly with your organization’s specific needs.

An End to the Metrics Mirage

The truth is, there’s no quick fix when it comes to vulnerability management. And even if there was, the hackers would find a way around it pretty quickly.

The layered approach is the only way to ensure you’re making the best use of your resources to lower your risk of attack and mitigate the scale of financial damage even if the worst does happen.

Of course, quantitative metrics are important. But to truly stay safe, you should constantly question what these metrics can’t tell you. Your goal should be to avoid falling into the same metric mirage trap as Jeff and his security team. Whatever you’re doing, your goal should always be to ruthlessly prioritize your resources toward the vulnerabilities that pose the greatest risk to your specific organization and IT environment.

FAQs

Quantitative vs. qualitative vulnerability management: What’s the difference?

Quantitative vulnerability management metrics often track the speed and responsiveness of your cybersecurity strategy. A more qualitative analysis can help you understand the specific risk of particular vulnerabilities to your IT environment. An effective security strategy should combine insights from both approaches.

What are some of the best tactics for cybersecurity vulnerability management?

Effective vulnerability management requires prioritizing the resources you have towards the most critical cybersecurity risks. Some of the most effective ways to do this involve:

• Incorporate risk data like CVSS;
• Categorize assets based on their importance to the business;
• Run behavioral analytics;
• Create a continuous feedback loop;
• Get regular penetration tests;
• Quantify the costs and risks of vulnerability management.

A combination of these strategies will create a strong, multi-layered security defense.

What are the limitations of quantifiable vulnerability management metrics?

Quantifiable vulnerability management metrics aim to tell you how fast and responsive you are at remediating known cybersecurity vulnerabilities. But most common metrics struggle to help you understand which vulnerabilities pose the highest risk to your specific organization. An effective strategy should therefore combine quantitative metrics with more qualitative insights.

If you want to keep up to date with everything we post, don’t forget to follow us on LinkedIn, Twitter, Facebook, and Youtube for more cybersecurity news and topics.