How to read a security product evaluation

Detection results need context. A buyer also needs to understand visibility, false alarms, analyst workload and the supplier's own security.

Cybersecurity3 min read

Hardware authentication key. Illustration.

A detection rate without its test conditions is hard to interpret. It may describe a particular threat set, an unusually rich source of telemetry or a configuration maintained by specialists. A security buyer needs to know whether that performance can be sustained in the organisation's own environment.

An evaluation needs to account for the protection offered and the work required to operate the product. That includes what the product can see, the decisions an alert supports and the additional access or sensitive data the supplier will receive.

Begin with the limits of visibility

Identify the asset, attacker behaviour and expected response. Detecting unusual account activity differs from blocking an unauthorised action; investigation and recovery add further requirements. A product comparison should keep these functions separate so a broad promise of protection does not obscure the actual purchase.

Rows of black server cabinets with organized cabling inside CERN's computing facility. — Server racks at CERN, Switzerland, 2009.Florian Hirzinger (opens in a new tab) · CC BY-SA 3.0 (opens in a new tab)

List the prerequisites for the claimed coverage. The product may need a log source, installed agent, access to decrypted traffic or permission to alter another system. Where a prerequisite is unavailable, establish what becomes invisible. The buyer's architecture determines whether a strong result elsewhere is relevant here.

Read the denominator

An evaluation should identify the release, configuration, data and scoring rules. Both ordinary activity and attacks must resemble the conditions the buyer expects to encounter. Legitimate behaviour that resembles an attack is particularly useful for testing the quality of an alert stream.

The frequency of real attacks in the test affects how useful alerts appear. A dataset dominated by malicious events can produce an apparently persuasive result while giving little insight into an analyst's normal workload. Ask how many relevant events were missed, how often benign activity triggered an alert and how ambiguous cases were labelled.

Read an independent assessment within the limits stated in its report. Check the date, deployment and version. A result for one configuration does not establish performance for all configurations or subsequent releases.

Follow one alert through the organisation

During a trial, take a finding through triage, investigation, a response decision and closure. Record where the supplier intervenes and where a specialist is required. An accurate alert can still be difficult to act on if supporting records are missing, permissions are unclear or the operator cannot reverse a response action.

Vulnerability response needs similar attention to context. CISA's Known Exploited Vulnerabilities catalog records vulnerabilities with evidence of exploitation in the wild and supports prioritisation. It does not provide a complete ranking of an organisation's risk. Exposure, mitigations and the consequences of interruption remain part of the decision.

Black YubiKey USB security token with a gold touch contact and exposed USB contacts on pale fabric. — Yubico YubiKey authentication token, 2019.Tony Webster (opens in a new tab) · CC BY 2.0 (opens in a new tab)

Inspect the privileged supplier

Security tools may collect sensitive telemetry and hold extensive privileges. Examine administrator authentication, separation between customers, default settings and data retention. Establish which information leaves the environment and how access can be withdrawn. CISA's Secure by Design guidance places customer security within the manufacturer's responsibility, including the security of the default experience.

Three DE-CIX equipment cabinets with Brocade networking hardware and yellow cables visible through their mesh doors. — DE-CIX network equipment, 2011.Stefan Funke (opens in a new tab) · CC BY-SA 2.0 (opens in a new tab)

NIST's Secure Software Development Framework offers a vocabulary for assessing development and vulnerability response. Ask for evidence of maintained releases, a vulnerability reporting channel and a process for delivering urgent fixes. A component inventory can help identify dependencies, but it cannot establish by itself whether a particular vulnerability is exploitable in the customer's installation.

Use frameworks to locate the remaining work

The NIST Cybersecurity Framework 2.0 describes outcomes without prescribing how organisations must achieve them. Mapping a product to those outcomes can clarify its intended contribution. A mapping is neither product certification nor evidence that the buyer has achieved the outcome.

Document the controls still needed, who owns them and how the product will be reassessed after material changes. This gives the operating team a record it can use after the purchase, when a feature comparison alone would leave its responsibilities unclear.