Through the Lens of a Security Team: The 2023 MITRE Engenuity ATT&CK, SE Labs, AV-Test, and AV-Comparatives Evaluations

By Trellix · September 20, 2023
This blog was written by Justin Buchanan and Fred House

Fundamentally, cybersecurity is about focusing your efforts to generate the best return for your organization. You want to detect what matters without falling down the rabbit hole of endless alerts. You want the confidence that comes with knowing your crack security team knows the ins and outs of your organization and its unique indicators of attack, and that your security solutions work so efficiently in the background, surfacing only what matters, that the noise of false positives is removed and they have the time they need to devote to response and threat hunting.

That’s where Trellix comes in and why we take the approach we do to technology evaluations. We participate in a range of third-party testing to provide a diverse and informed perspective, helping us build solutions with both efficacy and efficiency in mind. It’s not just what we detect that matters but also how we maintain system performance and decrease the noise analysts sift through daily – and how well we do that across the extended enterprise and a range of operating systems.

That’s also why we recommend organizations review these evaluations in concert to identify the right solution for their unique and multi-faceted requirements.

2023 MITRE Engenuity ATT&CK® Enterprise Evaluation: Turla

MITRE Engenuity ATT&CK® Evaluations are built on a knowledge base of real-world tactics and techniques threat actors use to breach organizations and exfiltrate data. The annual testing brings together cybersecurity providers and MITRE Engenuity experts to evaluate solutions within the context of this framework. This year the Enterprise ATT&CK® Evaluation is focused on emulating the Turla Russia-based threat group that uses targeted intrusions and innovative stealth against organizations in more than 45 countries.

Trellix turns to the MITRE Evaluations to improve our offerings and ensure we direct investment to areas that will most advance our mission of protecting customers. The Evaluations have led to many accomplishments and innovations in critical capabilities:

• We further refined the already tight integration between our endpoint protection, integrated endpoint firewall, and Trellix Global Threat Intelligence (GTI) reputation service. This unique combination of technologies creates firewall and endpoint protection that goes beyond simple allow and block lists and leverages our dynamic reputation system that adapts continuously based on what is happening in the wild. In the context of this year’s evaluation, Trellix Endpoint Security Firewall leveraged GTI to identify an unverified attacker IP address, preventing the attacker from deploying malware into the environment at the earliest possible phase of the evaluation.
• As one of the few vendors with comprehensive coverage for email-based threats, we leveraged our email security solution to successfully detect a simulated phishing campaign. Email protection is a critical security control for every security program, with 45% of ransomware attacks using email as the attack vector. Trellix provides not only an integrated portfolio of solutions but also industry-leading solutions in each area. SE Labs recently awarded Trellix Email Security a AAA designation for achieving a 100% Total Accuracy Rating.
• The most repeated technique in this year’s evaluation was Ingress Tool Transfer (tested 13 times), representative of the fact that introducing software into an environment is a fundamental aspect of every sophisticated attacker’s methodology and critical to catch. We are extremely proud of Trellix Endpoint Security’s ability to detect the introduction of files from remote systems through the inspection of network traffic, which led to detection and visibility of all related techniques in the evaluation.
• Trellix also demonstrated strength in detection for techniques observed early in the attack lifecycle, including Initial Access, Credential Access, Defense Evasion, and Discovery, which in the real world translates to a decreased burden on the security team. Particular areas of success we are most proud of include our enhanced protection of LSASS to prevent credential dumping, our continuously improving ability to detect and prevent process injection as a defense evasion technique, and our ability to leverage weak signals captured by our EDR technologies to identify techniques associated with discovery.
• As seen in prior years, Trellix performed exceedingly well in lateral movement detection across both Windows and Linux operating systems. Critical to limiting the blast radius of an attack, lateral movement is the focus of innovative solutions such as Trellix Forensics Logon Tracker. This and other innovations are serving as foundations for components we are building into our XDR.

Just as the MITRE Evaluations drive innovation at Trellix, we are proud of the role that we play in encouraging the innovation of the MITRE methodology. As the complexity of environments expands, we anticipate that future evaluations will evolve beyond an endpoint-only detection focus, enabling XDR vendors like Trellix to employ and continually evolve a range of capabilities for integrated, multi-layered defense, including network traffic introspection, data loss prevention, and email protection.

You can view the complete results for the 2023 MITRE Engenuity ATT&CK® Enterprise Evaluation: Turla on the ATT&CK Evaluations home page.

SE Labs Endpoint Security (EPS): Enterprise 2023 Q2

While ATT&CK evaluations assess vendors on their ability to prevent and detect the tactics, techniques, and procedures (TTPs) of a specific threat group or groups, other testing companies evaluate a vendor’s ability to detect and defend against a more general set of attack activity that reflects the day-to-day security challenges of the security team while also measuring false positive rates, the performance impact on everyday activities, and efficacy on different platforms such as macOS.

Building everything with the security team in mind means continually evaluating how effectively we enable them to be more effective and efficient. While blocking nothing would of course be far too inviting to any adversary, overwhelming security teams with alerts or blocking everything detected would bring an organization to a grinding halt. In the Endpoint Security: Enterprise 2023 Q2 report SE Labs awarded Trellix Endpoint Security a AAA rating, with Trellix achieving a perfect record in both Protection Accuracy, our ability to block malicious activity, and Legitimacy Accuracy, our ability to limit false positives by correctly identifying legitimate activity that could be misidentified as malicious.

These evaluations are performed while emulating common attacks and are tested in a series of stages. For example, a common test would be to open an infected email and activate the attached file that would then download and attempt to run malicious code from the internet in the evaluation lab. It’s designed to ensure the vendor effectively permits appropriate email attachments and internet activity while blocking nefarious behavior. Choosing a solution with low false positives makes the security team much more efficient.

Read more about the SE Labs Endpoint Security: Enterprise 2023 Q2 on the Trellix blog.

AV-TEST Windows 10: June 2023

It’s also important to ensure protection is not coming at the cost of the performance of end-user systems and services. False positives can be as disruptive to these systems as malicious attacks. When performance is a concern, consult testing companies that test solutions under the load of everyday activity.

Trellix Endpoint Security achieved Top Product designation in the AV-TEST Windows 10: June 2023 evaluation with perfect scores across protection, performance, and usability.

For the protection portion of the test, AV-TEST evaluates the vendor’s ability to fend off the latest attacks such as zero-day malware, drive-by attacks, downloads from websites, and attacks via infected emails. A portion of the June 2023 evaluation simulated a real-world targeted attack leveraging web-based, email-based, and file-based zero-day malware attacks. The performance test evaluates the ramifications the vendor’s software has on the performance of the systems. To emulate real-world scenarios during the testing, typical operations for daily work are carried out on the laboratory computers while the impact is measured and analyzed. To achieve a perfect score in usability the evaluated vendor must produce no false positives.

View the results for the AV-TEST evaluation on the Windows 10: June 2023 report page.

AV-Comparatives Mac Security Test & Review 2023 and Business Security Test 2023 (March – June)

Measuring detections, false positive rates, and the performance impact on everyday activities is imperative to determining how a solution will advance the goals of your security team. A final consideration is how well the solution protects across the many platforms your team is responsible for securing. Many organizations, for example, are now seeing an influx of macOS systems in their environment in addition to Windows endpoints.

In the AV-Comparatives Mac Security Test & Review 2023 Trellix achieved exceptional results, blocking 99.7% of Mac malware samples and 99% of potentially unwanted applications (PUAs). The reduction of PUAs in the environment also helps improve the performance of endpoints by ensuring that unnecessary resource hogs are not bogging down our systems, or worse, collecting system data that does not need to be shared.

Additionally, AV-Comparatives released their Business Security Test (March – June 2023), where Trellix Endpoint Security demonstrated leading protection results and very low false positives, receiving the AV-Comparatives Approved Business Security Product Award for July 2023. Trellix achieved the highest scores for malware protection with zero false positives on common business software.

View the results of the AV-Comparatives evaluations on the Business Security Test 2023 (March – June) and Mac Security Test & Review 2023 pages.

Summary

Our continuous participation in a wide variety of evaluations is a testament to our commitment to perpetually raising the bar on the overall quality of the protection we deliver to Trellix customers. Trellix solutions are validated by multiple third-party tests, and the solutions consistently perform across these evaluations. Each testing company has different rigorous methodologies that highlight different and important considerations for a security product, and considering these results collectively provides a comprehensive view of a product’s capabilities.