Anti-virus is dead and it’s all about the next-generation products that can detect, track and gather intelligence on sophisticated, targeted attacks. Or so Wall Street seems to think. If you also buy the hype then you might at least be interested to learn exactly how effective these new products and services are. Which of the extraordinary marketing claims contain a grain of truth and which are pure snake oil? It’s hard to know without independent test results but sadly there is currently a dearth of such reports. Why is that? I can give you a few reasons…
The APT myth
If you don’t clearly define the threats a product is supposed to manage then it’s very hard to test the vendor’s claims. There are many different definitions for an Advanced Persistent Threat (APT) and similar terms and acronyms used to describe anything from a specific entity of nation-state cyber-spies to a vaguely targeted phishing email. The minute a vendor clearly defines what it protects against the testers have something to test – so a vendor may not provide a clear definition. Vendors that rely heavily on marketing over engineering see no benefit and only potential problems with independent tests.
One way to infer a definition for ‘APT’ is to read the detailed blog posts that some vendors publish when they uncover an ‘APT’ campaign. If that campaign involves sending a spear-phishing email with an attached Word document (that contains an exploit that’s been public for two years) then you have the basis of a test methodology right there. If the attack involves primarily a well-managed infrastructure to push out zero days then things look harder for the tester.
A vendor might claim an interest only in the latter, hard-to-test areas, even if in reality it (and its customers) care a lot about more mundane issues. That way its product becomes untestable and its marketing claims unassailable.
Can you test the untestable?
Is it possible to test an anti-APT product? Yes, of course you can. If you can hack then you can attack test systems and see how an endpoint or appliance product responds. That is one answer, but another more thoughtfully considered one is, “probably, as long as you define your scope.” Are you testing the product’s abilities in handling basic attacks; attacks that require at least some knowledge of public hacking tools; or genuinely sophisticated attacks that use novel ways to evade detection? You need to decide which areas, or combinations of areas, to cover to create a defensible methodology.
Another equally valid answer is, “no, you cannot test.” This has nothing to do with a tester’s competence and everything to do with lawyers. Some products have end user license agreements that forbid comparative testing. One should ask why such restrictions are imposed on customers of equipment and software that costs thousands, or tens of thousands, in any first world currency. So, why are vendors shy of being tested? The answer may lie in history.
Over the years, and even within the last 12 months, there have been public spats between security vendors and third-party testers. The private arguments are even more frequent. Some vendors believe that testers are dishonest, incompetent or both. As such, their results are deeply suspicious and are of minimal value when marketing their products or seeking to improve them. I’ve heard one vendor refer to a certain testing organisation as running a coin-operated testing service, which provides the results you pay for.
Anti-APT products are often developed and managed by people who used to work in the ‘anti-virus’ industry. Some of these will have encountered testers who proved to be unimpressive. It’s not hard to imagine that their experiences will prejudice their view of any tester that now claims to offer APT testing. Experienced testers are tarred with the same brush, while new testers are inexperienced and most likely lack the necessary contacts to even get their operation off the ground.
Advice to testers
If you want to test one or more anti-APT products or services be clear about the purpose of your test. A basic test has worth as long as you don’t stretch its conclusions to breaking point. Decide which type of attacker(s) you want to replicate and then research the known tactics and tools that they apparently use.
Test all layers of a product’s capabilities, though, even in a basic test. For example, if you want to introduce a Word document infected with a certain exploit create this file, attach it to an email and send it to the target over the internet. If you just drop the file onto the Desktop from a virtual machine or USB drive you’re potentially ignoring a whole range of detections and protection layers.
Don’t have your exploit perform a harmless action like running the Windows calculator application. Get remote control of the system, issue commands and potentially escalate privileges to System. Or at least exfiltrate some data.
Nothing is untestable, but neither is a test ever perfect. Tests are useful simulations, which is why we strap crash test dummies into cars, rather than real people, before ramming them at speed into concrete blocks.
Simon P.G. Edwards will be talking about Effectively Testing APT Defences on Friday at the Virus Bulletin conference in Prague.
