Most Security Products Fail Initial Certification

[Gauravs90]

In commemoration of their twentieth anniversary, ICSA Labs on Monday released a comprehensive report summarizing 20 years of testing. The most surprising conclusion? Nearly all products fail certification on the first try.


Over the years, the company has moved from hands-on testing and paper reports to fully automated systems and databases. According to George Japak, ICSA's managing director and one of the report's authors, it took six months just to make the oldest data fully accessible for data mining. Over a dozen analysts and experts worked on the report, which offers serious insight into the evolution of the computer security industry.


PCMag draws on reports from ICSA and other labs to help rate the success of antivirus products. A dedicated test facility can use an immense set of samples to check a product's ability to detect and remove malware. While the company started off focusing just on antivirus products, over the years they've added testing for network firewalls, Web application firewalls, intrusion prevention systems, Internet protocol security, and Secure Sockets Layer, as well as custom testing services.


Certification sounds simple – take the test, get the award, display it with pride. But in truth most products have to go through several cycles of testing before they reach certification, and they can lose that certification if periodic re-testing shows they're not keeping up. The most striking results from the report center around the ways products fail to reach certification.


Out of all the products ICSA tests just 4 percent achieve certification on the first try. Antivirus products, with 27 percent, pull the average up; in all but one of the other categories no products have been certified on the first try. 92 percent of antivirus and 82 percent overall eventually achieve certification, averaging 2 to 4 test cycles to reach that point.


"It should not be assumed from this seemingly high success rate that faulty products are given a free pass or that the bar is progressively lowered until everyone easily steps over and attains certification," the report said. "In fact, the bar is routinely and systematically raised."


The report breaks down just what kind of violations prevent a product from getting certification. Naturally the top violation is failure of core functionality; for example, if an antivirus product fails at virus detection, it's out. The next most common violation involves logging, which is especially important for enterprise customers. Incomplete or inaccurate logging can actually be a violation of law. Surprisingly, the third most common violation involves security flaws in the product itself. For example, a web-based control console might be vulnerable to cross-platform scripting.


Not surprisingly, the report concludes that certification testing improves the industry. Customers, both consumers and enterprise, should make a point to look for and understand certification logos from ICSA and other major labs. Vendor size does not correlate directly with product success, but a history of successful certification does. And of course you're safest staying away from the bleeding edge – a mature product is generally a safer bet than a brand-new one. You can view the full report here.

Source

[rohanmathew]

Thanks for the information.