All companies face the challenge of evaluating security tools that they will procure, but knowing where to start can be a daunting task. While there's no perfect way to ensure that a product meets your needs a little due diligence is essential. Fortunately, various resources are available to assist.
The most logical place to start is by looking at third party product evaluations. Technology publications love to conduct bake offs of competing technologies and score the contestants. Moreover, technology vendors line up to be considered for awards bestowed by the same publications but buyer beware - awards and reviews may require vendors to pay a fee to be considered in the competition, so do your research to ensure that you're receiving an unbiased opinion. Personally, next to hands on experience with the tools themselves, I would place faith first and foremost in the past experiences of current customers that you have an existing relationship with - not the happy customers put forth by the vendor. No one knows the ins and outs of a technology better than those that rely upon it on a daily basis.
While it's great to know what others think, there's no substitute for hands on experience. Making an apples to apples comparison among competing technologies requires using an appropriate benchmark. I would actually recommend against testing security tools by using an in-house application as the benchmark. While this may seem to be a logical approach since you'll be using the tool in your own environment, an in house application is likely (we hope) to only contain a small population of vulnerabilities and as such will not provide a broad view of the strengths and weaknesses of the security tool being evaluated. In the web application security space there are fortunately a number of options available in the form of freely available, intentionally vulnerable web applications that can be used for testing purposes. Foundstone for example provides a series of ‘Hackme' web applications. Each of the Hackme applications are written in different languages which allows you to target the appropriate platform(s) used in your own development efforts. Hackme Bank for example, is written in C#, with a backend Microsoft SQL database, while Hackme Books is a J2EE application. OWASP makes available WebGoat, an insecure J2EE application, but a promising new initiative is their Site Generator project. Site Generator allows users to dynamically design different vulnerable web apps by selecting the desired vulnerable components.
If you still prefer to rely on third party evaluations but aren't comfortable with the potential bias of technology publications, the National Institute of Standards and Technology (NIST), is working on a solution. The Software Assurance Metrics and Tool Evaluation (SAMATE) project, sponsored by DHS, seeks to define the baseline functional behavior that should be present in security tools. The first SAMATE initiative will focus on source code analyzers (SCA). At this point, they are excluding byte code and binary code scanners from the SCA definition but a draft functional specification for this project is already available. Beyond the draft specification, they also plan to develop test suites that will allow for independent analysis of SCAs. In speaking with NIST, it appears that web application scanners will be their next project under the SAMATE umbrella. Given that these specifications only seek to identify baseline functionality for like tools, it remains to be seen how useful they will be in evaluating security tools, but we'll certainly follow their progress.
Regardless of the process that you use when evaluating security tools, never forget that as a buyer, you have more power than you realize. Don't simply hand over a hefty check simply because one vendor has a tool that is better than the others. If additional features are required to meet your needs, ensure that they make it into the product road map. These tools don't come cheap, so get your money's worth.
- michael
Posted
01-26-2007 3:50 PM
by
erik.peterson