[Update 01/25/06 - This blog is now also available as a webcast.] 

I'm often asked - "what's the best way to find vulnerabilities?". That's kind of like asking "what's the best way to become a millionaire?". It depends. It depends on a number of things such as the resources you have available, your timeframe, the skills that you have, etc. In short, there is no correct answer, just different approaches that are appropriate in different situations.

No one approach to vulnerability discovery will find everything. That said; let's start at a high level with the two basic approaches to vulnerability discovery - white box and black box testing. To overly simplify our definitions, think of white box testing as access to everything and black box testing as access to only those resources that would be available to any other end user. During a white box approach, we would have access to source code, developers, planning documents etc. and for a black box approach, available resources would generally be the application binary and end user documentation.

White box testing has it's advantages and disadvantages. Code coverage is a significant benefit provided by white box testing. It is much easier to determine if you've looked at all functions, libraries, etc, when you know what they all are. On the flip side, white box testing suffers from the fact that it is not looking at the code in a runtime environment. That's important for a number of reasons. Exploitation of a vulnerability is dependent upon all aspects of the platform being targeted and source code is just of those components. The underlying operating system, the backend database being used, third party security tools, dependent libraries, etc. must all be taken into account when determining exploitability. A source code review is not able to take these factors into account. Validation of exploitability requires black box testing whether it is done with automated tools or during a manual validation process Take for example a cross site scripting vulnerability that's uncovered in the application source code. From a source code review you could determine that the code is indeed vulnerable and that a path exists to allow user supplied data to access the vulnerable function. You may not however have access to the code of a third party library used to validate user input or know that an application firewall would succeed in preventing the attack. While that doesn't change the fact that the vulnerable code still exists, there is an important difference in risk between vulnerable and exploitable code. As a result, white box testing tends to suffer from a high number of false positives.

Black box testing on the other hand, while able to evaluate the system at runtime suffers from an inability to track code coverage. Testing an application without knowledge of it's internal structure makes it very difficult to know if your tests have covered all potentially vulnerable code that could be accessed with alternate user input. Code coverage/tracing tools such as PaiMei, are making it possible to track code coverage in binaries but such tools are still in their infancy and while they can show you what you've hit, they can't tell you how to get to the code that you've missed. Consider for example a hidden page on a website. While it could be chock full of vulnerabilities, if no other portion of the site includes a link to vulnerable page, a black box tool would never be able to scan it in the first place.

A new approach to vulnerability discovery is emerging which combines the advantages of both black box and white box testing in an effort to eliminate their deficiencies. This approach is referred to by many names such as gray box testing or hybrid analysis. Before continuing, allow me to reveal my bias by stating that SPI Dynamic's DevInpect tool uses a hybrid analysis approach and while we've trademarked the Hybrid Analysis name, we would never claim to be the first to conceive the overall concept. There are a number of excellent academic papers extolling the merits of combining white box and black box testing. If you're so inclined, take a look at an interesting paper discussing how this approach was applied to C++ code used to control a NASA rover. If you're a visual learner, take the time to view a Google video on DSD (Dynamic, Static, Dynamic) Crasher, a bug finding utility created by Christopher Csallner and Yannis Smaragdakis from the Georgia Institute of Technology.

While the approach makes sense, it too comes with its own challenges and pitfalls. The false positive rate during the black box component of hybrid analysis is still directly proportional to the quality of the generated test cases. Additionally, tying an appropriate black box test to vulnerable code identified during white box testing is not always a trivial matter. It will be up to security software vendors and researchers alike to overcome these challenges, but the approach does hold promise. Gartner tells us in their "MarketScope for Web Application Security Vulnerability Scanners, 2006" report that by 2008 they predict with a 0.8 probability that leading web application security vulnerability scanning vendors will move to a hybrid analysis approach. I would expect this same prediction to hold true for security vendors outside of the web application space.

With that said - "what's the best way to find vulnerabilities?" I stand by my earlier statement. It depends. However, if I were tasked with discovering vulnerabilities in my own application, I wouldn't rely on white box or black box testing alone, instead I'd do both. Whether that's done using a combination of existing white box and black box tools or by choosing to go with next generation tools taking a hybrid analysis approach is entirely dependent upon the quality of the products that you choose. As with any procurement decision, I'd line up the contenders, conduct a bakeoff and go from there. Let the games begin.

- michael