One of my all-time favorite quotes  is "When all you have is a hammer, the whole world is a nail"... but lately that quote has started to apply to the practice of web application security programs, and that's causing me to start losing sleep.

Speaking to large-scale enterprises about their budding web application security programs has seen its share of challenges - but when one of the problems is the program's implementation... then we're faced with a different kettle of fish altogether.  The issue arises from a mis-guided approach to solving enterprise problems based on a single, vendor-inspired, point solution.  Spotting this imminent failure is rather simple, here are some warning signs you're about to head down a dangerously failure-laiden path:

1. Your enterprise web application security program is based on a point solution from {insert vendor here}
1. "solution" can be a tool
2. "solution" can be a service
2. You determine risk rating based on a pre-defined risk rating of high | medium | low
1. if you're taking a vendor's risk rating without your own context
2. if you don't add context to your vulnerability rating
3. You believe that tools can replace people in security vulnerability detection
4. You believe that people are more effective than tools in security vulnerability detection
5. You believe your enterprise is safe because your {black-box scanner | white-box scanner | service-provider} says so

The foundation for any good web application security program, one that will stand the test of time and circumstance, is diversification in the methods for risk-detection and analysis.  I've said it over, and over... a diversified program must be at the root of your long-term strategy.  Remember, every good practicioner understands the place of tools, services, and internal processes ... and never trusts his company's security and personal success to any single one of the aforementioned items.

The reality is this - every tool and human tester has a shortcoming, a blind spot - something {it | they're} simply not great at.  Flipping that around ... every penetration tester has a specialty - SQL Injection for Oracle, Cross-Site Scripting, whatever... and tools are the same way.  The problem with all of these is that if only one of them is used you end up with a narrowly-focused spotlight, completely missing everything else.  As an example, if your web application is tested by a human being who's great at exploiting Oracle-based flaws but isn't well-versed in CSRF and other flaws, you will only get part of the story.  Just like if you were to scan your application with a white-box scanning tool... since the analysis is static-driven you won't actually know what's really going on... in a dynamic (data-driven) test.  Either way, you're left with an incomplete picture.  Relying on this incomplete picture as your security posture leads to a false sense of security, and a feeling that if you fix the identified flaws in {inserver vendor here} report, your enterprise will be secure.  That logic simply does not hold water.

This type of failing approach will lead to a condition of "You don't know, what you don't know"... and if you assume there is nothing else to know... your web application security program just failed to mitigate the most important risk - that of the unknown.  Therefore, the answer is unbelievably simple, and unfortunately uncomfortable for some of the vendors in the security practice out there... you have to dip into both buckets.  This is precisely why people seek the second opinion of another doctor - to get another look at the same problem.

The irony is that if you give 3 excellent penetration testers the same exact web application, which is relatively complex in nature (as almost all enterprise apps these days are), you limit their exposure window, limit the amount of money you can spend on their efforts...  odds are you will get 3 slightly varied results.  Worse-yet... one of those results may be the vulnerability that causes your next appearence on the front page of the Wall Street Journal.  The same goes for web application security scanners... black box, white box... doesn't matter.  Given the complexity of enterprise applications you can start to see why these differences arise...

So in these trying times which is the right course of action?  While your mileage may vary and there is no one-size-fits-all solution, here's my recommendation... these few simple rules have served me well in some of the most challenging enterprises in the world:

1. Layer your strategy on top of a baseline of education; this does not have to be in-depth security training only a base knowledge of the CWE Top 25, or some other list if you need something for reference
2. Implement a baseline of tools throughout your lifecycle because you simply can't go wrong with minimizing risk by removing "low-hanging fruit" off the tree
3. Adopt a major/minor release analysis model, which looks like this:
1. Major releases [Y.0, (Y+1).0, (Y+2).0] receive special attention from an outside party (your favorite penetration testing bunch) and your in-house set of vulnerability scanning and analysis tools
2. Minor releases [1.x, 1.x+y...] receive scrutiny from a set of in-house built tools, based on an SDL approach which gives greater coverage and better chance of catching bugs
3. Define major/minor releases as...you see fit for your organization's budget, release strategy, and SDL type

Good luck.  Remember... when all you have is a hammer, everything is a nail.