• The Options

If you're going to attempt to test web applications with some measure of automation there are a few options you have available.  There are full and partial automation opportunities, and application separation as well as multiple tools.  Addressing them in order here...

Full automation is what most people still think of when it comes to security testing their web applications.  Full automation involves simply putting a URL into a field and clicking GO and standing back to watch the action.  There are times when this is practical but there aren't many of those times, unfortunately.  I've spoke with many folks recently who feel that web application security testing should be done like vulnerability scanning was when it first kicked off.  Point, click, and receive results.  This isn't practical because of the fact that there are many possible ways that this option can fail.  Sadly, the less people understand the more they want to push into full automation.  Let's think about full automation for a minute.  In order for a tool to be able to perform a fully automated scan you have to assume that the tool can analyze site structure and compute an attack strategy on the site without human intervention.  Forget that you're asking a whole lot from a computer program ... think about what that actually means.  You'll be asking the tool you're using to be able to understand every part of the application ... fully.  Can you say you can understand every part of the applications you test fully?  Remember software is only as good as the people who write it, and unfortunately the people who write testing software can only make it as good as the examples they have to work with.  Herein starts to peek a problem we'll address later ... mounting complexity.  Full-on automation requires that the tool analyze every AJAX call, every FLASH object, every piece of JavaScript, every nook and cranny and every workflow through the application.  If you're heard me talk about the failure of automation on the frontier of workflows you already know why this is such a losing proposition without human automation - but it gets more complex than that.  You're hoping that the automation component can do all the work in a pre-defined amount of time, right?  Let's be realistic, most automated tools, if not properly tuned will run for days, hours or weeks before running themselves out of memory of stack space - hopefully completing the scan.  The reasons this happens I will address later on in the technical limitations but you're asking an awful lot of software that's testing software.  Say you do get a complete scan.  Say for the sake of argument that the tool you're using manages to completely cover the web application attack surface and finds a whole mother-lode of vulnerabilities.  What you're saying now is that you want that same piece of automation (or software) to be able to validate its own findings.  Fail.  You already know that automation isn't perfect at finding vulnerabilities ... and now you want validation for the same price?  Consider that ask...

Partially manual testing is the next logical choice.  Involving the human being as little as possible but still allowing for some intervention to do the set up and validation makes logical sense.  The problem here is that the human being here has to understand what he or she is doing otherwise this process fails.  Integrating a human being into web application security testing is a scary thing ... because now you're asking a human being to complement the software you're using but it certainly has its advantages.  In fact, I would argue that it's better to have a human involved than to attempt to do everything with automation as you'll get better results 4 out of every 5 times.  The problem is in the human part of this equation.  Knowing what you're doing ("I'm testing a web application's security") and actually knowing what you're doing are drastically different.  You also have to be trained in the tool you're using otherwise you'll fail with even more vigor.  But here's the deal, partial automation involves the human being (tester) interfacing with the tool in order to provide it not only analytical insight but also guidance on what to test, what variables to use, what to tweak and what to avoid ... then analyzing the results.  This is what most knowledgeable penetration testers and web application security experts do today with varying degrees of success.  Don't let anyone fool you, it's a lot tougher than you'd think to get results particularly when they have to be consistent!  Tweaking a piece of software and using it like a sledgehammer to find the low-hanging fruit is fairly easy ... getting deeper and better results than the tool could do on its own is a little more tricky.  Lots of testers simply never master this craft and either end up blaming the tool, or simply giving up.  Partially automating a testing tool, particularly one that's built to do evil, is an art-form and must be well-understood or the results could not only be catastrophic, but also inconsistent and more dangerous than when the tool is run fully automated.

Your other option, of course is do testing in a partially automated way.  What you probably don't know is that tools like WebInspect can function in this capacity brilliantly.  "Penetration tester assistance mode" is what the folks who do this all the time call it.  As the penetration tester looks at different areas of the site a black-box scanning tool is used surgically, with a large amount of human guidance.  This use-case really isn't a human being assisting an automated tool as much as an automated tool is used as a supplement to the human being's abilities to do the mundane and simple tasks.  Furthermore, more advanced tasks can be performed such as advanced XSS or SQLi testing within the framework of the tool so the tester doesn't have to do it by hand.  Using the tools as an extension of the tester is a great way for someone advanced in the art and science of breakage to function ...but that expertise has to be there first.  You can't just jump feet-first into this type of usage model and expect to succeed.

So there we have it, 3 possible ways to engage in "automated" testing tools, a la black box security testing.  The thing you must think about is which one is right for the situation you find yourself in, your knowledge level and experience, and specific use-case.  What works for one may not work for others, your mileage may vary, batteries not included and some assembly is required.