I've been witness to an interesting phenomena.  Several otherwise rational folks- customers, prospective customers, and pundits alike - have posed the question to me now over a the last several months.  I've been thinking a lot about the topic and have some thoughts I think it's time I share.

The question for discussion is this: "Shouldn't a security testing tool (Web App security, black-box specifically) be able to just accept a URL and credentials and test my site, providing results without me having to intervene?"

The answer, quite simply is an unabashed "No"... but I think it needs more of an explanation than that.  It's often all too simple to provide an answer without explanation; or worse with an explanation that not everyone can understand, so I'll both answer the question, explain it in detail and give some real-life examples of why I'm answering this way.  Grab a cup of coffee, get comfortable and let's think this through rationally together.  I'm going to do this as a multi-part blog entry ... I can already see this as taking a few hours to write much less to read and fully comprehend... 

• The Main Issue

The main issue in question here is not whether computers can replace humans entirely for security testing - which I hope we can all agree on is a solid no but whether computers and automation has come far enough to begin test automation to a point where a human can provide minimal input and have a test complete.  The problem with this request is that we're asking automation to make decisions within the process of testing.  Decision making, so far in evolution, is best left to the human analytical brain, rather than automation - and the primary rational is here is that humans possess the ability to reason rationally whereas computers ... cannot.  At the core of the question is the ability to make decisions or reason which then either makes or breaks an automated test.  Let's think about this in a different light... let's look at this from the viewpoint of a mechanic.  What we're really asking here is for a computer to hook up to the vehicle, diagnose the entire system without human input and then provide a solution, testing the effectiveness without a human in the loop.  Rationally we can already see where this would break down.  A computer can hypothesize a problem, apply a solution successfully without actually solving the problem the driver had in the first place.  Diagnosing a problem in a vehicle, as mechanics will tell you, is more than just something you can do from a text-book, or by taking a course.  It takes years of experience to understand vehicular cause and effect, and why a rattle in the front of the car may actually be a bad bearing in your rear wheel... computers can't tell you these things, yet.  The other issue here in the mechanical world is that not everything can be connected to a computer system for diagnostic yet - there are still limitations.  The problem can be easily extended to the digital world for web applications.  Not everything can be analyzed properly and we'll go into more detail in a minute for why that is.

Bringing this back to the question at hand and whether automation can simply "do the job" of assessing a web application's security viability ... we have to break the issue down into its bare components to further analyze.  First, there's the identification and site functional analysis ... typically we call this the "crawler phase" or "discovery phase" depending on which tool you're using.  Crawling the site (or application) means clicking buttons, inputting data, and traversing the site all while building a virtual map of what the site looks like, what the option trees are, and how traversal through the site is done legally without attempts to subvert the site.  The next major step is the pre-attack analysis - whereby the tool attempts to build the attack sequences and tree for how the site will be attacked.  This type of phase generally involves a lot of heavy memory and processor usage and building incredibly large and complex data structures (generally in machine memory).  Once this is done the attack sequence can begin.  Once the tool is confident that all attack patterns and plans have been laid out, the attacks are launched and the tool starts to do the heavy lifting it was built for.  Inevitably during the attack process something new is discovered.  Whether at attack pattern triggers some new function, or something breaks in a beautiful way ... the system has to put that newly found functionality back into the control-stack of the application for re-analysis and another pass.  The tool will continue making the start -> discover -> attack-build -> attack -> repeat loop over and over as long as new things are discovered... until there is nothing new left on the discovery stack.  Once the tool reaches that state it can be understood that the attack and discovery phases are complete and the tool moves to a final attack-analysis phase.  At this point it will have to correlate, verify and validate the findings from throughout the process to make sure that there aren't any issues with these findings.  The last step is to present it to the requester via a report.  Whether the report is a dashboard, a PDF, or exposted XML or CSV the reporting piece is usually pretty standard and well understood.  Having this process completely self-contained and automated is what some people seem to want - and I'm here to tell you that's a dangerous thing to ask for.

So now that we have the problem identified ... let's go talk about what options we have, why people are required and doing this completely in an automated fashion is a bad, bad idea.

...

There you have it ... the problem is now identified, unmasked, and ready to be discussed in detail.  The upcoming post will detail some of the options we have for solving this issue and what technological limitations we are faced with today, and into the future.  The last post in this series will go deep into the reasoning for why I continue to say that your brain will always be required.  Until next time!