Automated Security Testing - Can't I Just Point-n-Click? (Part 1)

RafalLos — Fri, 16 Oct 2009 16:14:00 GMT

I've been witness to an interesting phenomena. Several otherwise rational folks- customers, prospective customers, and pundits alike - have posed the question to me now over a the last several months. I've been thinking a lot about the topic and have some thoughts I think it's time I share.

The question for discussion is this: "Shouldn't a security testing tool (Web App security, black-box specifically) be able to just accept a URL and credentials and test my site, providing results without me having to intervene?"

The answer, quite simply is an unabashed "No"... but I think it needs more of an explanation than that. It's often all too simple to provide an answer without explanation; or worse with an explanation that not everyone can understand, so I'll both answer the question, explain it in detail and give some real-life examples of why I'm answering this way. Grab a cup of coffee, get comfortable and let's think this through rationally together. I'm going to do this as a multi-part blog entry ... I can already see this as taking a few hours to write much less to read and fully comprehend...

The Main Issue

The main issue in question here is not whether computers can replace humans entirely for security testing - which I hope we can all agree on is a solid no but whether computers and automation has come far enough to begin test automation to a point where a human can provide minimal input and have a test complete. The problem with this request is that we're asking automation to make decisions within the process of testing. Decision making, so far in evolution, is best left to the human analytical brain, rather than automation - and the primary rational is here is that humans possess the ability to reason rationally whereas computers ... cannot. At the core of the question is the ability to make decisions or reason which then either makes or breaks an automated test. Let's think about this in a different light... let's look at this from the viewpoint of a mechanic. What we're really asking here is for a computer to hook up to the vehicle, diagnose the entire system without human input and then provide a solution, testing the effectiveness without a human in the loop. Rationally we can already see where this would break down. A computer can hypothesize a problem, apply a solution successfully without actually solving the problem the driver had in the first place. Diagnosing a problem in a vehicle, as mechanics will tell you, is more than just something you can do from a text-book, or by taking a course. It takes years of experience to understand vehicular cause and effect, and why a rattle in the front of the car may actually be a bad bearing in your rear wheel... computers can't tell you these things, yet. The other issue here in the mechanical world is that not everything can be connected to a computer system for diagnostic yet - there are still limitations. The problem can be easily extended to the digital world for web applications. Not everything can be analyzed properly and we'll go into more detail in a minute for why that is.

Bringing this back to the question at hand and whether automation can simply "do the job" of assessing a web application's security viability ... we have to break the issue down into its bare components to further analyze. First, there's the identification and site functional analysis ... typically we call this the "crawler phase" or "discovery phase" depending on which tool you're using. Crawling the site (or application) means clicking buttons, inputting data, and traversing the site all while building a virtual map of what the site looks like, what the option trees are, and how traversal through the site is done legally without attempts to subvert the site. The next major step is the pre-attack analysis - whereby the tool attempts to build the attack sequences and tree for how the site will be attacked. This type of phase generally involves a lot of heavy memory and processor usage and building incredibly large and complex data structures (generally in machine memory). Once this is done the attack sequence can begin. Once the tool is confident that all attack patterns and plans have been laid out, the attacks are launched and the tool starts to do the heavy lifting it was built for. Inevitably during the attack process something new is discovered. Whether at attack pattern triggers some new function, or something breaks in a beautiful way ... the system has to put that newly found functionality back into the control-stack of the application for re-analysis and another pass. The tool will continue making the start -> discover -> attack-build -> attack -> repeat loop over and over as long as new things are discovered... until there is nothing new left on the discovery stack. Once the tool reaches that state it can be understood that the attack and discovery phases are complete and the tool moves to a final attack-analysis phase. At this point it will have to correlate, verify and validate the findings from throughout the process to make sure that there aren't any issues with these findings. The last step is to present it to the requester via a report. Whether the report is a dashboard, a PDF, or exposted XML or CSV the reporting piece is usually pretty standard and well understood. Having this process completely self-contained and automated is what some people seem to want - and I'm here to tell you that's a dangerous thing to ask for.

So now that we have the problem identified ... let's go talk about what options we have, why people are required and doing this completely in an automated fashion is a bad, bad idea.

...

There you have it ... the problem is now identified, unmasked, and ready to be discussed in detail. The upcoming post will detail some of the options we have for solving this issue and what technological limitations we are faced with today, and into the future. The last post in this series will go deep into the reasoning for why I continue to say that your brain will always be required. Until next time!

Misunderstanding the Purpose of Automated Tools

RafalLos — Wed, 11 Jun 2008 02:29:00 GMT

Let's get this out in the open - there is a misunderstood purpose of automated tools in web application security. Based on my personal experiences in front of both management and engineering teams in the last few months, I feel this needs to be addressed, and addressed now.

I know that as a vendor of tools, we would like everyone to use our wares to find and mitigate their web application security vulnerabilities - but no one here is dilusional. No one here in the HP ASC will ever tell you that buying/implementing our tools will give you total security for your web applications. No one here will ever advocate our tools as the sole solution to an enterprise web application security strategy.

So why do other vendors do it? More to the point - why is it that I am often asked the question... "So can you tell me if we implement (the HP ASC Security Suite, or some subset thereof) we will have secure web applications?" Still scarrier - why do people get upset at me when I answer them with a stout "No... our tools are but one part of a holistic strategy". Before you think that this can't possibly be anyone you know, or any manager you work for... think again. The list of places and teams that have posed this question starts in government, leads to the education sector and trails into large enterprises just the same.

I know there is some level of education that has to happen, and to some degree vendors are to blame for trying to sell "Magic Bullet" solutions at times to make the sale but the reality is no one piece of software will fix your web security woes holistically. Let me elaborate, and explain my case.

First, tools are just one piece of the security pyramid (People - Process - Tools). I've had that slide in my presentations as far back as I can remember presenting, and it's served me well but I do think it's time to preach that a little more emphatically. People and Process are the other two key factors to a successful web-app-sec strategy - without them the tools are of very little use. It's like having a 500Hp sports car with a nice manual gearbox and not being able to drive a manual and having no gas in the tank. Building a successful practice takes all 3 pieces of the pyramid to be well-established in order to function. While the *people* are the foundation of the whole pyramid, the processes and tools keep the pyramid from collapsing on itself. Without the other 2, no one piece can stand alone...

I'm writing a piece on the P-P-T (People/Process/Tools), but in the mean time ... this should give you something to think about. Let's just be clear one more time... no "tools" can solve the web application security problem holistically... but I will continue to argue that HP's ASC Suite provides the most comprehensive, most complete lifecycle solution out there, bar-none.

Following the White Rabbit Blog : automated testing

Automated Security Testing - Can't I Just Point-n-Click? (Part 1)

Misunderstanding the Purpose of Automated Tools