Following the White Rabbit Blog : security automation

Automated Security Testing - Can't I Just Point-n-Click? (Part 3)

RafalLos — Fri, 16 Oct 2009 21:49:00 GMT

So now that you've got the background from my other 2 posts in this series, you know the options and you have some background. Let's talk about the limitations of technology and why your brain is still required to do your job. Many folks continue to try and push the boundaries of technology, and while I applaud this effort greatly, I for one can't see us security analysts ever being replaced entirely by technology as some would have you believe. The analytical mind still trumps technology ... although I think there are some limitations based on levels of experience, etc. Read on for more ...

Technology & Automation's Limitations

Let's face it, there are some very serious limitations to technology today, even in the product-filled world of web app security. I think you will probably agree that there are many products that solve non-existant problems ... or what I would refer to as "brilliant solutions without purpose"... but we'll save that conversation for another time. Right now if we look at automation logically we can simply state that automation ,more specifically software, has its limitations at pattern-matching, for the most part (more in a minute). Immediately we can say that pattern-matching is a severe limitation to any technology, just look at the failed anti-virus installation on your computer. Does it protect you from every strain of every virus? what about new malware? Of course not ... that's why everyone pretty much agrees anti-virus in present form is a dead concept. Moving this into the web app sec world we can easily say that pattern matching is next to impossible when you look at static analysis (analysis of source-code) because thanks to the brilliance of the human mind we all do things just a little bit differently. To prove the point, ask 10 developers to write the same piece of code, even a simple function, you may find 2 that are the same ... maybe.

Static analysis is particularly difficult from the perspective of automation, although there are great attempts out there I will acknowledge, because you're dealing with code. As I've written previously static code analysis has enough of a hard time with dealing with producing theoretical vulnerabilities, much less trying to understand every developer's code. This is why there is no such thing as an "out of the box" tool that works on static code analysis. Let's be logical about it ... your "sanitization" function can't possibly be anticipated by the tool you just bought to analyze your code ... and while the tools available can make attempts to "learn" the way your developers code, and what functions are safe, which are scrubbers, etc in the end it's just hours and hours of "tuning" that require ... ta-da ... human intervention!

Taking the case to dynamic analysis doesn't make it any more pretty. Again, here we're pattern-matching against expected outcomes to "negative testing". We push javascript (such as the ever-popular pop-up) into a form field and expect it to come back to the browser in the same way that it was sent and execute a pop-up. Then we can determine that it's a vulnerability... right? It's not that simple though - because when you look at code coming into the browser you have to analyze the context it's being piped into! If you're pushing code you have to make sure it will actually execute first ... which is the challenge. Next, for your consideration think about how we test for database manipulation (SQL Injection). We send database command syntax appended or injected into the regular application fields to try and elicit a database response. Of course ... if the developers suppress databse responses to the end-user this makes it very difficult to detect injection when it's "incorrect"... Concepts like Blind SQL Injection are even more tricky because you're injecting database commands and not expecting a direct response but a change in page-state, or a positive/negative response which is also extremely difficult to script and contextualize. You'll notice that a lot of this comes back to context and while software can do catagorization pretty efficiently a la pattern matching, it's impossible to account for all possible states, responses and configurations. Yikes! This is all enough to make your head spin!

I certainly don't envy those developers who are writing security analysis tools, and I can tell you first-hand that the folks that work in our HP Web Security Research Group are absolute geniuses. Scripting and pattern-matching gray-area responses is like walking a tightrope between false-positives and false-negatives ... and remember that no matter what you do people will attempt to discount the tools you build because they're either too noisy, or miss too much. This is especially why I am so big on human interaction in the process!

Your Brain Required

Now we get to it ... your brain will continue to be required for the forseeable future in security, more specifically the analytical part of web app security. While technology and innovation will continue to drive better and "smarter" engines for analyzing and attacking web applications, my crystal ball tells me that people will always be necessary. Actually, not just people. People with a clue will always be necessary - there is a huge distinction! Let's venture into why [intelligent] humans are necessary, and why anyone selling "a point-n-click" security tool should be laughed out of the building. You see, people build software. Even the smartest people make mistakes. Therefore, even the best software will have mistakes which often manifest themselves as security vulnerabilities. Given that, why would you trust the analysis of this potentially vulnerable software with more potentially buggy software? Make sense yet?

Software-based testing, even software-driven testing is fine as long as there is someone who is schooled and reasonably accomplished in the art and science of interpreting results and analyzing them. What is required here is a 2-step method we like to refer to as "validation" of findings. You see, automated tools continue to get better at finding more and more complex defects yet the analysis of findings will always be the trickiest part of a security testing strategy. Looking at what an program/script/tool has uncovered and being able to critically deduce whether this is a positive vulnerability, a false-positive, or whether it simply requires more attention is critical to a security analyst's position and job description. The power of the human mind often kicks in where software leaves off, and can trigger a multitude of findings that would otherwise go undiscovered.

A great example of this type of need for a human analyst is from a penetration test I was a part of a while back. The automated tool uncovered a treasure trove of low-hanging vulnerabilities including some cool SQL Injection and Cross-Site Scripting issues, as well as a crossdomain.xml issue that was pertinent to our attack. On their own these attacks could do some damage but it wasn't until the analyst actually dug into these attacks and noticed that they could be chained together to produce an incredible attack vector that there was (at the time) no solution for! You see, we could test for XSS, and SQLi, and even the crossdomain.xml vulnerability ... but the software couldn't string those together and notice a gaping flaw in the design of the application that allowed for a complete compromise of the online application.

So the bottom line here is that I want you to walk away from this series being able to not only understand but intelligently speak about why a "point-n-click" security testing tool will never suffice, and why you have to have the human intellect to back it. That being said, there are a number of offerings such as HP's Web App SaaS offering which mix the automation and tools approach with an augmentation of the human factor for when you find yourself in a situation where you just don't have the in-house expertise! What I'm saying is don't trust your web application security to a tool, or even a collection of them - because alone they aren't telling you the whole picture. Throw away the notion that you can just point and click your way to being secure ... it's never going to happen that way.

The answer then? Education, first and foremost, is key. Make sure you either educate or hire smart & intelligent security analysts. Make sure that you have people who understand how attacks work, why they work, and how to detect them manually. Your analysts should be able to spot basic attacks like SQLi and XSS in a site by hand, and execute (or know where to get cheat-sheets for) the more complex attacks. You don't have to hire the uber-hax0r, just know enough to call one when you need one. The next thing is to ensure you've got the best tools in your toolbox... often this means mixing open-source and closed-source apps together into something that works best for you. Know your applications and which attacks apply ... PHP-style attacks certainly won't work against IIS-based ASP.Net apps ... usually. Be ready to raise your hand when you're in over your head. There's no shame in asking or acknowledging when you don't know ... I do it all the time and it's quite liberating.

I hope I've managed to convince you that point-n-click security is a failed prospect. What do you think? If you're interested in a further conversation please feel free to email me (via this blog) or get a hold of me through your HP sales rep (you probably already have one!)

A Perspective on "Dumbing Down" the Security Profession

RafalLos — Tue, 02 Dec 2008 04:41:00 GMT

Let me start off by reminding you that the main mission of this blog is to provide insight and perspective (from more than just the security angle) on web application security and risk management. Keep that in mind as you read on...

I read an article on ZDNet tonight as I usually do to catch up on things I've missed - and something caught my attention and my ire. This article by Shyama Rose made me read, and then re-read just to make sure I didn't miss anything in her train of thought. While I have to say she has some very good points (I will name them below individually) she's pointing out something that is inherently biased, I think, from a consultant's point of view. Googling her name, produces this LinkedIn profile which makes her bias more obvious and understandable. This post is *not* about bashing or negating someone else's opinion, I will only seek to point out some obvious (to me) flaws in the thought process.

Let's analyze the article, for content and idea so that I can make some relevant points. I realize that I may have my own bias, working for a vendor of "web application security automation tools", but bear with me and I will do my best to keep my vendor-bias out of this article.

There indeed is a large market "swelling" but it's not just for source code analysis tools

The vendor-space is actually *shrinking* (trust me on this one) due to the smaller boutique vendors ability to compete in a venture-capital starved market and budgets that increasingly go to a single-source vendor of multiple items (software/hardware/services combined)
Source-code-analysis is a small part of the overall security tools market; in fact - I think that while companies seek to purchase "source code analysis" engines they eventually realize the issues with that*, and instead purchase "web application security scanning tools" (such as WebInspect & AMP from HP/ASC... sorry, couldn't resist)
While tools are a hot commodity right now [emphasis on the commodity], there is no shortage of customers for boutique shops doing manual versions of this type of web application security testing (with a nod to my friends in white hats fighting the good fight) - Tools and services are not mutually-exclusive, rather, part of a comprehensive risk-mitigation strategy

The quote "...deficiencies associated with security guarantees that tools promote" is either absolutely inaccurate or we as security automation tools vendors are doing a terrible job selling our products, period.

Yes, automation-based security tools are in their infancy
Yes, automation-based security tools identify ~35% of the total possible attack surface of an application [more on this in a minute]**
Yes, metrics-based management practices are being increasingly used; why? because that is currently the only way to demonstrate effectiveness. I would love to debate the real-world application of security metrics in a qualitative vs. quantitative approach... anyone?
Yes, some of the most sophisticated security tools are partially signature-based... but the best ones are also data-flow and control-flow analyzers; tools by nature cannot "think"... Skynet, anyone?
Yes, most source-code-analysis tools fail on frameworks. Can you blame them? How many Java-based frameworks are there? What about .Net? How about AJAX? (Here's a fascinating read on AJAX frameworks... http://en.wikipedia.org/wiki/List_of_Ajax_frameworks) -this all relates back to maturity of tools vs. development language advances.
Yes, many tools (source-code-analysis or otherwise) base themselves around an assurance or compliance angle (PCI-DSS and others) - but that's because there is a business need for this, and not because they simply like it that way

Another interesting quote "...the differential between a comprehensive security review and the implementation of analysis in analysis tools is massive and harmful"

Massive? Probably not
Harmful? Absolutely not!

OK, so now that I've got that broken down a bit, let me give a sense of why I feel this article is a bit of a hate on the security tools out there today. Anyone who's ever heard me talk, or read my posts/articles/papers would hopefully agree that I have never, ever advocated tools as a replacement for intelligent people. There's no arguing that, there's no replacing intelligent people. Furthermore, tools in themselves are not a solution and won't magically make your code secure. Sales people will often try and sell you snake-oil and magic pixie dust to make your web applications "magically secure" if you use them. That's crap and we all know it.

So why even consider automation-based web application security tools? It's elemantary my dear Watson! Here's just a few reasons why you should be making that purchase sooner rather than later...

Automation - Automation is what makes tools viable. Tools are great at repeating processes and supplementing the human mind to enable it to move faster and have to do less of the mundane (like finding the "easy stuff"); also... testing is generally done off-hours so while humans need time off... machines and tools do not
Expertise - Do you (as a company) have $100k+ per year to spend on a talented web application security resource? Consider that you may need more than one depending on the size of your enterprise and number (and complexity) of your web applications
Metrics - Even though Shyama seems to be against metrics-based risk mitigation, I am thrilled to have tools that can track web application security defects throughout the lifecycle (development -> QA -> production) and show the ROI and risk-reduction metrics in a nice compact dashboard for my CIO
Process - Every good enterprise web application security process (and enterprise security strategy in general) must have some tools component to complement the people and processes... common sense right?

Let me quickly address the points I put a star next to above:

* Source-code-analysis tools are much more difficult to understand, implement, interpret and get use out of than most black-box testing tools. Also, given that the highest risks are to those applications already in production, it makes sense that black-box testing tools (and not source-code-analyzers) are the go-to choice for enterprises struggling with the "what do I do first?" question.

** This ~35% number is debatable, that is, the rough percentage of security defects a tool can find vs a human analysis. On the whole I think it's reasonably correct but people get too caught up in the details of the number and forget the 80/20 rule altogether. 80% (or more!) of hacks happen employing the "simple things" like SQL Injection and XSS... rather than highly-sophisticated complex logic or multi-stage attacks. I don't have any raw data to back that up - but I'd be willing to bet I'm not wrong. If a simple scan from the WASC group showed 85% of sites scanner were vulnerable to attack that would compromise data... that's tools + automation... and that's a scarry number!

I know this is a lot of information to wrap your brain around, and hopefully you won't judge too harshly for my bias (everyone has some bias...) but I hope I've inspired you to think a little before jumping to conclusion that the world of security-source-code-analysis is going to implode because consultants will go out of business due to security automation tools being used more. That's just a silly conclusion. Do companies rely too heavily on tools? Maybe, but what alternatives are we offering given the cost of "manual" testing vs. automation? I will leave you with some simple math. Say for a moment you have 200 web applications in your enterprise. Of those, taking the top 15% (see my Iceberg Principle presentation for more on this) as mission-critical we have 30 web based applications. Given that most of today's web applications are moderately to highly complex, say 100,000+ lines of code we can do some basic math:

Application Security Automation "Suite"... ~$500k - covers all phases of lifecycle and all applications (not just top 15%)
Outsourced Manual Testing - $10,000 - $20,000/application x 30 applications = $300,000 - $600,000... for just 15 applications

... now, factoring in time and efficiency, and given that most business can't afford to just test their super-critical apps... I think the case for security-automation tools is a case of simple math, don't you? (Of course, my numbers are rough SWAGs... do your own research to see what's right for your enterprise).

Compliance: Ushering in the Apocalypse!?

RafalLos — Mon, 17 Nov 2008 03:56:00 GMT

I read an interesting article tonight, on my flight out to Washington, DC for the CSI Conference (where I hope to meet some of you... ping me if you're here and I haven't talked to you yet). This article, titled "The Coming HIPAAcalypse", presented a very grim view of compliance with the HIPAA regulations, but the author could have just as easily been talking about PCI or any other regulation. As I read this article I couldn't help but think... "What does it have to be this difficult?" I say this from experience, having been there in the thicket of PCI compliance in a previous job - trying to manage the complexities of budget, compliance need, and resources with frustration to spare - but does it really have to be so hard?

Thinking in the context of web applications - that's the focus of this blog - everyone has them in their business. What's more, everyone has mission-critical applications in their business. Further than that... these applications must be available, usable AND secure... 24x7x365. This can at times seem like an unattainable goal when you add compliance to the mix. You know you've been there, and you've wondered how you're going to solve these issues.

I offer a glimmer of hope. When it comes to compliance, automation is the key. You won't get more staff, more budget, or more resources especially with the looming economic conditions so how do you get into the compliance winner's circle? Automation. If you can find a way to automate some of your security "testing", you may be able to get complianct faster and have a few dollars left over for other critical security initiatives. Automation of testing, data aggregation, and presentation of IT Risk as a component of compliance makes it easier to not only assess where your company is on the compliance journey - but also helps to (to use an old cliche) "do more with less". If you're facing compliance challenges, and web applications are involved... this should ring bells for you. If you're interested in hearing more, or a message more customized to your specific situation - contact me directly, I may have an answer you can live with.