A Perspective on "Dumbing Down" the Security Profession

RafalLos — Tue, 02 Dec 2008 04:41:00 GMT

Let me start off by reminding you that the main mission of this blog is to provide insight and perspective (from more than just the security angle) on web application security and risk management. Keep that in mind as you read on...

I read an article on ZDNet tonight as I usually do to catch up on things I've missed - and something caught my attention and my ire. This article by Shyama Rose made me read, and then re-read just to make sure I didn't miss anything in her train of thought. While I have to say she has some very good points (I will name them below individually) she's pointing out something that is inherently biased, I think, from a consultant's point of view. Googling her name, produces this LinkedIn profile which makes her bias more obvious and understandable. This post is *not* about bashing or negating someone else's opinion, I will only seek to point out some obvious (to me) flaws in the thought process.

Let's analyze the article, for content and idea so that I can make some relevant points. I realize that I may have my own bias, working for a vendor of "web application security automation tools", but bear with me and I will do my best to keep my vendor-bias out of this article.

There indeed is a large market "swelling" but it's not just for source code analysis tools

The vendor-space is actually *shrinking* (trust me on this one) due to the smaller boutique vendors ability to compete in a venture-capital starved market and budgets that increasingly go to a single-source vendor of multiple items (software/hardware/services combined)
Source-code-analysis is a small part of the overall security tools market; in fact - I think that while companies seek to purchase "source code analysis" engines they eventually realize the issues with that*, and instead purchase "web application security scanning tools" (such as WebInspect & AMP from HP/ASC... sorry, couldn't resist)
While tools are a hot commodity right now [emphasis on the commodity], there is no shortage of customers for boutique shops doing manual versions of this type of web application security testing (with a nod to my friends in white hats fighting the good fight) - Tools and services are not mutually-exclusive, rather, part of a comprehensive risk-mitigation strategy

The quote "...deficiencies associated with security guarantees that tools promote" is either absolutely inaccurate or we as security automation tools vendors are doing a terrible job selling our products, period.

Yes, automation-based security tools are in their infancy
Yes, automation-based security tools identify ~35% of the total possible attack surface of an application [more on this in a minute]**
Yes, metrics-based management practices are being increasingly used; why? because that is currently the only way to demonstrate effectiveness. I would love to debate the real-world application of security metrics in a qualitative vs. quantitative approach... anyone?
Yes, some of the most sophisticated security tools are partially signature-based... but the best ones are also data-flow and control-flow analyzers; tools by nature cannot "think"... Skynet, anyone?
Yes, most source-code-analysis tools fail on frameworks. Can you blame them? How many Java-based frameworks are there? What about .Net? How about AJAX? (Here's a fascinating read on AJAX frameworks... http://en.wikipedia.org/wiki/List_of_Ajax_frameworks) -this all relates back to maturity of tools vs. development language advances.
Yes, many tools (source-code-analysis or otherwise) base themselves around an assurance or compliance angle (PCI-DSS and others) - but that's because there is a business need for this, and not because they simply like it that way

Another interesting quote "...the differential between a comprehensive security review and the implementation of analysis in analysis tools is massive and harmful"

Massive? Probably not
Harmful? Absolutely not!

OK, so now that I've got that broken down a bit, let me give a sense of why I feel this article is a bit of a hate on the security tools out there today. Anyone who's ever heard me talk, or read my posts/articles/papers would hopefully agree that I have never, ever advocated tools as a replacement for intelligent people. There's no arguing that, there's no replacing intelligent people. Furthermore, tools in themselves are not a solution and won't magically make your code secure. Sales people will often try and sell you snake-oil and magic pixie dust to make your web applications "magically secure" if you use them. That's crap and we all know it.

So why even consider automation-based web application security tools? It's elemantary my dear Watson! Here's just a few reasons why you should be making that purchase sooner rather than later...

Automation - Automation is what makes tools viable. Tools are great at repeating processes and supplementing the human mind to enable it to move faster and have to do less of the mundane (like finding the "easy stuff"); also... testing is generally done off-hours so while humans need time off... machines and tools do not
Expertise - Do you (as a company) have $100k+ per year to spend on a talented web application security resource? Consider that you may need more than one depending on the size of your enterprise and number (and complexity) of your web applications
Metrics - Even though Shyama seems to be against metrics-based risk mitigation, I am thrilled to have tools that can track web application security defects throughout the lifecycle (development -> QA -> production) and show the ROI and risk-reduction metrics in a nice compact dashboard for my CIO
Process - Every good enterprise web application security process (and enterprise security strategy in general) must have some tools component to complement the people and processes... common sense right?

Let me quickly address the points I put a star next to above:

* Source-code-analysis tools are much more difficult to understand, implement, interpret and get use out of than most black-box testing tools. Also, given that the highest risks are to those applications already in production, it makes sense that black-box testing tools (and not source-code-analyzers) are the go-to choice for enterprises struggling with the "what do I do first?" question.

** This ~35% number is debatable, that is, the rough percentage of security defects a tool can find vs a human analysis. On the whole I think it's reasonably correct but people get too caught up in the details of the number and forget the 80/20 rule altogether. 80% (or more!) of hacks happen employing the "simple things" like SQL Injection and XSS... rather than highly-sophisticated complex logic or multi-stage attacks. I don't have any raw data to back that up - but I'd be willing to bet I'm not wrong. If a simple scan from the WASC group showed 85% of sites scanner were vulnerable to attack that would compromise data... that's tools + automation... and that's a scarry number!

I know this is a lot of information to wrap your brain around, and hopefully you won't judge too harshly for my bias (everyone has some bias...) but I hope I've inspired you to think a little before jumping to conclusion that the world of security-source-code-analysis is going to implode because consultants will go out of business due to security automation tools being used more. That's just a silly conclusion. Do companies rely too heavily on tools? Maybe, but what alternatives are we offering given the cost of "manual" testing vs. automation? I will leave you with some simple math. Say for a moment you have 200 web applications in your enterprise. Of those, taking the top 15% (see my Iceberg Principle presentation for more on this) as mission-critical we have 30 web based applications. Given that most of today's web applications are moderately to highly complex, say 100,000+ lines of code we can do some basic math:

Application Security Automation "Suite"... ~$500k - covers all phases of lifecycle and all applications (not just top 15%)
Outsourced Manual Testing - $10,000 - $20,000/application x 30 applications = $300,000 - $600,000... for just 15 applications

... now, factoring in time and efficiency, and given that most business can't afford to just test their super-critical apps... I think the case for security-automation tools is a case of simple math, don't you? (Of course, my numbers are rough SWAGs... do your own research to see what's right for your enterprise).

Following the White Rabbit Blog : security tools

A Perspective on "Dumbing Down" the Security Profession