Following the White Rabbit Blog : testing

Hybrid Analysis - The Answer to Static Code Analysis Shortcomings

Rafal Los — Thu, 15 May 2008 14:16:00 GMT

Hybrid Analysis - The Answer to Static Code Analysis Shortcomings

Given my previous article and the buzz it generated (both for and against the ideas I set forth)... I needed to hurry-up and write the follow-on article for "Static Code Analysis Failures". I've had so many conversations with people about Hybrid Analysis, and "why static code analysis fails" I've come to realize a few things, and wanted to share more of the base for my mindset...

Definition: Most of the folks I've spoken to (developer to security "expert") have a mis-guided idea of "static code analysis". With that in mind, here is the definition from the Wikipedia...

Static code analysis is the analysis of computer software that is performed without actually executing programs built from that software (analysis performed on executing programs is known as dynamic analysis). In most cases the analysis is performed on some version of the source code and in the other cases some form of the object code. The term is usually applied to the analysis performed by an automated tool, with human analysis being called program understanding or program comprehension.

Concept: The whole reason we (security professionals) have pushed to get a security tool into the "development" part of the SDLC is that it's easiest to fix defects when the code is still being worked on and built. We all know patching is a recipe for disaster, no one will argue that - so the idea in itself was sound.
Evolution: As the evolution of security marches on we have seen the evolution of "static code analysis" move forward as well. The concept of the traditional white-box testing strategy has gone from sending your code to a human being who would read it, line-by-line and provide an analysis of that code to programs which are essentially smart enough to build the code, trace "source-to-sink" and build data flow models and attack vector simulations. I am writing to contend that we have moved on in the next step of the evolution. Building the code and doing a "source-to-sink" trace with simulated attack scenarios is no longer good enough. (more on this in a minute)
Comprehension: It seems a few folks have missed the point of what I was talking about. I'm not saying that this type of testing (white-box, source-analysis, whatever you call it) shouldn't be done. Quite the opposite my friends - "code analysis" is vital to the success of any good security SDLC.
Solutions: Everyone quickly jumped on my case to defend whatever tools their company makes or uses... but I think I did my best to not attack any particular tool or vendor... hrmmm...

OK so now let's talk about what the answer to this whole mess of analyzing web-borne code is going to be. How will we, as security professionals, continue to be relevant to developers and the roots of the Software Development LifeCycle? I tell you that the future lies in what I can only call "Hybrid Analysis". This isn't a term I've developed, in fact, the term can be attributed to the developers of a certain software tool which is now distributed and built by the HP ASC group (to which I belong). Without turning this into a marketing pitch, I want to explain to you why the leap I made in the above bullet-point titled "evolution" is valid, and why you should care.

I love the quote I put in my first article so I think it's worth repeating... "Machines do not execute source code, they execute machine code"... absolutely true. So when people used to work with source code we can now understand why they only got the answers part of the time. Yes, I fully realize most modern source-scanners "or white-box scanner" tools don't just grep through code, as someone suggested at a blog posting that quoted me. In fact, I'll go out on a limb and say that to grep through the code for "vulnerabilities" is about as effective as anti-virus and IDS (both pattern-based recognition)... there are so many permutations of the bad/malicious that those tools are inherently broken. Anyway - to get back on track, I want to talk about this term "hybrid analysis".

What is Hybrid Analysis? (And more importantly, why do you care?) Hybrid analysis is the culmination and what I feel is the inevitable cross between white-box and black-box testing. I'm not sure if "gray-box" is the proper term or not, so I'll just keep using Hybrid Analysis for the sake of not mixing things up. Hybrid analysis analyzes the byte-code that a source-compiler generates and builds your standard "source-to-sink" data-flow model but then moves beyond the limitations of that approach by taking that "knowledge" of the application and submitting it seamlessly into a (modified) black-box scanner built into the solution.

Picture it - you have the blueprints of the bank vault, so you can try all the ways to theoretically break into the vault, then you hand those theoretical attack plans to a grunt who takes your information and goes and actually tries each of those attack vectors with multiple permutations and attack parameters to make the score and break in. That my friends, is the proverbial Holy Grail of application security testing. Data-modeling will only get you so far, before you have to actually try the attack to make sure it really works. Now, given the many parameters involved, for example, external libraries, compiler behavior, and so on that influence the way code actually behaves it's conceivable that you can accomplish this feat I've described without the hybrid analysis approach (the modified black-box scanner) ... but then I think you're looking at an incredibly complex and almost AI-driven analysis tool... and I simply don't believe that technology exists or will exist. I'm the kind of person that has to see something being broken before I'll believe it's real. Give me the proof.

If you take the hybrid approach - the proof looks you in the face each and every time. As a nice side-effect you can virtually forget false-positives! Source code scanners are infamous (notorious even!) for generating a lot of false-positives. This has always been one of the many reasons developers argue against using these tools. But what if you could offer your developers a way to eliminate those false-positives with little or no human intervention or double-checking? If you're still skeptical, I'll be happy to have someone from our team demonstrate how this type of approach works, yes - we have it exclusively in our toolset.

So let's recap. Here are all the ways that Hybrid Analysis will save the world (joking):

Reaches well beyond modern "source-to-sink" data-flow modeling for vulnerability detection
Addresses 3rd party components by reflection (analyzing byte-code or IL of DLLs, JARs, etc)
Provides a real validation of the theoretical attack scenarios that the above step generates
Virtually eliminates false-positives! This is a nice side-effect of testing using the Hybrid Analysis method

Static Code Analysis Failures

Rafal Los — Tue, 06 May 2008 17:32:00 GMT

Static code analysis failures are costing enterprises money and reputation.

White-box security testing is inherently a flawed proposition for many reasons -but it all comes down to a very simple concept:

Machines do not execute source code, they execute machine code (compiled code). --Paul Anderson (GrammaTech)

If you think this through for a minute you realize that there are a few specific reasons why the above statement fundamentally changes the way that people look at white-box testing, and why this is a losing proposition. Let's analyze this in the context of a web application project for a mythical online bank. Consider that the use-case here is that we are dealing with a bank that has an online presence (currently being analyzed) which will be integrated with a series of existing legacy applications, partners, and external 3rd party components. Given this information let's analyze why white-box analysis (or static source-code analysis) is doomed to fail this project with respect to security.

Compiler Optimizers Break Things - Think of it this way, compilers are designed to make machine code from your source code. That compiler's sole purpose (in most cases) is to create machine code that will be optimized, extremely fast-executing, but not necessarily secure. Often times security functions that people build into source code can be removed by compiler optimizers and most often without our knowledge. These actions often undo many of the advanced security features that developers may consciously insert into their code. Consider the following example:

Developer is paranoid about data-persistence in memory space, and wants to be doubly-sure that variables are expired and destroyed
Developer writes a routine whereby the variable will have a null value written to it before the memory is freed
Compiler optimizer sees this as a double-work scenario, and removes the null-value portion and simply opts to free memory
A potential security vulnerability is created with variable persistence in freed memory space

This example ideally demonstrates how a security vulnerability can be inserted in spite of the developer's best efforts to write secure code. Standard static-code analysis tools which are used to "scan code" at the static-file level will fail to catch this vulnerability. Quite simply - static code analysis fails if it is not supplemented with dynamic analysis.

3rd Party Library Integrations - There is another threat to developing and scanning static code in a white-box format. Inevitably, 3rd party libraries are used to complement features or functionality that are not natively provided by the local development effort. After all, no one re-invents the whole wheel everytime - we simply build what we cannot reuse from someone else's work, then use the publicly available libraries from 3rd parties to fill in the functionality and features that have already been written and (hopefully) tested before. White-box testing (or static code analysis) will absolutely fail in finding flaws when it comes to pulling in 3rd party libraries. By the definition of this type of issue, 3rd party libraries rarely provide you the source to be scanned and checked for weaknesses that will affect your application. What you're left with is someone else's code (in machine-compiled format!) which will be interacting with your application. Would you trust that model?
Static Code Analysis Rarely Understands Data-Flow Modeling (Data Tracing) - If you're scanning your application with a source-code-only analysis tool, you're going to not only miss things that will almost certainly come back to haunt you - but you may also be over-working yourself without a real purpose. Consider the following example to illustrate my point. Before I get into that example though, allow me to explain this idea of "data-flow modeling" for those that are not familiar with this idea.Data-flow modeling seeks to understand how data moves through your application, not just how the application code is written. After all, that's the whole pointn of the application, to work with data. Vulnerabilities lie in manipulating data either to or from the end users or the server(s). Data-flow modeling maps out the data in your appliaction from it's instantiation (maybe when the user types it in) to its resting state (maybe when it's finally written to a database, or handed off to another application or service for additional work). That being said let's consider a web application that has 1,000 forms across 100 pages written in the language of your choice, built to be AJAX. While each page does nothing individually to validate user input (the data source) all variables (data) are filtered through a central validation module deep within the application logic. A standard source-code analysis tool (I have evaluated this and can honestly say this is a real use-case but will not mention the tool) will flag on each and every input that is not validated (within the page) as vulnerable to hudreds of vulnerabilities ranging from XSS (Cross-Site Scripting) to SQL Injection and other attack types. What you are left with is a very lengthy report with hundreds of critical and high vulnerabilities that you now obviously must address... unless you do some dynamic analysis on the code and realize that *none* of those theoretical vulnerabilities are exploitable due to the fact that the application filters all data through the central validator/scrubber.

So, there you have it. Static code analysis is inherently doomed to fail. White-box testing of source-only is flawed. The sky is falling, global warming will kill us all. In my next installment of this column, I'll give you what you need to know to avoid failing in your security initiatives at the development step of the SDLC - remember, knowing is half the battle.

Stay tuned!

If this information disturbs you, and you would like to talk about it directly please don't hesitate to email me directly. I am not a sensationalist, and pride myself on presenting practical solutions to real-world problems which are realistically attainable. Thanks for reading.

Navigating the PCI DSS Standards...

Rafal Los — Tue, 22 Apr 2008 12:18:00 GMT

For those of you who keep up with the PCI DSS standard, the coucil today has issued an update titled: Information Supplement: Requirement 6.6 Code Reviews and Application Firewalls Clarified.

The standard item 6.6 has been further clarified in one of two options, as before, being either Application Code Reviews or an Application Firewall. I'll address the first option, since that is the more logical one, but will briefly talk about the Application Firewall as well - just to clear the air a bit. While the Standards Council continues to add clarification, which makes the standard more usable, more opportunities for compliance surface with less cost and effort. No doubt the IT manager feels like this is a good thing because now the cost of compliance won't necessarily be astronomical - and thereby make it viable. As we all know, the issue of compliance to a non-government regulation is always a balancing act. Compliance, as with most security components, is an equation balancing risk against spending and business value. We all know the results... if the equation balances just right, the business benefits from the added security and sees value while not spending more money than the risk is worth - and security feels worthwhile because risk has been decreased by some factor which affects the business in a positive manner. Granted, it doesn't always work out quite so rosy - but the PCI DSS standard is going a long way to make sure that these equations that happen every day, in many businesses throughout the world - balance.

Now - on to the meat of the standards update. First off, let me address the Web Application Firewall issue. While this is a topic that deserves a whole article onto itself, the short version is this - web app firewalls are very expensive, complex band-aids. That's the reality. While many of them work phenomenally well, and in fact I can name a few that do, they are difficult to implement into an existing production environment, they are primarily signature-based (remember how well we stop "unknown" viruses?), or have some other architectural quirk that makes them an impossibility in your enterprise. Personally, at my previous company I started to implement a particular WAF ... but it took over a year and a half of research, testing, approvals, and more testing to get them into a newly built environment... not even into a legacy production environment where they would have provided the most value. Anyway... the point is - a WAF is a tool you use when you don't have the resources to "do it right"... fix the code itself.

Section 6.6 of the PCI DSS standards, option 1 (the Application Code Reviews) now has 4 basic alternatives. Candidates are urged to implement at least one of the following...

Manual review of application source code
Propse use of automated application source code analyzer tools (static code scanners)
Manual web application security vulnerability assessment
Propse use of automated web application security vulnerability assessment tools

First, a few things of note. The above does not necessarily call for a "penetration test" which exploits vulnerabilities by an ethical hacker... only an "assessment" (which identifies but does not exploit) vulnerabilities is required. The distincion is important because it means that you can now do this on production code, or a production environment without the risk of damaging your applications by necessity to prove their vulnerability.

I find it interesting that the update goes and directly says "In all cases, the individual(s) must have the proper skills and experience to understand the source code and/or web application, know how to evaluate each for vulnerabilities, and understand the findings". The fact that the DSS requirement 6.6 now specifically addresses competence in the assessor should mean that there was some ... question... over the competence of assessors or possibly a need to specifically stamp out that only qualified people should be doing assessments. Interesting, at either angle. The same statement goes for assessors using automated tools - but now we have an interesting proposition. Do you (a) hire an extremely qualified application vulnerability tester, or (b) hire a knowledgeable user, and give him a software testing/scanning tool and some training... and are those even the same? Obviously the dollar amounts for the two are different... or are they? There is also the point about the testers having to be (the authors use the word "should be") organizationally separate from those writing the code... well that makes sense. No one wants the fox guarding the hen-house, right? You don't want the same developers that are potentially churning out insecure code to then review it and give themselves gold stars. So far so good.

So now we have 2 options for doing this internally - which will help our bottom line (external 3rd parties are typically very, very expensive)...

First option is to do an SDLC-integrated code review... actually reviewing the application code before it gets compiled and leaves the development group's control. We have the option to do it manually, or with some tools - using only highly trained and knowledgeable people.
Second option is to do a post-development analysis of the code. Once the code is written, built, and tested for usability issues it's time to security-test it with, again, either a human being, or some black-box testing tool(s) - but again, you must use trained and knowledgeable people here as well.

Well, if I'm a security manager...this is great! Thinking logically - you always want the security expertise in-house, and why in the world wouldn't you want to do application security throughout the application lifecycle? The DSS update also goes on to remind us of requirement 6.3, and the need to have an effective change-control process such that the security reviews are not bypassed, at any level. While the final sign-off must be done when the code is ready for production - it's imperative that the effectiveness of the application security policy be enacted to push as far back into the development (pre-development planning? requirements gathering?) lifecycle as possible (more on that in a separate article).

As a final note - the update talks about the need to stay current and abreast of new developments in application security testing. It's essential that whatever tools are purchased (whether they be the full SDLC suite from HP/ASC, or some other vendor) - that these tools and their users be continually updated from the brightest minds in the field. This is unfortunately a one-up battle against the "bad guys"... if you're behind you're sunk.

So what have we learned from the new Section 6.6 update?

There are at least 4 ways to interpret the "Application Code Review" guideline
You can have your code "reviewed" internally, as long as your assessor is trained and competent (but to who's qualifications?)
You can use automated tools, either static code analysis or black-box testing software, if you have your people trained in those tools, and application security
Your testers/assessors have to be organizationally separate from the development organization (but at what level?)
Your organization should absolutely integrate application security as early into the SDLC as possible, using "tools and rules" in combination
Your testers should always be up-to-date on the latest developments, techniques, and methods... otherwise you're bringing a knife to a gunbattle

Thanks for your attention!

"Security Vulnerability" != "Defect" ; why?

Rafal Los — Tue, 01 Apr 2008 11:18:00 GMT

It's one of those obvious things. A defect is a defect, right? Whether the airbag is faulty, or the gas cap doesn't hold pressure... a defect is a defect. The strange thing is - it hasn't been that way, and still isn't that way, in most of the IT shops I've been in. Why?

The reason is simple. Historically, security vulnerabilities have been in a class all their own. In an attempt to put some urgency to the matter, security professionals have labeled defects in the security of their projects (in this case I'm talking about web applications) as an entirely different thing than a functional defect. What we didn't realize is that we were actually doing a dis-service to ourselves and the security cause. You may not agree with me right now - but I'll explain this more clearly, and I think you'll be on board with my thought process.

Let's talk about defects, in general and then apply it to the matter at hand. First, let's identify what a defect is... A defect is, in the dictionary sense (cut from dictionary.com):

de·fect [n. dee-fekt, di-fekt; v. di-fekt] Pronunciation Key - Show IPA Pronunciation

–noun

1.	a shortcoming, fault, or imperfection: a defect in an argument; a defect in a machine.

2.	lack or want, esp. of something essential to perfection or completeness; deficiency: a defect in hearing.

OK, easy enough right? So the first meaning is clear; a defect is a shortcoming, fault, or imperfection. It is reasoned that a defect in a web-based application results when functionality X doesn't work as required. Say you have a button, and the functional specification (we'll get back to this gem in a minute) calls for the button to perform some action, A. During the testing phase of the application, before release to production, a tester or tool is utilized to test the functionality of that button, but instead of action A happening, some other action B happens. This is a defect. There is no doubt in anyone's mind that this immediately gets classified as a defect, put into the defect tracking system and sent back to the developer for remediation. The defect is classified as a higher-priority defect if the function happens to be one that is showcased, or important to the overall functionality of the application. Those of you that already use the HP Quality Center tools know exactly what I'm talking about, and know how this process works. Here's the strange twist though - why is quality testing only done with good data? I understand that you want to make sure that the test cases work properly - but why are the testing options limited? The issue at hand here is a very narrow view of defects, and defect testing.

Back in college, I took very basic programming class and had to write a program that was a calculator. It would ask for two inputs of numbers, and then give you an option to perform either an addition, subtraction, multiplication or division of the inputs. Generally, it was assumed that these would be numbers, but what if they weren't numbers? Most of the students in the class, myself included, never thought about ... "What if someone enters a letter or some other unexpected input?" Well, luckily, the professor chose my application, put it up on the screen for the whole class to see, and promptly entered a and b for the two inputs and tried to add them. When my application core dumped, he explained to the class why I had gotten my first F on a project. I learned a very valuable lesson that day - developers must brace their applications for unexpected input. "Why would anyone want to enter something other than numbers?" wasn't a good enough answer to explain why my application failed. Let's apply this lesson I learned back in college to today's application programmers and functional testers.

Here are the reasons why I think security vulnerabilities aren't seen as "defects" in general...

Security professionals have insisted that a vulnerability is its own separate category - While it is true, some security vulnerabilities are a whole new level of "bad" they should be considered just like any other defect in the application for the sake of tracking and remediation. Web platform managers are generally concerned with meeting the demands of their customers and producing code that is defect-free - and it's our own fault that "vulnerabilities" of the security variety have become some ethereal, magical issue for security nerds to worry about. This matter can only be fixed by changing the naming back... a vulnerability is a defect, period. Security vulnerabilities must be explained as "high-criticality defects" to developers, managers and customers otherwise this situation will never change.
Functional specifications rarely, if ever - call for for security validation - Functional specifications aren't written by security professionals, generally. At best, security professionals have a chance to review the functional specification way too late into the process, while the code is being written and readied for production. This is, once again, our own fault most of the time. The answer to this dilemma is a two-pronged attack. We as security professionals must educate those that write functional specifications, and enlighten them to the need for security features. At the same time, we must work hard to have an active input in the writing and release of functional specification documents. These two vectors are critical to getting secure code as an end-product.
Programmers don't typically think about malicious users - While it would be great to think that one day all developers will inherently write secure code because they will have "learned" what security is, and how important it is to the application - the fact is that it's a pipe dream. Developers care about one thing... meeting functional specifications in the least amount of time possible, and moving on to the next project. Developers like to write optimized code that accomplishes the required tasks in as little time as possible. Solving #2 above will also partly solve this problem. In addition, developers must be given the tools (such as static and dynamic code analysis tools as plug-ins to their IDEs) to make their jobs easier. It is not reasonable to expect developers to be security experts in all aspects, so we must arm them with the tools to be experts, without having to do too much extra work or they won't use those tools.

So what have we learned today?

Security vulnerabilities must be re-classified as easily-understandable "functional defects"
Funcitonal specifications must be written to include provisions for security validation
Quality professionals must be given the tools to test for "security defects" in web applications to close the loop in the lifecycle
Developers must be educated and also given the tools to write more secure code with minimal additional effort

-- I welcome your comments!