Hi Rafal,

Having worked in both QA and security fields, I can tell you that there is no consensus for these definitions -- even among experts within the same field.

The problem stems from the overlapping problem set. Picture a Venn diagram of two equal-sized circles that have a 95% overlap. One circle is "vulnerabilities" and the other is "defects".

The main distinction is based on intent. If the box is supposed to be green but it appears blue, then it is a "defect". If the box is supposed to be green, but the user can make it turn blue, then it is a "vulnerability". Any defect can be a vulnerability if it can be applied toward an exploit. This includes spelling and math errors, as well as memory leaks and remote command execution.

One common mistake made by non-security people is an assumption about an attacker's goal. Don't assume that the attacker wants to take something ("but there is nothing of value there"), destroy something ("but they can't modify that!"), or relay something ("but they can't get back out to the Internet from there"). There are plenty of other goals. (My favorite, and usually overlooked goal, is "distraction".) Without knowing your attacker, you won't know what they want. Don't assume and don't waste energy listing every possible motive (because you can never list them all). Similarly, you don't know where the attacker is coming from: internal, external, or a combination of both.

The thing to remember is that code without defects will have very few -- if any -- vulnerabilities. "Secure coding" is a (mostly) subset of "Robust coding".

Then there is the issue of finding defects. Software marketeers have bastardized the definitions of "alpha" and "beta" code. The terms actually come from statistics. (Good reference: www.intuitor.com/.../T1T2Errors.html) An "alpha test" or Type-I error, means that the code does not do what it is designed to do. If the calculator should add numbers, then "1+1=3" is a Type-I error -- it fails to do what it is supposed to do.

A "beta test" or Type-II error is much more difficult to identify. While "alpha" code should do what it is expected to do, "beta" code should *not* do what it is *not* expected to do. If the calculator should add numbers, then what happens if you give it a letter?

While alpha errors usually lead to "defects" and may present vulnerabilities, beta errors usually lead to "vulnerabilities" and account for most big security risks.

This is also where the terms "hacker mentality" or "hacker factor" come from. It is the mentality of thinking outside the box and using software in ways that it was not originally designed. Most developers and QA people focus on what it should do -- the Type-I errors. The Type-II errors are usually overlooked. (If those QA people cannot immediately think of a dozen different uses for a hammer -- besides hitting something -- then they definitely lack the hacker mentality. And don't say "paper weight" -- I called that one.)

Neal Krawetz, Ph.D.

Hacker Factor Solutions

http://www.hackerfactor.com/

Author of "Introduction to Network Security" (Charles River Media, 2006) and "Hacking Ubuntu" (Wiley, 2007)

ps. This web form has a Type-I error. The comment submission form says "Type the digits above". However, the captcha image shown contains letters and not digits.

We have been doing this training QA people around 8 years now with good/excellent results, so a bit suprised this is realigning. We are Fortune Global 200 oompany (don't want to be more specific as it reveals the company

interesting ... to echo your auto maker's concerns in say a healthcare manufacturing environment security problems are indeed a concern - but patient safety is paramount. security problems that demonstrate a clear effect on patient safety get attention.

even changing your terminology may not be enough. look at the definition of the words defect vs vulnerability: defect speaks to lack of completeness whereas vulnerability speaks to lack of defense. this shows a broader problem - that is in defining "something" "completely" often missing are provisions for proper defenses. for QA to get interested the vulnerability has to map to a defect and to be a defect the system's requirements must be affected. subsequently the requirements are reflections of the stakeholders wants/desires... (traceability). so the stakeholder has to want a robust system. ...major shift in thinking.

defect may get QA listening, but integration throughout the sdlc is really needed.

@neal - type1 error? perhaps.. that is assuming the word digit is not referring to say a base64 (or base37) system ;-)

I tend to take a similar tack on this issue.  there is no such thing as a vulnerability, only defects in the information security requirements documentation process.

Vulnerability:  A method or condition by which or in combination with other conditions/vulnerabilities,  information security requirements ma not be met.

Defect = Not meeting explicit business or underlying technical requirement.

Key gap:  Making sure Information Security becomes part of the business requirements gathering process.

If application security is explicitly stated in a requirement (e.g: "Application interface must be built in a manner to prevent unauthorized access to personal identifiable information of any individual's data,")  then a vulnerability such as SQL injection or access control issues are now in play, and during requirements decomposition,  we see explicit requirements being defined such as hardening against code insertion, buffer overflows, input and output filtering, etc.   The intent is that information security requirements must be up front, expected and aligned with functional business requirements (in our space, these requirements are owned  Compliance and Regulatory and Risk Management Corporate stakeholders), and not after the fact.

A Defect is quite simply, something the business/purchaser of the application/system implicitly or explicitly doesn't/can't have happen.

A data breach is something our business doesn't want to happen, and our Risk and Regulatory Officers have to be at the table when requirements are exposed.  Once signed off at the business level, then you have the traceability to declare a 'vulnerability' as a defect, when someone exposes one through penetration testing, or worse, security 'incidents'.

But the key is... never let testers or developers define vulnerabilities as something other than a requirement.

I saw a comment today on an e-marketing website which referred to a defect/vulnerability as a "usability flaw":

econsultancy.com/.../3346-ryanair-freaks-out-at-blogger-disses-wordpress-shoots-foot

It was interesting to see the security problem being defined in a way that marketers could relate to, and reminded me of the discussion here.