Imaging Challenges, Part Deux

StevenSimske — Sat, 20 Sep 2008 11:30:00 GMT

In a blog post earlier on this busy blogging week (hard to tell I'm spending a lot of time rotting in airports/hotels, no?), I introduced some of the difficulties in image clustering, or aggregation. This post introduces some of the broad approaches used to solve such imaging challenges.

Broadly, there are at least three classes of image analysis technologies used to compare photos (more broadly termed "images" to include scanners, cameras, inspection systems, video, and all other forms of "image capture"):

1. Machine vision/pattern recognition

2. Segmentation-based approach

3. Image modeling

There is some overlap among these three approaches, but they are distinct enough to proceed.

1. Machine vision/pattern recognition: This approach typically uses correlation (a statistical measure of image similarity) to compare two images. Correlation can be used to compare texture, frequency, color, shape and/or other content in the two images compared. The use of frequency-based (looking for how an image varies spatially in one or more directions) comparisons allows images of different scale (size of features) to be readily compared by simply scaling the frequency outputs relative to one another. Best matches provide the scale differences and subsequent alignment of the relatively scaled images. Machine vision systems usually are initialized through "training" the system by capturing one or more images of a calibrating (or "ground truth") feature to which the other image(s) is(are) compared. Clearly, such pattern recognition or "machine vision" based systems are especially amenable to inspection, wherein many images are to be compared to the calibration image. This type of comparison is especially effective for comparing Pictures #1 and #2 in the "Imaging Challenges" post of two days past.

2. Segmentation-based approach. Image "segmentation" is the process by which an image is divided into regions, called segments, which can thereafter be used as individual images for comparison. Therefore, this approach is recursive inasmuch as it affords refined segmentation, or sub-segmentation, as further information needs to be extracted. Images which contain steganographic (hidden) information such as digital watermarks, fiducial marks, and the like, are effectively analyzed by these approaches. Note that this approach is also taken on the "storage" side for many compression approches, such as tiled JPEGs. This segmentation-based approach can be used to cluster Picture #3 in the "Imaging Challenges" post with Pictures #1 and #2.

3. Image modeling. An image model is a description of salient features for the image analysis algorithm to find in the image. This may be conveyed through a template (description of layout), a feature set, or other means. An "image model" assumes that the image processing system is capable of "image understanding", meaning it is capable of accurately deciding whether or not an image contains a match to the model defined. Such a system relies on powerful statistical classifiers to decide on a "yes" or "no" for the match. Clearly, the goal is to have 0% false positives (no regions identified as matching that actually are not matches) and 0% false negatives (no regions that actually are matches but are missed, or unidentified). In reality, though, the system is tuned to provide the best overall accuracy based on the task at hand. If the cost of a false positive is much higher than the cost of a false negative, then the system should be tuned to favor false negatives (miss some matches) but have very high confidence in the matches. This is usually the case for the reading of security-related information [in future blogs, I will discuss classification in greater detail]. Note that, if each GPS location has a "model" for the images that can be captured (such as a 3-D panoramic), there is the possibility that Picture #4 in the "Imaging Challenges" post could be aggregated with Pictures #1-#3 (even though the content in the Picture has no overlap with the other 3). Currently, however, except in expensive systems for high-security locations, these models do not yet exist. But, they will in the future (think about 3-D video games, for example, in which a 3-D panoramic model is created for various "rooms" in the game).

-Steve

Imaging Challenges

StevenSimske — Thu, 18 Sep 2008 11:57:00 GMT

"Imaging" is a broad term meaning the ability to transform, interpret and/or associate an image. Sounds pretty easy, right? But when you consider what is actually involved, it's pretty hard. Most modern digital cameras (and other photo-capture devices) are packed with a host of "automatic" imaging, such as noise removal, contrast/exposure enhancement, etc., up to red-eye removal and photo album aggregation. All of which implies there are (hopefully reliable!) algorithms available to improve the quality of an image (and thus improve its value).

However, the way you will use an image (called its "workflow") impacts what algorithms you will use to clean it up, improve its quality, and otherwise transform it. Normal metrics for "image quality", for example, are not as important in security printing and imaging as are the more arcane concepts of inspectability, authenticability and forensics-capability. Consider, as an exemplar, if I have added information--such as a 2D bar code or a digital watermark [hidden, or "steganographic" information]--to an image, then whatever I do with my imaging should be focused on helping me reliably extract that information rather than improving the aesthetics of the image.

How hard can that be, you ask? Let's look at four photos that each contain one set of identical information; namely, the GPS location that the image was taken from.

Picture 1 is the new Sao Paulo bridge by day, from the 31st floor:

Picture 2 is the same bridge, from the same floor, with a different aspect ratio:

The third is taken from the same GPS location, 30 floors lower, and at dark:

And the last image is taken from the same GPS location, 6 stories up, in the opposite direction:

What information do these pictures have in common aside from the GPS location? Picture 1 and 2 are pretty similar to the human viewer, but to the imaging algorithm have a number of distinctions. Perspective, aspect ratio, contrast and exposure all differ considerably. Most image-clustering technologies, however, can aggregate (find similar) these two. But if the bridge were watermarked, would the photos equally represent those watermarks? Would the same bridge in Picture 3 also aggregate with Pictures 1 and 2? Humans would say yes, but machine algorithms are not so sure. Picture 4, not a chance for the machine (or for any human who did not have the memory of both images), without the GPS information.

This represents (somewhat figuratively) some of the challenge involved in security imaging. In practice, we are not usually required to associate Picture #4 with Pictures 1-3 except through metadata (image header) search. In future blogs, I will discuss how the imaging is actually able to work (and work well!) on Pictures 1-3.

-Steve

Security Printing and Imaging : authentication, imaging

Imaging Challenges, Part Deux

Imaging Challenges