Research, Technology, and Teamwork blog by Susie Wee : research

From cameraman to camerawoman in NIR imaging research

susie.wee — Sat, 14 Nov 2009 21:49:00 GMT

In the world of image processing research, it is important to have standard test images so people can compare their results. The cameraman image is a test image that has been used for decades. It can be found in many image processing textbooks and homework...(read more)

I'm back: From research to business

susie.wee — Mon, 09 Jun 2008 01:02:00 GMT

Sorry for my long break from blogging. Thanks to JCS for asking me to come back, as it was just the push I needed to get this post out. A few other readers have also noticed my absence from the blogosphere, thanks to all of you for your encouragement and support.

My silence does not mean I've been idle and it does not mean I've run dry on topics. In fact, it's been quite the opposite. My recent experiences over the last few months have given me plenty of reflections to write about, but very little time to write them! The biggest news I have to announce is that I changed jobs within HP. I moved out of the research lab and into the business.

I am now leading what I call the newest startup in HP--the HP Experience Software Business (ESB). ESB is within the Emerging Businesses unit in HP's Personal Systems Group. I coined the term "Experience Software" to refer to software that drives and is driven by the user experience. I have found that taking an experience-centered approach to software and business is as important as taking an experience-centered approach to design, research, and technology (which I've written about in the past). Putting the experience first is really a mindset that affects everything you do, ranging from deciding what feature you put in a product to asking your friends what they like or dislike about their coffee cup. I want to thank my experience-driven researchers in HP Labs for teaching me the importance of experience over the last few years- they knew the importance of this long before it became fashionable!

While I am no longer in HP Labs (after almost 12 years!), I am still very involved with the Labs. I am on one of the advisory boards that is responsible for approving, guiding, and reviewing the research projects. In fact, I rely on HP Labs research even more now than I did when I was a lab director there. I feel very fortunate to be able to work with the researchers- to get consulting advice in their areas of expertise, to listen to their newest and oldest ideas to inspire ideas for my new business, and hopefully to bring some of their great ideas to market!

I'll try to post more now that I have the monkey off my back regarding my silence in the blogosphere. I plan to keep my posts in the same flavor unless you suggest otherwise. (Any suggestions?) And, while I can't reveal everything we're doing, some posts will give hints about the areas that we're thinking about. In the mean time, please wish me luck with ESB!

p.s. We have a new blogging system that makes it easier to leave comments. Like any other blogger, I love to hear what my readers are thinking; so please feel free to leave comments.

A cute workshop

susie.wee — Sat, 05 Jan 2008 10:11:00 GMT

I came across an announcement for the following workshop: Workshop on Designing Cute Interactive Media in conjunction with the ACM conference on Designing Interactive Systems.

ACM is the premier professional research society for computer scientists. I think it is quite a statement that the broader research community is recognizing design, experience, and human emotion as bona fide research topics. Cuteness is being recognized as research by the research community!

In my mind, user adoption is the ultimate indicator of a technology's success, and adoption is driven by having a great user experience. The research discussed in workshops like these will help us understand and eventually formalize the coupling of experience and technology. Understanding how to provoke human emotions like cuteness will help identify new research directions and drive technology adoption.

Congratulations to the researchers who were pushing these ideas in their work before it reached broader acceptance! For example, 2007 was the 25th anniversary of CHI. Clearly your efforts are paying off!

What do you think? Is studying "cuteness" research?

Feel free to leave a URL with your comments.

The Life of a Packet for Mobile & Media Experiences (Packet Video 2007 Keynote)

susie.wee — Tue, 13 Nov 2007 01:02:00 GMT

I gave a keynote talk to kickstart the Packet Video 2007 Workshop in Lausanne, Switzerland. The audience was great, and the talk seemed to generate lots of discussion during the Q&A and for the remainder for the workshop. Here's a recap.

Abstract

Mobile & media experiences connect people with each other, with information, and with their environment. Media is increasingly being delivered in packets over networks. This raises a number of questions for today's networks:

How can we transport media packets?

How can we adapt media packets for diverse clients?

How can we protect media packets?
A number of emerging applications will impact future directions for packet networks. We also discuss the following questions:

What impact do globally distributed, immersive media environments have on media packet delivery systems?

What role does context play in next-generation mobile media experiences?
We consider these questions from the perspective of a user and the perspective of a packet.

Coupling experience and technology

I began by stressing the importance of coupling experience and technology. Rather than developing technology in a box, it is important to first consider the desired user experience and then develop the technologies that impact it. The most important factor for deciding whether a technology gets transferred to product is not how good the technology is, but rather how it impacts the user experience. I have been passionate about this theme for quite some time, and as time passes my passion for this only grows stronger.

The rest of my talk cycled between the following experiences and technologies.

Mobile & Media Experiences

Experience #1: Mobile, Diverse, Interactive: Diverse mobile video clients, desktop video, living room video
Experience #2: Immersive, Conversational, Worldwide: Halo collaboration experience, Panoply immersive gaming experience
Experience #3: Pervasive, Personalized, Context-aware: Mediascapes context-aware multimedia experience

Packet Technologies

Packet labeling & metadata
Transcoding & Processing in the network
Scalable Streaming
Secure Scalable Streaming
Multiple Distortion Measures
Public & private domains
Sensing context in the network

The first five technologies were discussed in the context of Experience #1. The last two were discussed with Experience #2 and #3.

Experience #1: Mobile, Diverse, Interactive

Packet labeling & metadata: The main point is that we live in a distributed networked world where media packets will traverse distributed network elements with multiple owners and administrative domains and be processed by devices and equipment made by different manufacturers. In this highly distributed world, one important thing that we can do is smartly label our packets in hopes that over time the smart network elements along the way will use these labels to improve the overall quality of the user experience. The key design principle is to design packet labels that are 1) specific enough to be useful and 2) general enough to be understood.

Example packet labels and metadata include:

Importance: Distortion values
Time requirements: Time stamps
Content type: Video, audio, text, data
Scalability: Is it truncatable?
Media attributes: spatial region, resolution, color; audio channel
Dropability: Can it be dropped? e.g., Drop video for audio-only session.
Processibility: Is it transcodable? Can it be processed?
Security: What are the rights and privacy implications of the media?

The research challenges are designing and standardizing the labels with the design principle above, and then developing algorithms that use these labels for delivering improved mobile media experiences. These algorithms should be evaluated for their performance gains with respect to the label overhead.

Transcoding & Processing in the network

I discussed the experience of delivering media to and from users over any network and on any device. This motivates the technology of performing transcoding operations in the network. In 3G networks, the streaming, recording, and transcoding capabilities can be performed by the IMS Multimedia Resource Function (MRF), which serves and receives the media packets to and from the handsets. Dynamic transcoding can be used to adapt the video for the target client device (e.g., to lower the resolution) and for the network (e.g., to seamlessly handoff media between 3G and 2.5G networks during a mobile media session).

The research challenge that lies ahead is designing and developing transcoding algorithms in a manner that is computationally efficient so that a single transcoding node (e.g., IMS MRF) can process many streams at once to serve multiple clients at one time.

Scalable Streaming

This brings us to a technology called scalable streaming that makes transcoding much more efficient by leveraging scalable coding methods. In essence, if scalable coding methods are used, then we can form scalable packets that pack scalable data, for example low, medium, and high resolution data, into the packet in a manner that allows it to be transcoded by simply truncating the packet. Furthermore, the scalable media packets can have packet labels that contain image metadata and truncation points that can be used by a scalable packet transcoder. The scalable packet transcoder is quite simple- it performs transcoding by simply reading the packet label and then truncating the packet as needed.

Research opportunities arise if the packet labels contain the distortion value of the particular media packet. If distortion values are included in the label, then they can be used as hints for rate-distortion optimized streaming algorithms and rate-distortion optimized transcoding algorithms to improve the quality of the user experience.

Secure Scalable Streaming

Another desired experience includes serving diverse clients while having end-to-end security. End-to-end security means that the media is protected in a manner that only allows the sender and allowed receivers to access the media, while delivering, storing, and transcoding the media packet over the network in a way that does not require decryption. It turns out that this can be achieved by using the same method as scalable streaming, where scalable packets are formed by leveraging scalable coding, and then coupling the packet formation with the encryption process. Specifically, encryption is applied to the packet in a manner that allows the packet transcoding operation to still occur by simple packet truncation. This can also leverage secure scalable image coding standards such as the newly created JPSEC standard for security of JPEG-2000 imagery.

Secure Scalable Streaming was published in ICASSP 2001 by Susie Wee and John Apostolopoulos.

Multiple Distortion Measures

I then described a new technology area that we are studying called Multiple Distortion Measures (MDM). This begins with the following observation: Consider a set of scalable media packets. Generally speaking, the best ordering of the packets is determined by the profit-to-size ratio (or distortion-to-size ratio, in tech terms, delta d over delta r). Surprisingly, we observed that the best ordering for low resolution display is NOT equal to the best ordering for high resolution display. The question that arises is how different are they?

I showed a graph from our ICASSP 2007 paper that shows the PSNR vs. Rate plot for the low resolution reconstructed image with packets ordered in the low-res optimal order and with packets in the high-res optimal order. It turns out that there are differences in performance of up to 4 dB. The graph aso showed the PSNR vs. Rate plot for the high resolution reconstructed image with packets ordered in the high-res optimal order and the low-res optimal order. It turns out that these can have differences of over 1 dB.

This raised a lot of interest from the crowd. I think we'll have lots of people researching MDMs in the years ahead.

This raises the idea of labeling scalable media packets with multiple distortion measures, specifically, with the distortion value of the packet with respect to the low resolution image, the medium resolution image, and the high resolution image. If the packet contains this information, then streaming algorithms can be developed to optimize the media delivery experience to users with diverse client devices.

Multiple Distortion Measures was published in ICASSP 2007 by Carri Chan, Susie Wee, and John Apostolopoulos.

The Future

The last part of the keynote focussed on experiences #2 and #3 to look at the impact of emerging applications on future packet networks.

Experience #2: Immersive, Conversational, and Worldwide

Delivering immersive, high-quality, worldwide experiences has a number of challenges for today's networks. The main problem is that network intelligence exists, but only in spots. For example:

QoS exists in spots, but is not guaranteed from beginning to end.
IPv6 exists in spots, but it is often tunneled over IPv4 and so is not available from beginning to end.
Significant congestion can occur in peering points between administrative domains, and it is very common for packets to traverse administrative domains many times in a single session.
Due to the sheer number of IP addressses, packets in countries such as India may go through many network address translations (NATs) before being delivered to the recipient.

Public & private domains

As a result, proprietary networks are being built to deliver guaranteed experiences. HP's Halo immersive collaboration experience is built on a proprietary network for that very reason.

In the long run, the right answer is to build out networks that contain IPv6 and QoS. However, until that occurs, there is likely to be a co-existence of public and proprietary networks.

This raises research opportunities of developing protocols and algorithms that improve media delivery over co-existing public and proprietary networks. This also motivates the need to develop packet labels that contain information that can be used by smarter network elements that understand them. And, this once again raises the design principle of designing the labels so that they are specific enough to be useful but general enough to be widely understood.

Experience #3: Pervasive, Personalized, Context-aware

Finally, I described Mediascapes as an example of pervasive, context-aware multimedia experiences. The main essence of Mediascapes is that it uses sensors to trigger multimedia experiences tied to your physical and personal context.

Sensing context in the network

This raises the question of using sensors to sense your context and getting the sensed context into packets that can be used by different applications and services. In the web world, the sensors may exist as GPS sensors, environmental sensors, or personal sensors. In the operator world the sensors may come through carrier-grade network elements as in IP Multimedia Subsystem (IMS) architectures. For example, IMS context can include location, presence, group lists, and subscriber info.

The key is to have the sensors provide context that is wrapped into packets in a manner that they can be easiliy used by applications and services. This raises the challenge of creating a semantic representation for sensed context. Again, like the packet labels, this must be designed in a manner that is both specific enough to be useful but general enough to be widely understood.

Acknowledgments

I'd like to take a moment to give special thanks to thank John Apostolopoulos, Carri Chan, Steve Froelich, Dave Penkler, Qibin Sun, and Zhishou Zhang for their contributions to various parts of this work!

Final note and questions

The audience was great and the talk seemed to generate lots of discussion throughout the workshop.

This was a fun topic to put together for the keynote and I'd like to develop it further. I'd love to hear your thoughts and ideas on any aspects of this.

What are your thoughts and comments on the life of a packet?
Did you attend the workshop and keynote? If so, what did you think?
I'd like to develop this further. Do you have any suggestions for improvements?

Please feel free to leave a URL with your comments.

Idea request: The Life of a Packet

susie.wee — Mon, 29 Oct 2007 07:53:00 GMT

I'll be giving a keynote talk at the Packet Video Workshop in Lausanne on Nov 12-13, 2007. The title of my talk is "The Life of a Packet". More specifically, I will be focusing on "The life of a media packet" and "the life of a packet as it relates to mobile & media experiences".

I have my ideas on what to talk about, but I'd love to hear your ideas as well.

What would you expect/want to be covered in this talk?
What would you expect/want to learn from this talk?

Thanks in advance for your inputs!

UPDATE!

It's been quiet... so I'll frame this up a little more. Here is the talk abstract. I'd love to hear your thoughts!

The Life of a Packet for Delivering Mobile & Media Experiences

People are mobile, and people interact through sight and sound. Multimedia communication can be used to provide mobile & media experiences that connect people with each other and with their environments. In these systems, media is delivered in packets. The use of media packets raises a number of questions that the research community and industry ha ve been tackling for decades.

How should media packets be transported across a packet network?
How can one deliver and adapt media packets for diverse media clients?
How does one protect media packets in a packet network?
What impact do globally distributed, immersive media environments have on media packet delivery systems?
What role does context play in next-generation mobile media experiences?

In this talk, we will consider these questions from the perspective of a user experience and from the perspective of a media packet.

One more update: Here is a link to the post that describes my keynote talk.

The first ACM Mobile Video Workshop

susie.wee — Mon, 01 Oct 2007 03:52:00 GMT

I was just in Augsburg, Germany to give a keynote talk at the first ACM Mobile Video Workshop in the ACM Multimedia 2007 conference. The workshop chairs were Professor Eckehard Steinbach of the Technische Universität München and Professor Chang Wen Chen of the Florida Institute of Technology. It was a single-track workshop with interesting talks by researchers in academia and industry.

There were three keynote talks:

Industrial Perspectives on Mobile Video by Minoru Etoh of NTT DoCoMo in Yokosuka Japan
Mobile Video Transmission by Thomas Wiegand of Heinrich-Herz-Institut in Berlin, Germany
Mobile and Media Experience and Technologies by Susie Wee of HP Labs in Palo Alto, California, USA

Etoh-san gave some great insights into the future of 3G and 4G mobile networks. He provided the operator's perspective on what is likely or unlikely to be implemented in future mobile networks. He gave researchers hints on what areas to do research given the real-world constraints of operators.

Thomas provided an overview of the H.264/AVC video coding standard and the evolution of the Scalable Video Coding standard (SVC). He also spoke about some recent research he is doing on multi-dimensional layered FEC. He also reminded us about how the H.264/AVC standard was just a theoreticians dream 10 years ago, but now is planned for deployment in nearly every video system in world.

I spoke about the importance of coupling technology with experience design, and I described a number of HP Labs projects including Halo, Conversa, and Mediascapes and briefly talked about the JPSEC standard. I also showed the ever-popular Roku video that shows what we see as the future of Mediascapes.

I had plenty of time for questions at the end of my talk, and there were some very good questions ranging from technologies to experiences. The last question was from a researcher asking for tips on technology transfer, and during the rest of the day a few other researchers wanted follow-up discussions on technology transfer and how to have impact with their research. I was quite pleased to see the research community so interested learning more about this.

There were also many interesting talks in the session given by researchers who submitted their recent work. Lots of good ideas are percolating for next-generation video systems.

Eckehard and Chang Wen closed the session with a few words about the useful interchange between industry and academia, and once again reminded us of how research moves from theory to practice. Many of the participants agreed that this was a useful workshop.

Did you attend the Mobile Video workshop?
What did you think of the workshop and the various topics that were presented?

Please feel free to leave a URL with your comments.

ICME panel on HCI for Multimedia Communications

susie.wee — Mon, 09 Jul 2007 07:30:00 GMT

Last week I was on a panel on "HCI for Multimedia Communications" at the ICME 2007 conference in Beijing. The organizers and panelists were as follows:

Title

HCI for Multimedia Communications

Organizers

Qiong Liu, FX Palo Alto Laboratory, USA
Zhengyou Zhang, Microsoft Research, USA

Panelists

Zhengyou Zhang, Microsoft Research, USA
Tom Huang, University of Illinois at Urbana-Champaign, USA
John Weng, Michigan State University, USA
Susie Wee, Mobile & Media Systems Lab, HP Labs
Arnold Smeulders, University of Amsterdam, the Netherlands
John Smith, Watson Research Center, USA

We kicked off with each panelist stating their position on the topic, and then went into Q&A with the audience. The panelists contributed a variety of perspectives and the audience raised many interesting questions. Below are a few of my thoughts that I shared in the discussion.

Coupling experience with technology: When we think about multimedia communication, we should start from the user experience, and then think about the technology to enable it. There should be a close coupling between experience design and the enabling technologies.

Existing communication experiences: Technology can recreate existing communication experiences in situations where it is not possible. For example, technology can recreate effective face-to-face conversations for people who are geographically distant, as in HP's Halo virtual collaboration system.

New communication experiences: Technology can also be used to enable new comunication experiences for people. For example, instant messaging, blogging, photo sharing, and social networking are examples of new types of communication experiences that did not exist without the enabling technology.

Experience- Many simultaneous communication threads: HCI for multimedia communications should consider the new ways that people communicate, and specifically account for today's multitasking world where many communication threads are used simultaneously. For example, it is common for people to be reading email, browsing the web for research, getting instant messages, and getting phone calls simultaneously.

Experience- Share experiences instantly: As I was in Beijing, I constantly took photos and uploaded them to Facebook for my friends and family to see. They were able to share my Beijing experience while I was still there. We take this capability for granted today, but note that this was not so easy to do a few years ago.

Experience- Anyone can publish instantly: On top of sharing photos instantly, I can also publish instantly with my blog. In other words, I can create a publication at the click of button for the world to see without asking anyone for help or permission. This has a few implications:

I can publish on a wide variety of topics. At one time, I imagined all my publications would be technical. Now I have publications called "Blogging on the Beach", "Team sports and work", and "Top 10 Tips for How to Talk in Groups".
Since so many people like me can publish instantly, there is information overload and a lot of irrelevant information out there.
On the other hand, the fact that there are so many publishers greatly increases the chance that there is something out there that is useful to you and perhaps written by someone who shares your perspective or background.

People tag and vote: We are in a world of too much information and a scarcity of attention, so the question is how do you find what you want. Search and analysis are important. Tagging and voting can help. So, the advice for multimedia communications HCI is to use the proper balance of analysis and user feedback. Don't spend too much time analyzing if you have a situation where you can just go out and ask for user feedback. Since we are now in a world where people can tag and vote, use it when appropriate!

Measure quality of multimedia communications HCI by usage: Since there is a scarcity of attention, I think the best way to measure quality in multimedia communications HCI is by measuring usage. My advice is to build prototypes, attract users, and study their usage. You will learn a lot by doing this, including what is important, what are you missing, and what are users interested in.

How do we measure quality in multimedia communications HCI? Quality metrics should consider the multi-session nature of the communication experience that exists today.

How do we measure quality in multimedia communications HCI? Build prototypes. Attract users. Track usage (e.g., google analytics), user feedback (e.g., diggs), and quantitative metrics (analysis).

What will be an ideal interface for multimedia communication? An ideal interface will be simple, intuitive, and easy to use. You shouldn't need to read a manual to figure out how to use it.

Should the ideal interface have intelligence behind it? Only if it has predictable behavior. The interface must be predictable so users can learn to use it.

Should the interface have a human form? Not always. An HCI system for medical surgery should not have a false human interface.

What are your thoughts on the topic of HCI for multimedia communications?
Did you attend the panel discussion? If so, what did you think of it?
What advice do you have for the panel organizers and panelists?

Tags: HCI for multimedia communications, communication experience, human computer interaction, interface, IEEE ICME, Qiong Liu, Zhengyou Zhang, Tom Huang, John Weng, Susie Wee, Arnold Smeulders, John Smith, HP

Please feel free to leave a URL in your comments.

HP Labs makes wikipedia

susie.wee — Wed, 16 May 2007 04:17:00 GMT

HP Labs made it into Wikipedia!

We'll add more details over time, but I wanted to share the news.

Thanks to Jaap Vermeulen (Jacoplane) for making the entry!

Tags: wikipedia, HP Labs, HP

Please feel free to include a URL with your comments.

Mediascapes/Mscapes: A new medium and a new community web site

susie.wee — Wed, 09 May 2007 18:55:00 GMT

At the HP gaming summit last month [1,2] I hinted at the fact that Mediascapes had something in the works for May. I'm at the HP Mobility Summit in Shanghai and we just made the announcement. HP is creating a community web site for Mediascapes users and creators and we just opened up the beta release at www.mscapers.com!

Full disclosure: I manage the team and I love their work!

Mediascapes: A new medium

A Mediascape is a context-aware multimedia experience that allows you to trigger multimedia content based on your context such as location [3]. A key concept of Mediascapes is linking media to context, and we view this combination as a new medium. For example, there are images (pictures), audio (sound), video (time-sequenced pictures), and mediascapes (context-triggered multimedia). Mediascapes can be used to create games, educational tools, historical tours, and many other applications.

We'd like to get this new medium into widespread use and do it in as open a way as possible. Since we're trying to create a new medium, we figured that it deserves a community. So, we created the mscape web site as a place where people can go to create, share, and play Mediascapes and discuss their experiences.

Downloadable software

The Mscapers site has mscape software available for download, including:

Mscape toolkit (PC software): A Mediascape creation and management toolkit for use on your PC.
Mscape player (client software): A Mediascape player for use on your GPS-enabled mobile device running Windows Mobile 5.0 or newer.

Mscaper web site

The Mscapers site allows Mediascape creators to create and upload mediascapes and it allows Mediascape users to download and experience Mediascapes.

It also has Mscape wizards that let you create Mediascapes on the web site.

The Mscapers site also has discussion boards and a wiki that allows developers to discuss their experiences with the toolkit. It also has rating tools that allow users to rate and discuss the Mediascapes on the site.

Measures of success

The Mediascapes project is an HP Labs research project that has been underway for a few years. It is now a project in the HP Innovation Program Office [4], where we are looking for opportunities for productization.

Full disclosure: Our measure of success is the number of Mscapers. This includes both the number of people who download and play Mediascapes and the number of people who create and upload Mediascapes. So, we'd like to get as many Mscapers as possible!

Try it out!

Give it a try and let us know what you think! Please tell your friends about it, too!

What do you think of Mediascapes as a new medium?
Do you have suggestions on how we can establish Mediascapes as a new medium?
Do you have suggestions on how we can grow the number of Mscapers?

Tags: mediascape, mscaper, mscape, mediascapes, mscapers, mscapes, context-aware multimedia,a new medium, gaming, HP Labs, VooDoo, HP

Please feel free to include a URL with your comments.

JPSEC: Security for JPEG-2000 images

susie.wee — Sun, 29 Apr 2007 18:15:00 GMT

A technical thread that I've been working on for a number of years is media security for scalable media. As part of this I have been developing a media security standard for JPEG-2000 images called JPSEC. This is an international team effort as JPSEC was developed by technologists from around the world.

This week I've been attending the JPEG/MPEG international standardization meeting in San Jose, California. It's been a long process, and at this meeting we just received the notice of publication for JPSEC!!! This means that the standard is finalized and you will soon be able to get the JPSEC specification from the ISO the same way that you get the JPEG and MPEG coding specifications.

In this post, I provide an introduction to JPSEC with some background and motivation. I'll dive into more details in future posts.

JPSEC security tools

JPSEC is Part 8 of the JPEG 2000 family of standards. JPSEC specifies ways of applying security to JPEG-2000 coded images. The three types of security tools specified included in the normative part of the standard are: confidentiality, authentication, and integrity. While these are very standard security tools that are commonly used in many applications, the thing that is different in JPSEC is how these tools are applied to media. Specifically, JPSEC applies security tools to JPEG-2000 images in a media-aware way.

Media-aware security

The traditional way to apply security to media would be treat the media data like any other data file and secure the entire file in a media-unaware way. If a security tool such as encryption is applied in a media-unaware way, then any structure in the media data would be lost.

However, some structure in the media data can be quite useful. For example, scalable coding methods code images into a bitstream that has a structure that makes it easy to access to a low-resolution version of the image, without requiring one to decode or transcode the entire bitstream. If this image data is encrypted for confidentiality in a media-unaware way, then the ability to extract the low-resolution version would be lost, or it would require you to decrypt and then extract the low-resolution data. However, once the image is decrypted, it is no longer secure.

On the other hand, if media-aware security tools are used, then security can be applied in a way that preserves the useful structure in the media. JPSEC recognizes the fact that media data actually has some useful structure to it, so it specifies how to apply security tools to JPEG-2000 images in a media-aware way.

Scalable coding of media: The structure of JPEG-2000 image data

JPEG-2000 has a particularly useful structure because it was designed to be "scalable". While people often talk about the compression performance of JPEG-2000 vs. JPEG and other image coding methods. I think one of the biggest advantages of JPEG-2000 is its built-in scalability.

Scalable coding methods code media (images, video, or audio) in a manner that makes it easy to extract and decode different versions of it. For example, a scalably coded image can easily be decoded in high or low resolution. Decoding the image in high resolution involves decoding the entire bitstream. Decoding the image in low resolution simply involves extracting and decoding the low-resolution segments of the coded media data.

JPEG-2000 is a scalable image coding method. JPEG-2000 was designed in a way that makes it very easy to extract and decode a resolution, a tile, a color component, or a quality layer of the coded image. This can be done by simply scanning the bitstream, identifying and extracting the desired segments of the bitstream, and decoding those segments. This ability of transcoding to a lower resolution or quality level by simply grabbing portions of the image can be very useful for many applications.

Example application for scalable images

Let's say a server stores a very large, high-resolution image and a client with a smaller display would like to look at and virtually navigate around this image. Because the client's display resolution is much smaller than the original image resolution and because the bandwidth between the client and server may be limited, the options are to serve the client a small portion of the image in full resolution or the entire image in low resolution. In order to do this, the server would have to extract a portion of the image or extract a low resolution version of the image.

This can be achieved in different ways. If regular image coding is used, then the server would have to decode, process (select an area or downsample the image), and encode the image or transcode it accordingly. On the other hand, if scalable image coding is used, then it is very easy to extract portions of the image in different resolutions. Transcoding to lower resolutions or smaller tiles simply involves extracting the appropriate set of coded data. This requires very little computation, so it allows the server to support simultaneous image streaming sessions for many clients.

Adding security to media

A question that arises is what happens if end-to-end security is required for the application? For example, what if the application requires the image to be encrypted at the source and decrypted only by people who are allowed access? If this is required, then when the media data is transported between the sender and receiver, it must remain encrypted at all times, including when it is stored on the server.

When the media data is encrypted, what happens to the nice property of being able to adaptively stream portions of the high-resolution original image to lower-resolution clients? If the media is encrypted in a media-unaware way, then this property is lost, or the only way to do adaptively stream is by decrypting the image, but this breaks the end-to-end security of the system.

On the other hand, if media-aware security is used, then the security tools can be applied to the media data in a media-aware way in order to preserve the structure of the protected media and allow the server to adaptively stream portions of the protected media data.

JPSEC media-aware security tools

JPSEC was designed to provide media-aware security tools for JPEG-2000 images. It recognizes the structure of the JPEG-2000 image data, and it secures the media data within that structure. Specifically, it recognizes where the media data is located and which parts of the data correspond to which image components (tile, resolution, quality layer, color component, or image subband). It then allows security tools to be applied to subsets of the image data, and specifies the signaling data that must be included in the protected bitstream to allow the protected subsets of data to be extracted. In other words, JPSEC simultaneously allows mid-network transcoding and end-to-end security!

I hope this provides an introduction to JPSEC along with some background and motivation. I'll dive into more details in future posts! I'll be evolving this into a publication, so please let me know if you have any comments on the description or the text.

What do you think about JPSEC?

Any questions or comments?

Tags: JPSEC, JPEG 2000, scalable coding, security, image security, media security, media-aware security, media-unaware security, end-to-end security, transcoding, media adaptation, secure transcoding, HP

Please feel free to include a URL in your comments.

VooDoo+HP Labs: The Next-Generation Gaming Experience

susie.wee — Sat, 07 Apr 2007 16:26:00 GMT

The Next-Generation Gaming Experience is immersive, social, mobile, and physical. It raises a number of technology challenges to make it real, but researchers love a challenge! Here is our view of the next-gen gaming experience and a few projects that our researchers are working on to get us there. We showed them at the HP Gaming Summit last week in San Francisco.

There are many press articles and blog posts on the gaming summit. Kate Greene of Technology Review did a great job capturing my thoughts in this article.

Full disclosure: I manage some of these projects and I think they're cool!

The Next-Generation Gaming Experience

Immersive visual experience with pixels anywhere and pixels everywhere.

Pixels anywhere- including on walls, on floors, on ceilings, on your watch, and on table tops. When pixels are on your coffee table, there is a technology challenge of creating new interaction models to share, control, and interact. When many people are looking at an upright display, there is a shared view of what is up-down-left-right. When many people are sitting around a coffee table, this assumption needs to be revisited, since each person sitting around the table has their own view of up-down-left-right. There is a challenge in inventing new interfaces and interaction models to make sure everyone around the table gets a first class experience. And, there is a challenge in designing applications that work well in this experience.

We showed Misto, our research project that has a touch-screen display embedded into a coffee table. It has applications such as solving jigsaw puzzles, web browsing (including Google earth fly-by's), and photo sharing, where you can slide photos across the table and spin them to be upright for each person.

Pixels everywhere means that we have flexible pixels, on flat and curved surfaces, with any shape and size. Why limit yourself to a 4:3 or 16:9 aspect ratios? You can use many displays along with real-time video processing algorithms to make displays of any shape, size, and quality. The challenge comes from making many displays, such as projectors, look like one seamless display. Since each individual display has different display characteristics, you need to do some real-time video processing to adapt the color and geometry of the rendered pixels to produce one seamless display.

We showed Panoply, our research project that uses multiple projectors pointed side-by-side on a curved screen to give you an immersive visual experience that covers your entire field of view. We use real-time photometric and geometric calibration algorithms to create one seamless immersive display.

We also showed Pluribus, our research project that uses multiple projectors pointing at one large screen to give you a better-than-theater visual experience. (At the gaming summit we showed Pluribus with 12 projectors, but you can use any number.) We use real-time video processing algorithms to modify the pixels so that they add up to one super-projector, one that has higher image quality in terms of brightness, contrast, resolution, and color quality.

Social experience. It's a social experience that brings people together and forms communities, with a broader audience than what we have today (maybe including me!). Technologies such as motion sensors are making games easier and more intuitive to play, which will expand the gaming demographics. Also, games will be more integrated with communications such as instant messaging, chatrooms, voice, and video. The challenges lie in integrating these different modalities into one session in the context of a game, and making sure the network priotizes the different types of traffic accordingly. For example, you need very quick response for the game controls, but could delay a half second on a chat message.

Mobile experience. It's a mobile experience that that lets people play games together on any network, on any device, anywhere in the world. By mobile, we don't only mean playing a game on your cell phone. We mean that people are mobile- you will game on your TV, on your computer, on your portable player, or on your cell phone- but you still want to have your gaming experience wherever you are. Of course, your experience will be different depending on your device and your connectivity, so your gaming experience may be different in each situation, but it should still be fun!

Challenges come from rendering the game on any device through any network. It is useful to consider different device-network paradigms. For example, if there is enough network bandwidth you could go to an in-network rendering model, where the players view is rendered on a machine in the network, and the resulting video is streamed to the player through a regular video streaming connection. This way, the device only needs to decode a video stream (e.g., an MPEG) rather than have full graphics rendering capabilities. On the other hand, if your device has plenty of graphics capabilities but little network bandwidth, you could go to mode where you only send commands over the network but render the game on the device itself.

Physical experience that maps the physical and virtul worlds. It senses your physical context and triggers experiences accordingly. Your physical context can be your location, your movements, and even your heartrate or who you are next to. Based on your context, it then triggers a multmedia experience. Technology challenges lie in appropriately sensing your context, which requires having sensors that are accurate and power-efficient. Challenges also lie in developing applications and experiences that use context.

We showed Mscape, a.k.a. Mediascapes, our research project aimed at creating and sharing context-aware multimedia experiences. We've had Mediascapes deployed around the world, most recently at the Tower of London and Yosemite National Park.

What is your view of the next-gen gaming experience?

What technologies and projects are there out there that will make it real? (include URLs)

Tags: gaming, next generation gaming experience, HP Labs, VooDoo, HP Gaming Summit, misto, panoply, pluribus, mscape, mediascape, HP

Feel free to include a URL in your comments.

VooDoo acquires HP!

susie.wee — Tue, 03 Apr 2007 07:56:00 GMT

I've worked at HP for over 10 years, and I love HP and its core values and its culture. HP continues to evolve organically and through mergers and acquisitions, some big (Compaq) and some small (VooDoo). Amazingly, HP's core values strongly live on... and its culture grows with each acquisition.

It's been amazing to have VooDoo join HP and watch our combined company grow. It's no secret that VooDoo co-founder Rahul Sood loves HP Labs. I'm the lucky lab director who cares for much of the cool research that Rahul loves!

We started to put HP Labs and VooDoo together before we (HP Labs) knew about the acquisition. We had some technology that we were developing for the HP Halo business. Then our then-secret HP gaming guys led by Phil McKinney came by and asked us to apply our technology to gaming. My researchers pounced on the opportunity, working day and night to modify the technology for gaming and create a working prototype. They flew the prototype to NYC for a mystery event. At the event, they found out that it was to announce the HP acquisition of VooDoo! Our technology was demonstrated as part of the announcement. My researchers were pumped!

Immediately following the announcement, Phil brought Rahul to HP Labs. HP Labs instantly became Rahul's sand box. I had a lab offsite the following week, and I invited Rahul and the HP gaming folks to attend. Rahul gave one of his killer talks which instantly show what Rahul and VooDoo are all about. It was such a perfect match. Rahul believes in passion, technology, and innovation. HP Labs researchers believe in passion, technology, and innovation. As Rahul said, "it's like we were separated at birth".

The one thing that is different is that Rahul pursued passion, technology, and innovation as a gaming entrepreneur while the HP Labs folks pursued them as researchers. Put the two together, and you have quite a combination!

Rahul's energy is infectious and the HP Labs researchers are energized. Rahul and HP Labs truly want to get HP Labs technologies into VooDoo products and customer hands. VooDoo provides us with a whole new avenue for technology transfer, and this excites us! On top of that, we have a large underground gaming community that now gets to pursue gaming for their day jobs. Yes, our gaming community is ecstatic... and growing! Managing research is about aligning passions, and I can safely say, we're aligned!

HP Labs and VooDoo are working together on a number of projects. HP and HP Labs have embraced so much of VooDoo's culture that we can say "VooDoo acquires HP!"

Tags: VooDoo, HP Labs, gaming, culture, passion, acquisition, Rahul Sood, HP

Feel free to include a URL in your comments.

Multimedia search 2.0: tagging vs. analysis / man vs. machine

susie.wee — Fri, 16 Mar 2007 03:07:00 GMT

Yesterday I was in a large meeting with a number of people from across the company. My blogreader was in the corner of my screen, and this post popped up on my screen (screenshot shown on right).

The article is about over-the-air digital video broadcasting (DVB-H), specifically about Quantum's new portable media player that can be used to watch it. The focus of the article is the device, which is interesting.

What caught my eye at that moment was not the device, but the video that happened to be showing on it. The video shows my work buddy and famous Killer Innovations podcaster, Phil McKinney. As a coincidence, Phil was in the meeting and he was actually talking at the moment the article popped on my screen. So, I ripped him a short email that said: "Look!" with the link. This email caught his eye, and after he finished talking, he clicked on the link. And we privately laughed together across the room, because we both thought this was pretty funny.

Later, Phil told me that my email was actually the second email he received about the link. But he clicked on my email first because of the catchy title. So, I surprised him! (I'm just showing off here- We've worked together for a number of years, but I think this is only the 2nd time in my life that I've actually been able to surprise him.) By now, he must have received many more emails about it.

So, my question in all this is Which is better: user-generated tagging or machine-based content analysis?

I was able to spot and get this content over to Phil pretty quickly, and by now I'm sure many other people have as well. Would a machine running a content analysis algorithm linked into a subscription-based alert service have been able to pick this up and get it over to Phil as quickly? Would it have been able to identify Phil? Would it have labelled Phil as part of the content, or just the main content of the story?

There are many researchers working on multimedia analysis. They are trying to create media processing algorithms that can understand what is going on in the pixels of a scene. I do think this is a very important and very hard research problem. There are a number of applications that require you to analyze large existing archives of multimedia streams that do not have any metadata associated with them, so these algorithms can help to process it.

On the other hand, for many applications, users may be around to tag content. These users could tag it much faster and more accurately than a machine. In the web 2.0 and video 2.0 world, make sure to think about the power of the users! Let them help you with the analysis!

Researchers: Please think hard about the broader problem or application that you are trying to solve, and frame the problem accordingly. You don't want to spend too much time working on the wrong problem!

Which is better: Tagging or analysis? Man or machine?

UPDATE
Alex Vorbau poses a better question: What are the best uses of human tagging and machine analysis and how can we make them better?

Tags: multimedia search, tagging, content analysis, video 2.0, multimedia, research

How Singapore manages research

susie.wee — Fri, 09 Mar 2007 16:51:00 GMT

I just finished a week in Singapore serving as an international panelist for A*STAR's Thematic Strategic Research Programme on mobile media. Every time I go to Singapore, I am impressed with how the country manages its research. Singapore has a very good understanding and appreciation of how research can benefit the country and its people, and it invests accordingly.

The government provides significant research funding to universities and research institutes and it has established organizations to manage this investment. It directs research by having calls for proposals on strategic themes such as mobile media. It also has nation-wide strategic initiatives such as Intelligent Nation 2015 and Interactive Digital Media.

The country takes a long term view on research. For example, the A*STAR Science & Engineering Research Council states its objectives as:

To develop a foundation of high quality research in key disciplines;
To nurture human capital for research; and
To promote information dissemination and technology transfer

Singapore continually evolves its approach as it sees what works and what doesn't. It solicits feedback by asking international reviewers to serve on boards and panels to evaluate their work and hold it to the highest quality standards.

Singapore views research as a way to develop technologies that can help the country stay competitive. It actively tries to find ways to transfer technology to industry, building linkages with the global industry. It also encourages and supports its researchers to participate in international standardization efforts.

Singapore very strongly views research as a way to train and develop its people. It explicitly states manpower training as a primary objective. It has established programs to identify its brightest early on and provide them with special development opportunities.

Singapore understands the importance of working globally. It allocates funds to bring international researchers into the country to help develop the in-Singapore researchers. It also has programs to send its researchers to other countries to give them international exposure. Singapore struggles with finding ways to keep researchers in the country, since many of the researchers value international work experience. But the country also has an understanding that it should develop researchers as best as it can and it maintains a goal of having a fraction of them to stay in the country.

It is very respectable how Singapore views and respects research, keeps the long-term benefits in mind, and invests in its research and researchers accordingly!

Singaporeans: Did I get this right? Do you have anything else to add?

All: How does this compare to how other countries manage research?

It's easier to do a Ph.D. thesis the second time around!

susie.wee — Tue, 27 Feb 2007 07:11:00 GMT

Let's start with three little stories:

#1: Today I gave a guest lecture in a Stanford graduate course on the first research topic I worked on after grad school as a young researcher at HP Labs (compressed domain video processing and transcoding). As you can guess, the students are incredibly bright. The class was very interactive and upbeat, and the students asked great questions and soaked up the material like sponges. At the end of class, one of the students asked me if this was my thesis topic. I said "No, it was the first research area I worked on after I finished my Ph.D. thesis, and my goal after graduating was to do (the equivalent of) a second Ph.D. thesis." Instantly, their jaws dropped and their eyes widened; and then I realized this was a traumatic thought to students in the middle of their first Ph.D. thesis. So I quickly added "Oh, I realize that you probably didn't want to hear this right now, but don't worry, after you finish your thesis you will know that it's much easier to do a Ph.D. thesis the second time around since you learned how to do one." They smiled with great relief and a glimmer of hope that this could be true!

#2: As a coincidence, earlier today at work I met a bright young Japanese visitor (a co-worker's son) who just finished his Ph.D. in laser optics in Japan and was spending 6 months in the US as a visiting researcher before starting his full-time job in Japan. I told him that I worked on optics for my Master's thesis. He was pleasantly surprised and asked me if my Ph.D. thesis was in optics. I said "No, it was in video." His eyes widened as he looked at me incredulously and he asked "You switched fields?!?" I said "I guess so. But I found digital image and video processing very similar to fourier optics, but on pixels. So there was a relationship." I guess in my mind I didn't switch fields; I just evolved fields.

#3: This reminded me of a time when I had a new upper manager at HP Labs who came in from a different company and was learning his way around. I told him about my and my team's newest research area on media overlays and mobile streaming media content delivery networks. As he made his way around the organization, he found that nobody believed I was working in that area! They would say "Susie is doing that?!? But she does transcoding!" He wisely advised me that I should shake my old reputation around HP of only doing transcoding and let them know what I was doing now. It was interesting advice!

There is a relationship between these stories, and it's one of evolving your research area as you move forward in your career, and learning at the same rate that you learned your first field. Don't be afraid to attack new problem areas in new domains. In fact, you must because your original problem and expertise area won't stay the same for the rest of your career! Also, if you keep the same attitude towards learning as you did when you first learned your first field, then you will be able to keep up with the latest advances in technology and in the industry, and you will be able to stay current.

You may not have to make completely disruptive shifts into a completely new areas all the time. In fact, I never felt like I was switching fields. Rather, you can think about evolving your domain of expertise over time and building up your portfolio with each area that you master. Also, you shouldn't switch out of an area too quickly, as it is good to stay in the area long enough to build up enough knowledge and make a contribution that has impact.

Also, if you're in research, you might end up putting some of your technology on the shelf for a while as you wait for the industry to be ready. Then, when this happens, you can dust off your technology and put it into action! Actually, the funny thing is that the topic that I lectured on at Stanford today and worked on 10 years ago is very relevant to the industry today! Who knew it would take 10 years? That's the life of a researcher!

Closing thoughts:

It's much easier to do a thesis the 2nd time around.
Evolve your expertise to stay current!
Stay in a domain long enough to make a contribution with impact.
Learn new fields at the same rate that you learned your first field.
Build up your reputation... then shake it!
Put your research on the shelf for a while as you wait for the right time. When the time is right, dust it off and put it into action!

Note: This post is written in the context of a Ph.D. because I was speaking with graduate students. However, I think these ideas apply to everyone one, regardless on whether you ever have or will formally pursue a Ph.D. In my mind, when I say "pursue a Ph.D." I mean "pursue a level of knowledge, expertise, and contribution in an area with real passion and focus". For example, you can do the equivalent of pursuing a formal Ph.D. in your own life without going to grad school. Just embrace an area with conviction and passion and drive to make a significant contribution and have impact. This can apply to your work whether you are in research or in sales and marketing, and it can apply to your personal life whether you are mastering a sport or French cooking! In my mind, any person with this drive and focus can "get a Ph.D.". Many of the best researchers and technologists I know don't have or need formal Ph.D.'s. They just keep on driving!

Please do share your comments and little stories. I'd love to hear from a few grad students and graduated students, both in the formal and informal do-it-yourself Ph.D. programs!