Emerging Technologies and Markets

HP Techcon

rkrish67 — Thu, 15 May 2008 03:51:00 GMT

I was in Boston from May 4-May 7 at HP Techcon. HP Techcon is an HP internal technical conference where some of HP’s top technologists meet and present their work. Competition is extremely fierce and the probability of getting a paper accepted is lower than most technical conferences. HP has probably the widest portfolio of products in the technology industry and one gets to see and understand lots of interesting technologies during the course of this conference. HP’s memristor breakthrough was presented in this years Techcon and it was fascinating.

Techcon also provides opportunities to understand other HP businesses. I met a technologist from HP’s Procurve business. This is a fast growing business and it was fascinating to learn and understand HP’s strategy in this business.

I also got an opportunity to meet Bob Tarjan in a mentoring session. Bob is probably the youngest Turing award winner for his work on algorithms (in 1986). I had the pleasure of studying some of Bob’s algorithms in a graduate algorithms course (including the famous Sleator-Tarjan self adjusting trees algorithm). The professor who taught us this course humbly admitted that there was no way he could teach us to design algorithms like these. Meeting Bob was an inspirational experience. We chatted about the importance of doing good science and other HP specific topics.

HP also organizes a half day outing during the Techcon. I did the Boston city tour, we had a fantastic tour guide who cracked jokes all the while.

Electricity bill payment and CRM 2.0

rkrish67 — Tue, 25 Mar 2008 08:41:00 GMT

Recently, I had an unpleasant experience with my electricity provider in Bangalore. I had set up automatic bill payment with my bank. Something went wrong, and the bill was not paid on time. I did not check my monthly balance to verify whether the debit took place and the utility company yanked off the electricity without warning. I had to then go and pay the bill manually to get the supply back again. We were off the grid for only a couple of hours but it was an irritating experience.

The missed payment happened once with my earlier bank too (though we didn’t get yanked off the grid then). As a result of being bitten twice (twice bitten, always shy), I have decided to dispense off with the automatic modes of payment through my banks. I can go back to paying the bills manually but the ride to the utility office every month to drop the cheque is cumbersome. After discussion with friends and neighbors, two solutions have emerged as frontrunners.

Solution A) Have someone (maid, watchman) drop off the cheque manually. Has some risks (cheque might get misplaced) but most of my neighbors swear by this solution.

Solution B) Prepay the utility. Since my average bill is of the order of Indian rupees 2000 ($50 per month), I could pay an advance sum of say Rupees 24000 at the beginning of every year and forget about electricity bills for the rest of the year. This is the option I use with my satellite TV provider (who provides a reasonable discount for doing this).

I will be trying solution A for sometime, if it doesn’t work I will switch to solution B.

You might ask: Why not switch to a more friendly electricity provider? In India, electricity distribution is still a monopoly (generation and transmission of electricity is deregulated in parts). So, switching providers is not an option. To be fair to the electricity company, they have installed kiosks that have made the bill payment process an order of magnitude easier. Also, in Indian businesses where monopolies don’t exist (banking, telephony), service is still a problem (though they don’t pull the plug abruptly like the electricity company).

Why is customer service in India such a problem? Most probably because the average revenue (ARPU) per customer is low and there are too many customers. In some services (like mobile telephony), there is huge customer churn because of the stiff competition. Also, because the economy is in growth mode, companies are more focused on acquiring customers than retaining them. Hence, incentive to provide good customer service is low.

I believe India could be a fertile ground for newer approaches to CRM. India needs cost effective CRM solutions (CRM 2.0) where the CRM cost per customer is significantly below the revenues generated per customer. This cannot be achieved by humans-in-the-loop, there are far too many customers for doing that. Hence, CRM has to happen mostly by self-service and there is a need for automated solutions that leverage mobile phones (both for information delivery and payment). Also, instead of each company trying to do its own CRM, economies of scale could be achieved by outsourcing the CRM function to a CRM service provider.

Personalization tutorial slides

rkrish67 — Thu, 24 Jan 2008 09:31:00 GMT

We have put up the slides of the Personalization tutorial we gave at the ACM Compute Conference in Bangalore on Slideshare. The slides are available here

Tutorial on Personalization technologies

rkrish67 — Mon, 14 Jan 2008 03:02:00 GMT

We will be giving a tutorial titled Personalization: Techniques and Applications at the ACM Compute conference in Bangalore on January 20, 2008. Besides myself, two other researchers from HP Labs, Geetha Manjunath and Somnath Banerjee will be conducting this tutorial.

Personalized information retrieval and search promises to improve the Internet experience. This tutorial covers the theory and practice of building personalized information systems. This tutorial is in three parts. In the first part, we will give an overview of some of the personalization systems currently used in industry and techniques for creating and evaluating user profiles. In the second part, we will cover techniques for document modeling (including techniques like Latent Semantic indexing and probabilistic LSI) and recommender systems. The third part of the tutorial will cover Semantic web technologies and use of semantic information in improving personalization.

So, if you are attending the ACM Compute conference and are interested in Personalization technologies, welcome to our tutorial.

The Wikia search engine

rkrish67 — Wed, 09 Jan 2008 02:34:00 GMT

I am a big fan of Wikipedia and had been waiting for the launch of the Wikia search engine. I tried it as soon as it launched and liked the clean interface and the manner in which the search results are presented. Wikia uses the open source Nutch search engine. There is an explanation of rankings alongside the search results (though this would be useless to the layman, it provides transparency to the search results).

An interesting aspect of the search is the request for a “mini-article” on the search query. Presumably, this will be the first result of the search query after the “mini-article” gains in quality. I was surprised that Wikia doesn’t use the Wikipedia page on the topic as the first search result, especially because other search engines point to Wikipedia pages as the first search result about 30% of the time.

Community based personalization of search results is a promising direction for improving search results. Wikia combines a search engine with a social network and it would be interesting to see how the effects of being part of different communities beings to impact the search results. Wikia should make the community based ranking algorithm public at the earliest so that it could benefit from reviews by experts.

I think it is premature to evaluate the results and think it will take some time before the effects of the community rating on the search results kicks in. In any case, it is pretty hard to compare search engines. I have been experimenting with different search engines lately. I have not observed any significant differences in the quality of search results (or in the speed of retrieval) when I switched search engines. Most of the query specific ads are not very relevant in India, so the quality of ads also do not influence search engine preference.

Finally, what about the personalization feature claimed by search engines? This has been a big disappointment. I have given enough data to search engines to personalize the results for me. At least in my books, the quality of the personalized results so far do not compensate the privacy risk I have taken in providing data to search engines.

The Web and India

rkrish67 — Thu, 13 Dec 2007 10:41:00 GMT

The Web and India

The Web has ushered in the most dramatic improvements in productivity in the history of mankind. Why is this? A simple explanation is: The Web makes everything go faster.

Communication has become much faster. Today, email delivery is almost instantaneous and there are few reasons to use fax (which by the way created a lot of productivity gains).

Information delivery has become much faster. Online news sites are a faster way to remain on top of the latest news.

Transactions have become faster. Even in India, where Internet has been slow to take off, Information technology has changed life for the better, atleast in the bigger cities and towns. Ten years back, I withdrew cash standing in a line in a bank. Today very few people (atleast in the bigger cities) withdraw cash this way, ATM's are ubiquitous. I pay all my bills online. Stock trading is dematerialized and mostly conducted online. A lot of people book tickets online.

The World Wide Web is extraordinarily important to India, perhaps more than any other country in the world. Investments in infrastructure in India have not kept pace with GDP growth. Hence the Web will probably make more difference to the life of the Indian consumer as compared to consumers in developed countries.

However, consuming information on the web is still difficult for the average Indian consumer. IT literacy is not widespread (apart from the metros and leading two-tier cities). User Interfaces like search and RSS are just too complex for the average consumer. Most people without an IT background still seem to prefer Web portals for getting their information. Maybe this will change in the next few years (especially with more people from the younger generations accessing the web). But for now, the interface will have to provide a “Do-it-for-me” method of consuming the web (much like television)

What will be the best interface for the Indian consumer (in general a consumer who doesn’t have much IT training)? Is it a single device like the mobile phone? Or is it multiple special purpose devices that mask the complexity of connecting to the web and fetching information from the web? Time will tell.

Life, identity and consciousness

rkrish67 — Tue, 11 Sep 2007 04:22:00 GMT

Came across some interesting blog posts and articles on the nature of life and consciousness in the last month or so.

John Wilkins provides a great summary of various attempts at providing a definition of life. The thermodynamics view views life as using free energy and reducing entropy. Chemistry has shown that organic compounds can be synthesized through inorganic processes. The information theoretic view is that life is a process of information production.

An older article by Susan Blackmore questions the reality of identity. Using a fascinating experiment, she goes on to state that the idea of self is an illusion. Human beings pass around ideas and concepts (which she calls memes) from generation to generation and the idea of self is a trick being played by memes. She makes the interesting observation that meditation is a meme clearing meme, it prevents other memes from creating and maintaining a false sense of identity.

Finally, this post by Nova Spivack asks the question whether it is possible for artificially constructed machines to possess consciousness. Nova postulates that it will never be possible for a machine to be conscious. Consciousness and awareness are fundamental properties of the Universe and only awareness can be aware of itself, nothing that is not aware to begin with can become aware.

Science has begun to ask fundamental questions that philosophers have been asking for centuries. There is a long history of such reasoning in India beginning with the ancient Vedas and Upanishads. Even in the twentieth century, there have been philosophers like Nisargadatta Maharaj expounding on these topics.

Craig Mundie on Personalization

rkrish67 — Mon, 06 Aug 2007 05:51:00 GMT

Came across this speech by Craig Mundie of Microsoft on personalization and next generation personal computing. He asks the question “How will desktop applications of the future look” and makes the point that applications on desktop computers today wait for a mouse click or keyboard input and then try to do something. He envisages a world in which personal computers would be more like personal assistants: anticipating user needs and proactively fetching information that the user may need. An excerpt from his talk

“Today certainly I, and perhaps many of you, enjoy having people who work to help you get things done - an administrative assistant or some type of staff. The great thing about people in those roles is that they are not automatons. They think. They learn about you. They understand what you value. They understand what's important. They make decisions. And, perhaps interesting, they speculate about what might be interesting. And so one of the things I dream about personally is being able to move to where the computer is also able to speculate, to do things on the anticipation that it might turn out to be useful for you

….

We'll see local execution, importantly things that would not be plausible to do at scale in the cloud. Why? Both ends of the wire are built out of the same chip. The only reason you could do some of those things in the cloud is because there's low utilization or slow utilization by the people at the edge. But if the machine actually moves to become fully productive, to anticipate things and to attempt things on your behalf, then, they would be qualitatively different and more valuable”

In other words, the cloud cannot personalize well enough because

The cloud can do only so much computation for one person. As multi-core desktop computers proliferate, the desktop can spend even more cycles on you than computers sitting in remote data centers
Parts of the cloud know only parts of you. Amazon knows only about your book purchases, Ebay knows only about the auctions you participate in etc.

This is a very compelling vision and Craig captures its essence nicely. This vision of the world is similar to the one put forth by Esther Dyson recently where she spoke about going from a “search and fetch” mentality to “deliver, act and transact”.

Apart from improving productivity, such speculative fetch and execution should improve privacy as well. A lot of personal data is being sent into the cloud today because of the low utilization on the edge. As computation and decision making shifts to the edge, there will be less need to send personal data to the Internet cloud.

Attributes of a good ad experience

rkrish67 — Thu, 21 Jun 2007 08:12:00 GMT

All mass media from print to the mobile phone have utility because of the content they deliver or the service they enable (such as communication in a mobile phone). Almost all mass media depend on advertising for sustenance. However, advertising when delivered very frequently and without relevance becomes a nuisance.

I still remember some of the ads I saw as a kid in comic books and magazines such as the Readers Digest. The magazines had fewer (and quieter) ads. These days, I get calls from my mobile service provider where a pre-recorded voice advertisement starts playing when I answer the call. This is intrusive, doesn’t provide any value and is extremely irritating. If this was not bad enough, the service provider also started SMS spamming to advertise a call prohibit service (paid for of course).

Today, as different mass media from print to blogs jostle for one’s attention, it appears that the ability to switch off ones attention from unwanted advertising will become as important as consuming information. Advertising suffers from the tragedy of the commons. However, the following attributes seem important from a perspective of making ad consumption a better experience.

1. Predictable placement

With my newspaper, I know I will not see any ad on the editorial page and maybe one ad on the front page. With search engine ads too, the location and number of ads are mostly predictable. Most Web pages and blog pages are un-predictable in both the ad location and the number of ads shown. While a predictable ad location on web pages might cause people to tune out ads, an unpredictable location might make them tune out the website altogether.

2. Timing of the ad

With a TV serial, we know when the ad slot will start and how long it will last. This allows one to anticipate the ad and adjust ones disposition. Throwing up a popup ad when a website is loading slowly is bad timing. Calling a mobile subscriber who is attending an important event to play a pre-canned message is equally bad timing. Getting an ad when one is not expecting it creates a negative image of the brand being advertised.

3. Relevance of the ad

Of all the ads I see, about 95% are irrelevant. This is a pity because there are many ads that are informative and useful. But these get crowded in the din of irrelevant advertising. While search engine advertising is more relevant than other forms, it suffers from the curse of a limited vocabulary (the average search query is between two to three words). A richer vocabulary seems to be required for more targeted (and useful) ads. Search advertising is also pull based and one has to enter a query before seeing ads. Hence the discovery element that is present in advertisements in other media forms is missing.

4. Ability to switch off attention

Consumers should have control on how many ads they would see and should also be allowed to switch off d attention when they are not inclined to watch ads. The print medium has this attribute; if one has seen an ad before in print, one can avoid reading through it again. Before Personal video recorders, one could not do this with television. Even today, one cannot do it with live television. Sometimes, in a one day cricket match in India, the same ad is telecast 100 times during a seven hour match. Even the best of ads can get tiring after a time.

All this leads to the question: What is the ideal ad experience? For me, it would be an advertisement system that was not intrusive, somehow understood my needs and also allowed control and ability to switch off.

Are there any other attributes you can think of that characterize a good ad experience?

First day at work

rkrish67 — Sun, 27 May 2007 03:28:00 GMT

Came across this amusing video (via Sadagopan) on how a Bangalore software company welcomes new joinees.

Brought back memories of my first day at work. I took a bus to my workplace where a VAX veteran introduced me to the VAX Job control language. How times have changed !

The economic value of a Personal Computer

rkrish67 — Tue, 08 May 2007 11:14:00 GMT

Mass media such as books, cinema, radio and television are usually consumed in an entertainment context. One doesn’t usually think of the economic value derived from reading a book or watching a television show and in most cases it might be hard to quantify it.

Both the Internet and mobile media are different from other mass media in that they can deliver economic value that is quantifiable. This in turn means that the value of devices used to access these media (the PC and the mobile phone) and the services that enable the access (a broadband subscription or a mobile subscription) can also be quantified.

The high penetration of mobile phones in India is partly because of the economic value it delivers. A high percentage of India’s GDP (about 54 %) is derived from services and mobile phones have enabled a better market for services. Hence plumbers, electricians and people from other service oriented professions all carry mobile phones. A (pre-paid) mobile subscription for a year is about $50 and the cost of the entry level handset is about $50. The economic value of having the phone far exceeds the cost of the phone and the subscription (in most cases).

Personal computers (like mobile phones) are capable of delivering both economic and entertainment value. The success of gaming PC’s and media center PC’s attests to the fact that is possible to create niche PC markets largely focused on yielding entertainment value.

What about the economic value of having a PC? In India, my guess is that the PC currently yields an economic value of about $100-$200 per year (I suspect it would be far higher in the US and other developed economies). In India, cheap broadband rates coupled with decreasing PC prices and services going increasingly online (banking, ticket booking, stock trading, auction sites, bill payment) are poised to vastly increase the economic value of having a personal computer. In many ways, the PC allows one to bypass inefficient infrastructure and systems. My belief is that the economic value will increase further as computers are increasingly used in a one-to-one service delivery model. For instance, one of my neighbors in Bangalore, who is a music teacher (vocalist), gives music lessons using her PC (and VOIP software) to her student in the US.

Dear reader: Do you think of your PC as an entertainment device or as a device that yields economic value or both? What kind of applications do you think will allow you to derive more economic value from your PC ? What are the economic value equivalents of gaming PC’s and media center PC’s ?

Learning mathematics

rkrish67 — Mon, 23 Apr 2007 06:09:00 GMT

A few months back, I had the privilege (along with Shekhar Borgaonkar of HP Labs India) of getting a five minute tutorial on information theory from Abraham Lempel (of Lempel-Ziv fame) at Los Angeles airport. During the course of the conversation, Abraham remarked "India was always strong in mathematics". While most educated Indians are familiar with Srinivasa Ramanujan and his contributions, India has produced a number of other good mathematicians who have not gained the same popularity. One possible reason for this is that the real value of mathematics comes from applying the results of mathematics to other disciplines and there are only a few Indians I know who are extremely good at this. One of them is Dr. M. Vidyasagar who works at TCS research in Hyderabad. Besides knowing lots of mathematics, Dr. Sagar is also very good at applying them to domains like control theory, neural networks and learning theory. He also writes in a very engaging style, a good example is this survey of problems in computational biology.

My own Math education has been limited to the Computer science-oriented Math courses I took in University (Linear Algebra, Discrete Maths, Graph theory and Numerical methods). I really liked graph theory because there are so many interesting applications of graph theory. It was only much after University that I bought Hamming's book on Numerical methods and gained some appreciation for the subject. A few years back, I tried improving my mathematics knowledge by reading some texts on real analysis, differential geometry, topology etc but soon decided that it was not the right approach.

While it is not difficult to comprehend the theorems and results, it is difficult to appreciate the results without knowing how and where they can be applied. Dr. Vidyasagar suggested to me that it is better to read textbooks written by Russian professors as they usually teach undergraduate courses (and some like AN Kolmogorov even taught in schools). I briefly taught a couple of undergrad level math courses, so I can appreciate that it might be hard to keep an undergrad or schooler engaged without explaining where the theorems are applied. If anyone reading this blog knows of Russian books on advanced math topics, please recommend some. Until a few years back, it was common to find Russian text books by Mir publishers (of Moscow) on Indian footpaths, somehow they have become harder to find in recent years.

Around a year back, I read this post by Steve Yegge and have been trying to further my knowledge of mathematics from Wikipedia. Some Wikipedia mathematics pages, such as the pages on Eigen Vectors, Singular value decomposition and Riemann hypothesis are well written and give a very comprehensive overview. The links on these pages are also extremely informative. However, a lot of Wikipedia math pages appear to be edited only by mathematicians and do not appear (atleast to me) to be of the same quality as the eigen vector page. Nevertheless, the Wikipedia approach has worked reasonably well and I am a lot more awareness of math theorems and results than I was a year ago.

Does your field of work require you to learn and use mathematics? If it does, what are your approaches to learning mathematics?

Product localization: Content, packaging and design

rkrish67 — Mon, 16 Apr 2007 06:30:00 GMT

I recently bought some picture books for my daughter that were re-published by a local publisher under license from foreign publisher. The “foods we eat” page had pictures of foods that are rarely (if ever) eaten in India. There were a few other pages where there was a big cultural mismatch and I had to buy another picture book. If the publisher had localized a few pages in the book for the Indian audience, the book would have been much more useful.

Another example: The Thali is an Indian meal with bowls containing different dishes. The Thali is pretty ubiquitous in Indian restaurants but it is localized for different geographies and locations within India. There is the North Indian Thali (more emphasis on bread and north India vegetables), the South Indian Thali (mostly rice items). There are region specific Thalis such as the Maharashtrian Thali and the Gujarati Thali. When MNC food chains started making their appearance in India about 15 years back, I noticed people coming in, asking for a Thali and leaving when told that the restaurant did not have a Thali. A repackaging of the food offerings as a Thali might have helped.

A final example: When we bought a double-door refrigerator a few years ago , the ratio of the freezer compartment to the main compartment was the most important factor in the selection. We wanted a smaller freezer as we are vegetarians and had less use for a large freezer. Only one international company had a model with a ratio close to what we wanted, so the choice was automatic.

A few years back, the gap between locally produced products and imported products in the Indian market was quite large. This gap is narrowing quite rapidly and in some sectors, the Indian products are as good as or even better than those produced in other countries. In this competitive scenario, companies that go the extra mile to introduce features that are useful in the local context or repackage their content to fit local expectations will have an edge.

What are your experiences? Have you used products where the manufacturer localized the content, design or packaging? Have you used products where you wished the manufacturer had paid more attention to localization?

Document delivery over television broadcast networks

rkrish67 — Thu, 05 Apr 2007 10:27:00 GMT

There are multiple mass communication media in existence today. Print is the oldest and has been in existence since the fifteenth century. Other mass communication media include recordings, cinema, radio (all introduced between 1900-1920), television (introduced around 1950) and the Internet and mobile communication (introduced around 1990).

The different media forms have relative strengths and weaknesses. While print is portable and lends itself to annotation, it is a low bandwidth medium. The Internet provides higher bandwidth and interactivity but requires a display and network connectivity. Mobile phones have inbuilt payment mechanisms, are portable and offer connectivity by default but have limited displays. Newer mass media forms have often combined the best qualities of the earlier ones. Television combined the picture element of cinema and the advertising model from newspapers and magazines. Tomi Ahonen’s October 2005 post has a nice discussion on the evolution of different mass media.

Before the advent of the Internet, each media form had a distinct delivery mechanism. Print, cinema and recordings required physical channels for delivery on a mass scale. Television required a broadcast channel and the mobile phone required a radio spectrum. The Internet has allowed the delivery of multiple media forms over the same pipe. In addition to documents, it has become feasible to deliver video and radio over the Internet. In some countries, cinema is also being delivered over the Internet using BitTorrent like P2P technologies.

However, enabling information consumption in countries with low Internet penetration is a challenge. This is the case with India where television is the mass medium with the highest penetration (600 million Indians have access to TV as opposed to 37 million having access to the Internet). Internet penetration is growing at a decent clip, however it will take some time before everyone who has access to a TV has access to a PC.

At HP Labs India, we have developed a system that delivers printable documents over television broadcast (on both satellite as well as IPTV channels). Television has a broad geographic reach and delivers the “last mile” link virtually everywhere. Also, the incremental cost of delivering data through television to an additional reception point is zero. Trials of the technology in an educational setting have shown that print complements television nicely by improving the retention of information broadcast on television.

The wisdom of the crowds versus herd mentality

rkrish67 — Wed, 28 Mar 2007 05:14:00 GMT

The wisdom of the crowds is an important aspect of Web 2.0. User generated meta-data such as deli.icio.us bookmark tags, Digg tags and Amazon reviews use the aggregate behavior of multiple people for content recommendation. Question answering sites like Yahoo Answers allow experts in the crowd to provide answers to questions. Wikipedia founder Jimmy Wales is planning a search engine that intends to replace algorithmic ranking of web pages with human rankings.

In all the above applications, the belief is that a collective can solve certain problems better than individuals (including experts). Combining classifiers is not new, techniques for combining weak classifiers that differ in mis-classifications to get a stronger classifier (using techniques such as boosting) is well known in the machine learning community. However, most of the evidence for crowds being better has so far been anecdotal.

It appears this is about to change and we could have a theory that allows us to understand in what situations the crowd is wise. This article by Michael Mauboussin (based on the book “The difference” by Scott Page) identifies three problem situations where the wisdom of the crowds might be useful. One is where many people in the crowd (but not all) know the answer to a problem. The second is when one person in the crowd knows the answer. The third is when no one knows the answer (a prediction problem) but the answers from multiple people is averaged to get a good prediction. This paper by Dennis Wilkinson and Bernardo Huberman of HP Labs postulates that the number of distinct editors for a Wikipedia article has a strong correlation to the quality of the article.

However, the crowd is by no means always wise. Stephen Dubner has this post on “herd mentality” where he describes how walking to the next bus stop (which was visible from the current one) allowed him to sit with his daughter instead of standing on the ride. One of the comments to his post mentions that people fear short or non-existent lines in a grocery store checkouts and prefer lines with a few people.

India’s disastrous performance in the cricket world cup has led to many fans criticizing the team selection committee. What we perhaps need is a way to select the Indian cricket team using the wisdom of the crowds.