How a recommendation engine works

A common thing to notice on websites is a simple list that mentions things that you may like. If it is a shopping website then based on the product you were looking at, the website will indicate some other similar products which you might wish to buy. Let’s say, you buy a game console on an ecommerce website, the very next thing you’ll see is stuff that other people who’ve purchased the same console have bought as well. It may be an extra controller or the most popular game for that console at the moment.

Shifting away from commerce, you could be listening to some music or perhaps be renting a movie, you’re bound to see another “stuff you’ll like” listing. Some may consider it intrusive, some may consider it a boon but the fact is that these things work. Sometimes they’re so eerily spot on that you’d wonder if the “machines” have been observing you all your life. There has been one instance where Target, a popular store in the US started recommending a teenage girl things that pregnant women tend to buy. It was only when the father of the girl noticed this that he complained. Little did he know that his daughter was pregnant and her shopping patterns actually gave it away. On a subconscious level, she was buying or simply looking at things that pregnant ladies would, this made Target’s website recommend things of similar nature.

We’re, of course, talking about recommendation engines. Recommendation engines are complex programs that analyse patterns to deduce a signature for each customer. This in turn is used to sort out all products and come out with a few products that the customer will most likely purchase. There are different approaches to getting a recommendation engine working and they have also been included in CMS packages as part of the ecommerce platform. The origin of the concept on the web is a bit hazy but the there are two primary forms of approaching the concept.

Approaches
Content-based filtering

This approach relies on take each product and using as many parameters as possible to describe it. Take a mobile phone for example, it can be a candy bar or a flip phone. This is

one parameter, the pixel density of the screen becomes another parameter, then you have the battery capacity, screen-size, Wi-Fi protocols supported, basebands supported, weight, colour, etc. the list goes on and on. The more you break down a product the more parameters you have to fiddle with. These parameters are then given weights, or you could say are appropriated a level or importance on a relative scale. The product category is often given the highest weight, so if you were looking at mobile phones then the recommender system is likely to recommend mobile phones and not sanitary products. These systems may provide the user to rate products, this way all the parameters of a rated product are assigned a greater weightage that a non-rated product’s parameters.

Recommendation engines choose for you

This in turn improves accuracy. All these parameters are then used to create a user profile and each time you check out another product this profile gets modified. Suffice to say, the system learns more about you the more you use it.

Popular platforms that use content-based filtering: IMDB, Rotten Tomatoes and Pandora.

Collaborative filtering

This method as the name suggests, requires a certain amount of participation. This method relies upon directly learning about the customer which may require active or passive participation. Active forms include asking the customer to give feedback for each item viewed, either in the form of a slider or a points system. The five star rating that is seen in most places does just that. Passive participation involves observing what all items the customer has viewed and which of them have been purchased. Then sniffing the customer’s cookies in the web-browser also helps draw patterns about the user. All of these methods are used in the generation of a user’s profile.
Popular platforms that use collaborative filtering: Pandora, MySpace and Facebook.

Problems faced by these systems

You must have guessed by now that these systems depend on having plenty of information to work with. However, when a user is new there doesn’t exist any profile and even if the user does surf through a few items it is simply not enough to generate a profile. This scenario is called a Cold Start. And it affects both content-based and collaborative filtering. Another problem is that of Scalability, when there are simply a huge number of products to choose from, filtering and processing a recommendation list will consume not only time but also computational power. Excess time taken would lead to poor page load time and an increased computational power affects the working of the entire website and other sites which may be hosted on and sharing the same resources. These problems have led to a change in the kind of approaches towards making a recommender system and thus, arose the hybrid system.

Hybrid filtering

Needless to say, this method works towards overcoming the drawbacks of the aforementioned methods. A hybrid system can be more accurate than either of the methods. They can overcome the problem of Cold Start or that of Scalability by using a different algorithm for each scenario.

Algorithms

The whole concept comes under machine learning, specifically Collective Intelligence. They make use of multiple algorithms at different stages to generate recommendations. Data Mining algorithms are use to generate the user profile while clustering algorithms are used to shortlist recommendations. Some of the commonly used algorithms are:

Similarity (Jaccard index & Tanimoto coefficient)
k Nearest Neighbour algorithm
n-Dimensional Euclidean distance
k-means algorithm
Bayes Theorem
Singular Value Decomposition

We’ll look into one of these algorithms which is very basic and easy to understand. Clustering of data is one of the first steps towards generating categories which can be then used to generate recommendations. A recommendation engine is based on many such algorithms.