I. Retrieve movies synopses using Python

The MovieLens 100k dataset, fully described in the previous tutorial, contains an IMDb URL field allowing a direct access to the IMDb page of a movie to retrieve its synopsis. However, this URL is not always reliable: it can redirect us to a search page with several movies results or can even be unknown. In addition, some movies don’t have a synopsis but a short description and/or a story line. Also, it is important to note that the synopsis is not included in the movie page, but is available through a link to another page.

In the following, we present a python code addressing these limitations. We use the lxml library to get the html content of a page, and the IMDb library to get the IMDb information based on the movie IMDb id. Using the following code, we successfully retrieved the summary of 1021 movies out of 1661. To load the movie DSV file (u.item) refer to the third code cell of the previous tutorial.

When the URL doesn’t redirect to the right page, we can use the IMDb library to get information about a movie based on its title using the search_movie(title) function. However, this function is a simple search action which returns a list of movies sorted by relevance. In the following code, we ensure that the first element of the search list is the expected movie title. We then get its description and synopsis using its IMDb id.

Using this code, we successfully retrieve the summary of 345 additional movies. As a result, we retrieve summaries for 1366 movies in total, out of 1661. In the next section, we show how these synopses can be used to establish a similarity relationship between movies.

II. Latent semantic analysis of movies synopsis

Principle

To identify the similarity between movies based on their synopsis, we use the « latent semantic analysis » (LSA) which is a widely used technique in natural language processing. LSA transforms each synopsis into a vector in a space of features. In our case, the features are words occuring in the synopses. Then, a matrix containing all vectors is created: columns represent the synopses of movies and rows represent unique words. Accordingly, the number of rows can amount to tens of thousands of words. In order to identify the relevant features of this matrix, the « singular value decomposition » (SVD), which is a popular dimension reduction technique, is used to reduce the number of rows -words- while maintaining and highlighting the similarity between columns -synopses-. The dimension of this approximation matrix is set by a hyper-parameter which is the number of topics. In this framework, a topic consists of a set of words with associated weights defining the contribution of each word to the direction of this topic. Based on this low-dimensional approximation matrix, the similarity between two columns -synopses- is calculated using the cosine of the angle between these two vectors.

Implementation

Hereafter we detail all steps to create a LSA model and compute the similarity between synopses. Here are the mains steps:

1- Pre-processing the synopses: First of all, we remove punctuations and irrelevant words like stop words and persons’ names. This will avoid topics in which these words represent the keywords (for instance, « however », « michael », « none » and « laura » will not define a topic).

In addition, we use the Snowball stemmer, also called Porter2 stemmer, to detect similar words present in different formats (remove suffix, prefix, etc.). Snowball is a language, developed by M.F. Porter, to efficiently define stemmers (see Snowball: A language for stemming algorithms for more information). This stemming algorithm is the most used one in the natural language processing domain.

The Natural Language Toolkit (nltk) provides a large number of methods covering different topics in the human language data domain, like classification, stemming, tagging, parsing and semantic reasoning.

2- Creating the dictionary: The dictionary consists of a concatenation of unique words of all synopses. Gensim is an efficient library to analyse the latent semantic similarity between documents.

This function maps unique words with their ids. In our case, the length of the dictionary is equal to 15035 words which means that each synopsis will be represented through a 15035-dimensional vector space.

3- Creating the corpus: The corpus is the collection of all synopses pre-processed and transformed using the dictionary.

Accordingly, the corpus object contains the movies synopses, represented as sparse vectors. Here is the example of the fourth movie synopsis.

The vector feature (48, 1.0) means that the word with id=48 (‘name’) appears once in the entire corpus[3] (fourth movie synopsis).

4- Creating the tf-idf transformation: The term frequency-inverse document frequency (tf-idf) transformation reflects the importance of words in the corpus. The (tf-idf) value of a word depends not only on the word count in a document, but also on the number of occurrences in the corpus.

To explain the principle of this transformation, let us consider emphasizing movies relating to « the artificial intelligence ». A straightforward method would consist in counting the occurrences of each of these three terms (« the », « artificial », « intelligence ») in movies synopses and computing the similarity based on their « term frequency ». The problem of such a method is that the term « the » is a very common word. The associated term frequency will tend to select movies in which the term « the » is used more frequently. The idea of « inverse document frequency » is to give a relevance weight to different terms. Accordingly, the idf associates a low weight to terms that appear frequently in the corpus and increases the weight of terms that rarely appear.

The « tfidf » function transforms the corpus from a bag-of-words integer counts representation into tf-idf real values weight.

5- Choice of the number of topics: As mentioned above, LSA seeks to identify a set of topics related to the movies synopses. The number of these topics N is equal to the dimension of the approximation matrix resulting from the SVD dimension reduction technique. This number is a hyper-parameter to be carefully adjusted. It results from the selection of the N largest singular values of the tf-idf corpus matrix. These singular values can be calculated as follows:

A histogram of the singular values of the tf-idf corpus matrix is presented in the following figure.

6- Creating the LSA model:

The « LsiModel » function transforms our tf-idf corpus into a latent 30-dimensional space (number of topics = 30).

7- Computation of the similarity between synopses:

III. Adding of the « Similarity » relationships to the Neo4j movies graph

Now, we will introduce a new relationship between two movies based on the semantic similarity computed between their synopses. These similarity relationships can be easily added to the Neo4j graph as shown in the previous tutorial.

IV. Performing recommendations

Hereafter, we show how the semantic similarity between movies can be used to perform recommendations. Quite simply: for a given user, we identify the movies that were appreciated (for example, rated more than 4) and then we recommend movies similar to the appreciated ones (for example, similarity score more than 0.7). You are probably wondering how we manage a movie which is similar to several appreciated ones. Well, we simply compute the sum of the relative similarity scores to emphasize movies similar to several appreciated ones (cf. following figure).

Accordingly, for a given user, the recommendation is performed in three steps:

• Identification of movies rated above a predefined appreciated threshold,
• Identification of unwatched movies similar to the appreciated ones with similarity score above a predefined similarity threshold,
• For movies similar to several appreciated ones, computation of the sum of the similarity scores.

As in the previous tutorial, for each of these methods we asked our guinea pig to attribute ratings to the top 20 suggested movies in order to evaluate the performance of recommendations.

The above recommendation description can be formulated by the following py2neo/Cypher query:

1- Recommendation based on the sum of similarity scores

The following table presents the top 20 suggestions with the ratings attributed by our guinea pig.

Here, our guinea pig provides ratings to 12 movies out of 20 which means that he has watched these movies. Two movies that don’t match his tastes are proposed. However, « Dracula: Dead and Loving It » is not a popular movie (average rating equal to 2.8). Accordingly, this recommendation strategy can be slightly modified to take into account the global rating (average of all users’ rating) associated to the recommended movies. In other words, we recommend movies similar to those appreciated by the user, and we sort this list of recommendations by the sum of similarity scores of a movie weighted by its global rating.

In what follows, the py2neo/Cypher query corresponding to these modifications.

2- Recommendation considering the global rating and based on the sum of similarity scores

The following table presents the top 20 suggestions with the ratings attributed by our guinea pig.

The introduction of the movies rating has greatly improved the recommendation performance. Despite « Highlander », all the already watched movies match the tastes of our guinea pig. Accordingly, we could hope that he will also appreciate the unwatched ones. The reason for which « Highlander » remains in the list of recommendations is that this movie is similar to highly appreciated movies and its average rating (3.5) is relatively good.