{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Train our first movie recommender\n", "\n", "In this tutorial, we build our first recommender system using a simple algorithm called [P3alpha](https://nms.kcl.ac.uk/colin.cooper/papers/recommender-rw.pdf).\n", "\n", "We will learn\n", "\n", "- How to represent the implicit feedback dataset as a sparse matrix.\n", "- How to fit `irspack`'s models using the sparse matrix representation.\n", "- How to make a recommendation using our API." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import scipy.sparse as sps\n", "\n", "from irspack.dataset import MovieLens1MDataManager\n", "from irspack import P3alphaRecommender" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Read Movielens 1M dataset\n", "\n", "We first load the [Movielens1M](https://grouplens.org/datasets/movielens/1m/) dataset. For the first time, you will be asked to allow downloading the dataset." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
userIdmovieIdratingtimestamp
01119352000-12-31 22:12:40
1166132000-12-31 22:35:09
2191432000-12-31 22:32:48
31340842000-12-31 22:04:35
41235552001-01-06 23:38:11
\n", "
" ], "text/plain": [ " userId movieId rating timestamp\n", "0 1 1193 5 2000-12-31 22:12:40\n", "1 1 661 3 2000-12-31 22:35:09\n", "2 1 914 3 2000-12-31 22:32:48\n", "3 1 3408 4 2000-12-31 22:04:35\n", "4 1 2355 5 2001-01-06 23:38:11" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "loader = MovieLens1MDataManager()\n", "\n", "df = loader.read_interaction()\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`df` stores the users' watch event history.\n", "\n", "Although the rating information is available in this case, we will not be using this column. What matters to implicit feedback based recommender system is \"which user interacted with which item (movie)\".\n", "\n", "By `loader` we can also read the dataframe for the movie meta data:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titlegenresrelease_year
movieId
1Toy Story (1995)Animation|Children's|Comedy1995
2Jumanji (1995)Adventure|Children's|Fantasy1995
3Grumpier Old Men (1995)Comedy|Romance1995
4Waiting to Exhale (1995)Comedy|Drama1995
5Father of the Bride Part II (1995)Comedy1995
\n", "
" ], "text/plain": [ " title genres \\\n", "movieId \n", "1 Toy Story (1995) Animation|Children's|Comedy \n", "2 Jumanji (1995) Adventure|Children's|Fantasy \n", "3 Grumpier Old Men (1995) Comedy|Romance \n", "4 Waiting to Exhale (1995) Comedy|Drama \n", "5 Father of the Bride Part II (1995) Comedy \n", "\n", " release_year \n", "movieId \n", "1 1995 \n", "2 1995 \n", "3 1995 \n", "4 1995 \n", "5 1995 " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "movies = loader.read_item_info()\n", "movies.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Represent your data as a sparse matrix\n", "\n", "We represent the data as a sparse matrix $X$, whose element $X_{ui}$ is given by\n", "\n", "$$\n", "X_{ui} = \\begin{cases}\n", "1 & \\text{if the user }u\\text{ has watched the item (movie) } i \\\\\n", "0 & \\text{otherwise}\n", "\\end{cases}\n", "$$\n", "\n", "For this purpose, we use `np.unique` function with `return_inverse=True`.\n", "This will return a tuple that consists of\n", "\n", "1. The list of unique user/movie ids appearing in the original user/movie id array\n", "2. How the original user/movie id array elements are mapped to the array 1.\n", "\n", "So if we do" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "unique_user_ids, user_index = np.unique(df.userId, return_inverse=True)\n", "unique_movie_ids, movie_index = np.unique(df.movieId, return_inverse=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "then ``unique_user_ids[user_index]`` and ``unique_movie_ids[movie_index]`` is equal to the original array:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "assert np.all( unique_user_ids[user_index] == df.userId.values )\n", "assert np.all( unique_movie_ids[movie_index] == df.movieId.values )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Thus, we can think of ``user_index`` and ``movie_index`` as representing the row and column positions of non-zero elements, respectively.\n", "\n", "Now $X$ can be constructed as [scipy's sparse csr matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html) as follows." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "<6040x3706 sparse matrix of type ''\n", "\twith 1000209 stored elements in Compressed Sparse Row format>" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X = sps.csr_matrix(\n", " (\n", " np.ones(df.shape[0]), # values of non-zero elements\n", " (\n", " user_index, # rows of non-zero elements\n", " movie_index # cols of non-zero elements\n", " )\n", " )\n", ")\n", "\n", "X" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We encounter this pattern so often, so there is `df_to_sparse` function in irspack:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "from irspack import df_to_sparse\n", "X_, unique_user_ids_, unique_item_ids_ = df_to_sparse(df, 'userId', 'movieId')\n", "\n", "# X_ is identitcal to X.\n", "assert (X_ - X).getnnz() == 0" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fit the recommender.\n", "\n", "We fit `P3alphaRecommender` against X." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "recommender = P3alphaRecommender(X)\n", "recommender.learn()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Check the recommender's output\n", "\n", "Suppose there is a new user who has just watched \"Toy Story\". Let us see what would be the recommended for this user.\n", "\n", "We first represent the user's watch profile as another sparse matrix (which contains a single non-zero element)." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "title Toy Story (1995)\n", "genres Animation|Children's|Comedy\n", "release_year 1995\n", "Name: 1, dtype: object" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "movie_id_vs_movie_index = { mid: i for i, mid in enumerate(unique_movie_ids)}\n", "\n", "toystory_id = 1\n", "toystory_watcher_matrix = sps.csr_matrix(\n", " ([1], ([0], [movie_id_vs_movie_index[toystory_id]])),\n", " shape=(1, len(unique_movie_ids)) # this time shape parameter is required\n", ")\n", "\n", "movies.loc[toystory_id]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since this user is new (previously unseen) to the recommender, we use `get_score_cold_user_remove_seen` method.\n", "\n", "`remove_seen` means that we mask the scores for the items that user had watched already (in this case, Toy Story) so that such items would not be recommended again.\n", "\n", "As you can see, the score corresponding to \"Toy Story\" has $-\\infty$ score." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[ -inf, 8.18606963e-04, 4.30083199e-04, ...,\n", " 4.30589311e-05, 1.09994485e-05, 2.71571993e-04]])" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "score = recommender.get_score_cold_user_remove_seen(\n", " toystory_watcher_matrix\n", ")\n", "\n", "# Id 1 (index 0) is masked (have -infinity score)\n", "score" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To get the recommendation, we ``argsort`` wthe score by descending order\n", "and convert \"movie index\" (which starts from 0) to \"movie id\"." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([2858, 1265, 2396, 3114, 260, 1210, 1196, 1270, 2028, 34])" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "recommended_movie_index = score[0].argsort()[::-1][:10]\n", "recommended_movie_ids = unique_movie_ids[recommended_movie_index]\n", "\n", "# Top-10 recommendations\n", "recommended_movie_ids" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And here are the titles of the recommendations." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titlegenresrelease_year
movieId
2858American Beauty (1999)Comedy|Drama1999
1265Groundhog Day (1993)Comedy|Romance1993
2396Shakespeare in Love (1998)Comedy|Romance1998
3114Toy Story 2 (1999)Animation|Children's|Comedy1999
260Star Wars: Episode IV - A New Hope (1977)Action|Adventure|Fantasy|Sci-Fi1977
1210Star Wars: Episode VI - Return of the Jedi (1983)Action|Adventure|Romance|Sci-Fi|War1983
1196Star Wars: Episode V - The Empire Strikes Back...Action|Adventure|Drama|Sci-Fi|War1980
1270Back to the Future (1985)Comedy|Sci-Fi1985
2028Saving Private Ryan (1998)Action|Drama|War1998
34Babe (1995)Children's|Comedy|Drama1995
\n", "
" ], "text/plain": [ " title \\\n", "movieId \n", "2858 American Beauty (1999) \n", "1265 Groundhog Day (1993) \n", "2396 Shakespeare in Love (1998) \n", "3114 Toy Story 2 (1999) \n", "260 Star Wars: Episode IV - A New Hope (1977) \n", "1210 Star Wars: Episode VI - Return of the Jedi (1983) \n", "1196 Star Wars: Episode V - The Empire Strikes Back... \n", "1270 Back to the Future (1985) \n", "2028 Saving Private Ryan (1998) \n", "34 Babe (1995) \n", "\n", " genres release_year \n", "movieId \n", "2858 Comedy|Drama 1999 \n", "1265 Comedy|Romance 1993 \n", "2396 Comedy|Romance 1998 \n", "3114 Animation|Children's|Comedy 1999 \n", "260 Action|Adventure|Fantasy|Sci-Fi 1977 \n", "1210 Action|Adventure|Romance|Sci-Fi|War 1983 \n", "1196 Action|Adventure|Drama|Sci-Fi|War 1980 \n", "1270 Comedy|Sci-Fi 1985 \n", "2028 Action|Drama|War 1998 \n", "34 Children's|Comedy|Drama 1995 " ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "movies.reindex(recommended_movie_ids)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The above pattern - mapping item IDs to indexes, creating sparse matrices, and reverting indexes of recommended items to item IDs - is a quite common one, and we have also created a convenient class that does the item index/ID mapping:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titlegenresrelease_year
movieId
2858American Beauty (1999)Comedy|Drama1999
1265Groundhog Day (1993)Comedy|Romance1993
2396Shakespeare in Love (1998)Comedy|Romance1998
3114Toy Story 2 (1999)Animation|Children's|Comedy1999
260Star Wars: Episode IV - A New Hope (1977)Action|Adventure|Fantasy|Sci-Fi1977
1210Star Wars: Episode VI - Return of the Jedi (1983)Action|Adventure|Romance|Sci-Fi|War1983
1196Star Wars: Episode V - The Empire Strikes Back...Action|Adventure|Drama|Sci-Fi|War1980
1270Back to the Future (1985)Comedy|Sci-Fi1985
2028Saving Private Ryan (1998)Action|Drama|War1998
34Babe (1995)Children's|Comedy|Drama1995
\n", "
" ], "text/plain": [ " title \\\n", "movieId \n", "2858 American Beauty (1999) \n", "1265 Groundhog Day (1993) \n", "2396 Shakespeare in Love (1998) \n", "3114 Toy Story 2 (1999) \n", "260 Star Wars: Episode IV - A New Hope (1977) \n", "1210 Star Wars: Episode VI - Return of the Jedi (1983) \n", "1196 Star Wars: Episode V - The Empire Strikes Back... \n", "1270 Back to the Future (1985) \n", "2028 Saving Private Ryan (1998) \n", "34 Babe (1995) \n", "\n", " genres release_year \n", "movieId \n", "2858 Comedy|Drama 1999 \n", "1265 Comedy|Romance 1993 \n", "2396 Comedy|Romance 1998 \n", "3114 Animation|Children's|Comedy 1999 \n", "260 Action|Adventure|Fantasy|Sci-Fi 1977 \n", "1210 Action|Adventure|Romance|Sci-Fi|War 1983 \n", "1196 Action|Adventure|Drama|Sci-Fi|War 1980 \n", "1270 Comedy|Sci-Fi 1985 \n", "2028 Action|Drama|War 1998 \n", "34 Children's|Comedy|Drama 1995 " ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from irspack.utils.id_mapping import ItemIDMapper\n", "\n", "id_mapper = ItemIDMapper(\n", " item_ids=unique_movie_ids\n", ")\n", "id_and_scores = id_mapper.recommend_for_new_user(\n", " recommender,\n", " [toystory_id], cutoff = 10\n", ")\n", "movies.reindex(\n", " [ item_id for item_id, score in id_and_scores ]\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "While the above result might make sense, this is not an optimal result.\n", "To get better results, we have to tune the recommender's hyper parameters\n", "against some accuracy metric measured on a validation set.\n", "\n", "In the next tutorial, we will see how to define the hold-out and validation score." ] } ], "metadata": { "interpreter": { "hash": "2c8d963a96d0919c97b2debbb8f097e0c884c04e5d1a6aef0590fcc7fac9e98a" }, "kernelspec": { "display_name": "Python 3.10.0 64-bit ('main')", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.0" } }, "nbformat": 4, "nbformat_minor": 4 }