{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation of recommender systems\n",
    "\n",
    "In this tutorial, we explain how to evaluate recommender systems with implicit feedback\n",
    "by holding out method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from irspack.dataset import MovieLens1MDataManager\n",
    "from irspack import (\n",
    "    P3alphaRecommender, TopPopRecommender,\n",
    "    rowwise_train_test_split, Evaluator,\n",
    "    df_to_sparse\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read the ML1M dataset again.\n",
    "\n",
    "As in the previous tutorial, we load the rating dataset and construct a sparse matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = MovieLens1MDataManager()\n",
    "\n",
    "df = loader.read_interaction()\n",
    "\n",
    "X, unique_user_ids, unique_movie_ids = df_to_sparse(\n",
    "    df, 'userId', 'movieId'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split scheme 1. Hold-out for all users.\n",
    "\n",
    "To evaluate the performance of a recommender system trained with implicit feedback, the standard method is to hide some subset of the known user-item interactions as a validation set and see how the recommender ranks these hidden groundtruths:\n",
    "\n",
    "![Perform hold out for all users.](./split1.png \"split1\")\n",
    "\n",
    "We have prepared a fast implementaion of such a split (with random selection of these subset) in ``rowwise_train_test_split`` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_valid = rowwise_train_test_split(X, test_ratio=0.2, random_state=0)\n",
    "\n",
    "assert X_train.shape == X_valid.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "They sum back to the original matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<6040x3706 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 0 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X - (X_train + X_valid) # 0 stored elements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is no overlap of non-zero elements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<6040x3706 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 0 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.multiply(X_valid) # Element-wise multiplication yields 0 stored elements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This scheme however has a problem regarding the performance because we have to compute the recommendation score for all the users. So in the next tutorial, we will be using a sub-sampled version of this splitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Obtain the evaluation metric\n",
    "\n",
    "Now we define the `Evaluator` object, which will measure the performance of various recommender systems based on ``X_valid`` (the meaning of ``offset=0`` will be clarified\n",
    "in the next tutorial)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluator = Evaluator(X_valid, offset=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We fit ``P3alphaRecommender`` using ``X_train``, and compute its accuracy metrics\n",
    "against ``X_valid`` using `evaluator`.\n",
    "\n",
    "Internally, `evaluator` calls the recommender's ``get_score_remove_seen`` method, sorts the score to obtain the rank, and reconciles it with the stored validation interactions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('hit@5', 0.7988410596026491),\n",
       "             ('recall@5', 0.09333713653053713),\n",
       "             ('ndcg@5', 0.396153990817333),\n",
       "             ('map@5', 0.06632471989643292),\n",
       "             ('precision@5', 0.37317880794701996),\n",
       "             ('gini_index@5', 0.9738255665593293),\n",
       "             ('entropy@5', 4.777123828102488),\n",
       "             ('appeared_item@5', 536.0),\n",
       "             ('hit@10', 0.8925496688741722),\n",
       "             ('recall@10', 0.15053594583416965),\n",
       "             ('ndcg@10', 0.3662089065311077),\n",
       "             ('map@10', 0.08982972949880254),\n",
       "             ('precision@10', 0.32049668874172194),\n",
       "             ('gini_index@10', 0.961654047669253),\n",
       "             ('entropy@10', 5.1898345912683315),\n",
       "             ('appeared_item@10', 764.0)])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommender = P3alphaRecommender(X_train, top_k=100)\n",
    "recommender.learn()\n",
    "\n",
    "evaluator.get_scores(recommender, cutoffs=[5, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparison with a simple baseline\n",
    "\n",
    "Now that we have a qualitative way to measure the recommenders' performance,\n",
    "we can compare the performance of different algorithms.\n",
    "\n",
    "As a simple baseline, we fit ``TopPopRecommender``, which recommends items\n",
    "with descending order of the popularity in the train set, regardless of the users'\n",
    "watch event history. (But note that already-seen items by a user will not be commended again)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('hit@5', 0.5473509933774835),\n",
       "             ('recall@5', 0.04097159018092859),\n",
       "             ('ndcg@5', 0.21914854330480912),\n",
       "             ('map@5', 0.025239375245273265),\n",
       "             ('precision@5', 0.20884105960264907),\n",
       "             ('gini_index@5', 0.9972226173414867),\n",
       "             ('entropy@5', 2.608344593727912),\n",
       "             ('appeared_item@5', 42.0),\n",
       "             ('hit@10', 0.6637417218543047),\n",
       "             ('recall@10', 0.0667908851617647),\n",
       "             ('ndcg@10', 0.19939297384808613),\n",
       "             ('map@10', 0.033298013667913656),\n",
       "             ('precision@10', 0.17811258278145692),\n",
       "             ('gini_index@10', 0.9950046639957398),\n",
       "             ('entropy@10', 3.1812807131889786),\n",
       "             ('appeared_item@10', 69.0)])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toppop_recommender = TopPopRecommender(X_train)\n",
    "toppop_recommender.learn()\n",
    "\n",
    "evaluator.get_scores(toppop_recommender, cutoffs=[5, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we see that `P3alphaRecommender` actually exhibits better accuracy scores compared to rather trivial `TopPopRecommender`.\n",
    "\n",
    "In the next tutorial, we will optimize the recommender's performance using the hold-out method."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "2c8d963a96d0919c97b2debbb8f097e0c884c04e5d1a6aef0590fcc7fac9e98a"
  },
  "kernelspec": {
   "display_name": "Python 3.10.0 64-bit ('main')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}