{ "cells": [ { "cell_type": "markdown", "id": "9f530e19", "metadata": {}, "source": [ "# Sampled item evaluation protocol\n", "\n", "This notebook aims to reproduce the result for ML1M & Pinterest in the paper [\"Revisiting the Performance of iALS on Item Recommendation Benchmarks\"](https://arxiv.org/abs/2110.14037). On these two datasets, for each user, we try to rank 1 held-out positive (actually touched by the user) item over 100 randomly selected negative (untouched) items.\n", "\n", "Since the protocol is widely used for recsys benchmarking after the [NeuMF paper](https://arxiv.org/abs/1708.05031), below we see how we can measure the recommenders' performance following it. Note, however, there is [a study](https://dl.acm.org/doi/10.1145/3394486.3403226) which asserts that this ranking metric may not be a good indicator for recommender performance." ] }, { "cell_type": "code", "execution_count": 1, "id": "ad493486-f40f-4403-bf28-96320374f53c", "metadata": {}, "outputs": [], "source": [ "from IPython.display import clear_output\n", "from irspack import Evaluator, IALSRecommender, df_to_sparse, split_last_n_interaction_df\n", "from irspack.dataset.neu_mf import NeuMFML1MDownloader, NeuMFMPinterestDownloader\n", "import numpy as np\n", "import pandas as pd\n", "\n", "\n", "# Either ml-1m or pinterest\n", "DATA_TYPE = 'ml-1m'\n", "assert DATA_TYPE in ['ml-1m', 'pinterest']\n", "\n", "USER = 'user_id'\n", "ITEM = 'item_id'\n", "TIME = 'timestamp'\n", "\n", "if DATA_TYPE == 'ml-1m':\n", " dm = NeuMFML1MDownloader()\n", "else:\n", " dm = NeuMFMPinterestDownloader()" ] }, { "cell_type": "markdown", "id": "996da3fb", "metadata": {}, "source": [ "## Read the train & test dataset\n", "\n", "The train set is a usual user/item interaction dataframe." ] }, { "cell_type": "code", "execution_count": 2, "id": "146c9ec5-d90e-4c95-970c-4c3efc373733", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_iditem_idratingtimestamp
003242001-01-06 23:38:50
103442001-01-06 23:38:50
20452001-01-06 23:38:11
303542001-01-06 23:38:11
403042001-01-06 23:38:11
\n", "
" ], "text/plain": [ " user_id item_id rating timestamp\n", "0 0 32 4 2001-01-06 23:38:50\n", "1 0 34 4 2001-01-06 23:38:50\n", "2 0 4 5 2001-01-06 23:38:11\n", "3 0 35 4 2001-01-06 23:38:11\n", "4 0 30 4 2001-01-06 23:38:11" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train, test = dm.read_train_test()\n", "train.head()" ] }, { "cell_type": "code", "execution_count": 3, "id": "a1525773", "metadata": {}, "outputs": [], "source": [ "item_list = sorted(list(set(train['item_id'])))\n", "item_set = set(item_list)" ] }, { "cell_type": "markdown", "id": "3a9d42d6", "metadata": {}, "source": [ "## Create validation data\n", "\n", "Split `train` into train (`tt`) & validation (`tv`) pair.\n", "\n", "The validation data is created in the same way as the test set." ] }, { "cell_type": "code", "execution_count": 4, "id": "3038367a-af71-40e0-86ff-f797b8fd838e", "metadata": {}, "outputs": [], "source": [ "g=train.groupby('user_id')['item_id']\n", "user_id_vs_interacted_items = g.agg(set).to_dict()\n", "\n", "rng = np.random.default_rng(0)\n", "\n", "# tv is users' last interaction with item.\n", "tt, tv = split_last_n_interaction_df(train, USER, timestamp_column=TIME, n_heldout=1)\n", "tv['positive'] = True\n", "dfs = []\n", "for user_id in tv[USER]:\n", " items_not_interacted = list(item_set - user_id_vs_interacted_items[user_id])\n", " negatives = rng.choice(items_not_interacted, size=100, replace=False)\n", " dfs.append(pd.DataFrame({USER: user_id, ITEM: negatives}))\n", "valid = pd.concat(dfs)\n", "valid['positive'] = False\n", "valid = pd.concat([valid, tv[[USER, ITEM, 'positive']]]).sort_values([USER, 'positive'])" ] }, { "cell_type": "markdown", "id": "299af34b", "metadata": {}, "source": [ "The validation dataframe has an extra column to indicate the positivity of the pair." ] }, { "cell_type": "code", "execution_count": 5, "id": "c5b60908", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_iditem_idpositive
001014False
10131False
201281False
302669False
40372False
\n", "
" ], "text/plain": [ " user_id item_id positive\n", "0 0 1014 False\n", "1 0 131 False\n", "2 0 1281 False\n", "3 0 2669 False\n", "4 0 372 False" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "valid.head()" ] }, { "cell_type": "markdown", "id": "4a67d2de", "metadata": {}, "source": [ "Let us convert the data frame into sparse matrix." ] }, { "cell_type": "code", "execution_count": 6, "id": "d54b3cd3-fc99-4b20-b7c2-e7e0f51d32ad", "metadata": {}, "outputs": [], "source": [ "X_tt, tt_users, _ = df_to_sparse(tt, USER, ITEM, item_ids=item_list)\n", "X_tv_gt, _, __ = df_to_sparse(valid[valid['positive']], USER, ITEM, user_ids=tt_users, item_ids=item_list)\n", "X_tv_recommendable, _, __ = df_to_sparse(valid, USER, ITEM, user_ids=tt_users, item_ids=item_list)" ] }, { "cell_type": "markdown", "id": "fe215b64", "metadata": {}, "source": [ "\n", "- Non-zeroes in `X_truth` indicate the positive pair location.\n", "- Non-zeroes in `X_recommendable` are positive & randomly seledted negative pairs.\n", "\n", "In the parameter tuning procedure, I found that too eary start of pruning harms the final quality of recommendation.\n", "We can control the parameters of pruning by explicitly providing `optuna.Study`." ] }, { "cell_type": "code", "execution_count": 7, "id": "49988a11-5657-4c71-8ff5-eeeda2a90fc5", "metadata": {}, "outputs": [], "source": [ "validation_evaluator = Evaluator(X_tv_gt, per_user_recommendable_items=X_tv_recommendable, cutoff=10)\n", "best_parameter, validation_recoder = IALSRecommender.tune(\n", " X_tt, validation_evaluator, n_components=192,\n", " n_trials=40, random_seed=0, prunning_n_startup_trials=20\n", ")\n", "clear_output()" ] }, { "cell_type": "code", "execution_count": 8, "id": "00f5e431", "metadata": {}, "outputs": [], "source": [ "X_train_all, user_ids, _ = df_to_sparse(train, USER, ITEM, item_ids=item_list)\n", "X_test_gt, _, __ = df_to_sparse(test[test[\"positive\"]], USER, ITEM, user_ids=user_ids, item_ids=item_list)\n", "X_test_recommendable, _, __ = df_to_sparse(test, USER, ITEM, user_ids=user_ids, item_ids=item_list)" ] }, { "cell_type": "markdown", "id": "5452f612-1cd8-40da-9c24-0bb098c01e10", "metadata": {}, "source": [ "NDCG@10/HIT@10 is similar to that reported in the reference." ] }, { "cell_type": "code", "execution_count": 9, "id": "031e02cf", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", " \n", " 100.00% [6/6 00:00<00:00]\n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "{'appeared_item': 2714.0,\n", " 'entropy': 7.405239384868466,\n", " 'gini_index': 0.6666932769922957,\n", " 'hit': 0.7310367671414376,\n", " 'map': 0.3656433198210277,\n", " 'n_items': 3704.0,\n", " 'ndcg': 0.4521301417636353,\n", " 'precision': 0.07310367671414376,\n", " 'recall': 0.7310367671414376,\n", " 'total_user': 6040.0,\n", " 'valid_user': 6038.0}" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Evaluator(X_test_gt, per_user_recommendable_items=X_test_recommendable, cutoff=10).get_score(\n", " IALSRecommender(X_train_all, **best_parameter).learn()\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "de20957a-70cd-4914-9402-463bde378920", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3.10.0 64-bit ('main')", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.0" }, "vscode": { "interpreter": { "hash": "2c8d963a96d0919c97b2debbb8f097e0c884c04e5d1a6aef0590fcc7fac9e98a" } } }, "nbformat": 4, "nbformat_minor": 5 }