irspack.evaluation.EvaluatorWithColdUser#

class irspack.evaluation.EvaluatorWithColdUser(input_interaction, ground_truth, cutoff=10, target_metric='ndcg', recommendable_items=None, per_user_recommendable_items=None, masked_interactions=None, n_threads=None, recall_with_cutoff=False, mb_size=1024)[source]#

Bases: Evaluator

Evaluates recommenders’ performance against cold (unseen) users.

Parameters:

  • input_interaction (Union[scipy.sparse.csr_matrix, scipy.sparse.csc_matrix]) – The cold-users’ known interaction with the items.

  • ground_truth (Union[scipy.sparse.csr_matrix, scipy.sparse.csc_matrix]) – The held-out ground-truth.

  • offset (int) – Where the validation target user block begins. Often the validation set is defined for a subset of users. When offset is not 0, we assume that the users with validation ground truth corresponds to X_train[offset:] where X_train is the matrix feeded into the recommender class.

  • cutoff (int, optional) – Controls the number of recommendation. Defaults to 10.

  • target_metric (str, optional) – Optimization target metric. Defaults to “ndcg”.

  • recommendable_items (Optional[List[int]], optional) – Global recommendable items. Defaults to None. If this parameter is not None, evaluator will be concentrating on the recommender’s score output for these recommendable_items, and compute the ranking performance within this subset.

  • per_user_recommendable_items (Optional[List[List[int]]], optional) – Similar to recommendable_items, but this time the recommendable items can vary among users. Defaults to None.

  • masked_interactions (Optional[Union[scipy.sparse.csr_matrix, scipy.sparse.csc_matrix]], optional) – If set, this matrix masks the score output of recommender model where it is non-zero. If none, the mask will be the training matrix (input_interaction) it self.

  • n_threads (int, optional) – Specifies the Number of threads to sort scores and compute the evaluation metrics. If None, the environment variable "IRSPACK_NUM_THREADS_DEFAULT" will be looked up, and if the variable is not set, it will be set to os.cpu_count(). Defaults to None.

  • recall_with_cutoff (bool, optional) –

    This affects the definition of recall. If True, for each user, recall will be evaluated by

    \[\frac{N_{\text{hit}}}{\min( \text{cutoff}, N_{\text{ground truth}} )}\]

    If False, this will be

    \[\frac{N_{\text{hit}}}{N_{\text{ground truth}}}\]

  • mb_size (int, optional) – The rows of chunked user score. Defaults to 1024.

__init__(input_interaction, ground_truth, cutoff=10, target_metric='ndcg', recommendable_items=None, per_user_recommendable_items=None, masked_interactions=None, n_threads=None, recall_with_cutoff=False, mb_size=1024)[source]#
Parameters:

  • input_interaction (csr_matrix | csc_matrix)

  • ground_truth (csr_matrix | csc_matrix)

  • cutoff (int)

  • target_metric (str)

  • recommendable_items (List[int] | None)

  • per_user_recommendable_items (None | List[List[int]] | csr_matrix | csc_matrix)

  • masked_interactions (csr_matrix | csc_matrix | None)

  • n_threads (int | None)

  • recall_with_cutoff (bool)

  • mb_size (int)

Methods

__init__(input_interaction, ground_truth[, ...])

get_score(model)

Compute the score with the cutoff being self.cutoff.

get_score_from_score_matrix(scores)

Compute metrics from a user-by-item score matrix.

get_scores(model, cutoffs)

Compute the score with the specified cutoffs.

get_scores_from_score_matrix(scores, cutoffs)

Compute metrics at multiple cutoffs from a user-by-item score matrix.

get_target_score(model)

Compute the optimization target score (self.target_metric) with the cutoff being self.cutoff.

Attributes

n_users

n_items

masked_interactions

get_score(model)#

Compute the score with the cutoff being self.cutoff.

Parameters:

model (BaseRecommender) – The evaluated recommender.

Returns:

metric values.

Return type:

Dict[str, float]

get_score_from_score_matrix(scores)#

Compute metrics from a user-by-item score matrix.

The score matrix must have shape (n_users, n_items), in the same order as the ground truth supplied to this evaluator. Scores are copied before masking, so the input array is not modified.

When masked_interactions was supplied to the evaluator, its entries are excluded from the ranking. Otherwise, no entries are masked.

Parameters:

scores (ndarray) – A float32 or float64 user-by-item score matrix.

Returns:

Metric values for this evaluator’s default cutoff.

Return type:

Dict[str, float]

get_scores(model, cutoffs)#

Compute the score with the specified cutoffs.

Parameters:

  • model (BaseRecommender) – The evaluated recommender.

  • cutoffs (List[int]) – for each value in cutoff, the class computes the metric values.

Returns:

The Resulting metric values. This time, the result will look like {"ndcg@20": 0.35, "map@20": 0.2, ...}.

Return type:

Dict[str, float]

get_scores_from_score_matrix(scores, cutoffs)#

Compute metrics at multiple cutoffs from a user-by-item score matrix.

The score matrix must have shape (n_users, n_items), in the same order as the ground truth supplied to this evaluator. Scores are copied before masking, so the input array is not modified.

Parameters:

  • scores (ndarray) – A float32 or float64 user-by-item score matrix.

  • cutoffs (List[int]) – Cutoffs at which to compute metrics.

Returns:

Metric values keyed by metric name and cutoff.

Return type:

Dict[str, float]

get_target_score(model)#

Compute the optimization target score (self.target_metric) with the cutoff being self.cutoff.

Parameters:

model (BaseRecommender) – The evaluated model.

Returns:

The metric value.

Return type:

float