irspack.evaluation.Evaluator#

class irspack.evaluation.Evaluator(ground_truth, offset=0, cutoff=10, target_metric='ndcg', recommendable_items=None, per_user_recommendable_items=None, masked_interactions=None, n_threads=None, recall_with_cutoff=False, mb_size=128)[source]#

Bases: object

Evaluates recommenders’ performance against validation set.

Parameters:

  • ground_truth (Union[scipy.sparse.csr_matrix, scipy.sparse.csc_matrix]) – The ground-truth.

  • offset (int) – Where the validation target user block begins. Often the validation set is defined for a subset of users. When offset is not 0, we assume that the users with validation ground truth corresponds to X_train[offset:] where X_train is the matrix feeded into the recommender class. Defaults to 0.

  • cutoff (int, optional) – Controls the default number of recommendation. When the evaluator is used for parameter tuning, this cutoff value will be used. Defaults to 10.

  • target_metric (str, optional) – Specifies the target metric when this evaluator is used for parameter tuning. Defaults to “ndcg”.

  • recommendable_items (Optional[List[int]], optional) – Global recommendable items. Defaults to None. If this parameter is not None, evaluator will be concentrating on the recommender’s score output for these recommendable_items, and compute the ranking performance within this subset.

  • per_user_recommendable_items (None | List[List[int]] | csr_matrix | csc_matrix) – Similar to recommendable_items, but this time the recommendable items can vary among users. If a sparse matrix is given, its nonzero indices are regarded as the list of recommendable items. Defaults to None.

  • masked_interactions (csr_matrix | None) – If set, this matrix masks the score output of recommender model where it is non-zero. If none, the mask will be the training matrix itself owned by the recommender.

  • n_threads (int | None) – Specifies the Number of threads to sort scores and compute the evaluation metrics. If None, the environment variable "IRSPACK_NUM_THREADS_DEFAULT"` will be looked up, and if the variable is not set, it will be set to ``os.cpu_count(). Defaults to None.

  • recall_with_cutoff (bool, optional) –

    This affects the definition of recall. If True, for each user, recall will be computed as

    \[\frac{N_{\text{hit}}}{\min(\text{cutoff}, N_{\text{ground truth}})}\]

    If False, this will be

    \[\frac{N_{\text{hit}}}{N_{\text{ground truth}}}\]

  • mb_size (int, optional) – The rows of chunked user score. Defaults to 1024.

__init__(ground_truth, offset=0, cutoff=10, target_metric='ndcg', recommendable_items=None, per_user_recommendable_items=None, masked_interactions=None, n_threads=None, recall_with_cutoff=False, mb_size=128)[source]#
Parameters:

  • ground_truth (csr_matrix | csc_matrix)

  • offset (int)

  • cutoff (int)

  • target_metric (str)

  • recommendable_items (List[int] | None)

  • per_user_recommendable_items (None | List[List[int]] | csr_matrix | csc_matrix)

  • masked_interactions (csr_matrix | csc_matrix | None)

  • n_threads (int | None)

  • recall_with_cutoff (bool)

  • mb_size (int)

Return type:

None

Methods

__init__(ground_truth[, offset, cutoff, ...])

get_score(model)

Compute the score with the cutoff being self.cutoff.

get_score_from_score_matrix(scores)

Compute metrics from a user-by-item score matrix.

get_scores(model, cutoffs)

Compute the score with the specified cutoffs.

get_scores_from_score_matrix(scores, cutoffs)

Compute metrics at multiple cutoffs from a user-by-item score matrix.

get_target_score(model)

Compute the optimization target score (self.target_metric) with the cutoff being self.cutoff.

Attributes

n_users

n_items

masked_interactions

get_score(model)[source]#

Compute the score with the cutoff being self.cutoff.

Parameters:

model (BaseRecommender) – The evaluated recommender.

Returns:

metric values.

Return type:

Dict[str, float]

get_score_from_score_matrix(scores)[source]#

Compute metrics from a user-by-item score matrix.

The score matrix must have shape (n_users, n_items), in the same order as the ground truth supplied to this evaluator. Scores are copied before masking, so the input array is not modified.

When masked_interactions was supplied to the evaluator, its entries are excluded from the ranking. Otherwise, no entries are masked.

Parameters:

scores (ndarray) – A float32 or float64 user-by-item score matrix.

Returns:

Metric values for this evaluator’s default cutoff.

Return type:

Dict[str, float]

get_scores(model, cutoffs)[source]#

Compute the score with the specified cutoffs.

Parameters:

  • model (BaseRecommender) – The evaluated recommender.

  • cutoffs (List[int]) – for each value in cutoff, the class computes the metric values.

Returns:

The Resulting metric values. This time, the result will look like {"ndcg@20": 0.35, "map@20": 0.2, ...}.

Return type:

Dict[str, float]

get_scores_from_score_matrix(scores, cutoffs)[source]#

Compute metrics at multiple cutoffs from a user-by-item score matrix.

The score matrix must have shape (n_users, n_items), in the same order as the ground truth supplied to this evaluator. Scores are copied before masking, so the input array is not modified.

Parameters:

  • scores (ndarray) – A float32 or float64 user-by-item score matrix.

  • cutoffs (List[int]) – Cutoffs at which to compute metrics.

Returns:

Metric values keyed by metric name and cutoff.

Return type:

Dict[str, float]

get_target_score(model)[source]#

Compute the optimization target score (self.target_metric) with the cutoff being self.cutoff.

Parameters:

model (BaseRecommender) – The evaluated model.

Returns:

The metric value.

Return type:

float