irspack.dataset.MINDDataManager#

class irspack.dataset.MINDDataManager(train_zippath=None, dev_zippath=None, size='small', force_download=False, download_source='huggingface', hf_token=None)[source]#

Bases: object

Manage the Microsoft News Dataset (MIND).

MIND contains timestamped news impressions as well as article categories, titles, abstracts, linked Wikidata entities, and knowledge-graph embeddings. The small variant is the default because it is suitable for local experiments (the two archives are roughly 190 MB in total).

The official training and development archives are chronologically separated. read_interaction() only returns clicks from the timestamped impression field; it deliberately does not expand history because timestamps for those historical clicks are not available.

Before downloading MIND, read the Microsoft Research License Terms linked from https://msnews.github.io/. Setting force_download=True confirms that you have read and accept those terms.

Parameters:

  • train_zippath (Path | str | None) – Path to the training archive. Defaults to ~/.mind/MIND{size}_train.zip.

  • dev_zippath (Path | str | None) – Path to the development archive. Defaults to ~/.mind/MIND{size}_dev.zip.

  • size (str) – Either "small" or "large".

  • force_download (bool) – Download missing archives without an interactive confirmation. This also confirms acceptance of the dataset’s license terms.

  • download_source (str) – "huggingface" (default) downloads the original archives from the gated yjw1029/MIND mirror. "azure" retains the legacy official URL, which currently returns HTTP 409 because anonymous blob access has been disabled.

  • hf_token (str | None) – Hugging Face read token. If omitted, HF_TOKEN and then HUGGING_FACE_HUB_TOKEN are checked. The token is only needed when downloading from Hugging Face, not when archives already exist locally.

__init__(train_zippath=None, dev_zippath=None, size='small', force_download=False, download_source='huggingface', hf_token=None)[source]#
Parameters:

  • train_zippath (Path | str | None)

  • dev_zippath (Path | str | None)

  • size (str)

  • force_download (bool)

  • download_source (str)

  • hf_token (str | None)

Methods

__init__([train_zippath, dev_zippath, size, ...])

close()

Close the training and development archives.

read_behaviors([split])

Read the original impression-level behavior log.

read_entity_embeddings([split])

Read 100-dimensional TransE entity embeddings.

read_impressions([split, clicked_only])

Read candidate impressions in long form.

read_interaction([split])

Read positive, timestamped item interactions.

read_item_info([split, parse_entities])

Read article metadata, indexed by item_id.

read_relation_embeddings([split])

Read 100-dimensional TransE relation embeddings.

Attributes

AZURE_BASE_URL

DEFAULT_DIR

HUGGINGFACE_BASE_URL

close()[source]#

Close the training and development archives.

Return type:

None

read_behaviors(split='train')[source]#

Read the original impression-level behavior log.

Parameters:

split (str)

Return type:

DataFrame

read_entity_embeddings(split='train')[source]#

Read 100-dimensional TransE entity embeddings.

Parameters:

split (str)

Return type:

DataFrame

read_impressions(split=None, clicked_only=False)[source]#

Read candidate impressions in long form.

Parameters:

  • split (str | None) – "train", "dev", or None to concatenate both.

  • clicked_only (bool) – If true, retain only clicked candidates.

Returns:

A dataframe with impression, user, timestamp, item, position, and click-label columns.

Return type:

DataFrame

read_interaction(split=None)[source]#

Read positive, timestamped item interactions.

history is intentionally excluded because MIND provides neither the individual timestamps nor impression context for those clicks.

Parameters:

split (str | None)

Return type:

DataFrame

read_item_info(split=None, parse_entities=True)[source]#

Read article metadata, indexed by item_id.

When both splits are requested, duplicate articles are returned once. By default the entity JSON fields are parsed into lists of dictionaries.

Parameters:

  • split (str | None)

  • parse_entities (bool)

Return type:

DataFrame

read_relation_embeddings(split='train')[source]#

Read 100-dimensional TransE relation embeddings.

Parameters:

split (str)

Return type:

DataFrame