beir

• Description:

BEIR is a heterogeneous benchmark containing diverse IR tasks. It also provides a common and easy framework for evaluation of your NLP-based retrieval models within the benchmark.

• Additional Documentation: Explore on Papers With Code north_east

• Homepage: https://github.com/beir-cellar/beir

• Source code: tfds.datasets.beir.Builder

• Versions:

  • 1.0.0 (default): Initial release.

• Feature structure:

FeaturesDict({
    'passage': Text(shape=(), dtype=string),
    'passage_id': Text(shape=(), dtype=string),
    'passage_metadata': Text(shape=(), dtype=string),
    'query': Text(shape=(), dtype=string),
    'query_id': Text(shape=(), dtype=string),
    'query_metadata': Text(shape=(), dtype=string),
    'score': float32,
})

• Feature documentation:

• Supervised keys (See as_supervised doc): None

• Figure (tfds.show_examples): Not supported.

• Citation:

@inproceedings{
    thakur2021beir,
    title={ {BEIR}: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models},
    author={Nandan Thakur and Nils Reimers and Andreas R{"u}ckl{'e} and Abhishek Srivastava and Iryna Gurevych},
    booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)},
    year={2021},
    url={https://openreview.net/forum?id=wCu6T5xFjeJ}
}

beir/msmarco (default config)

• Download size: 1.01 GiB

• Dataset size: 4.53 GiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/trec_covid

• Download size: 70.45 MiB

• Dataset size: 292.04 MiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/nfcorpus

• Download size: 2.34 MiB

• Dataset size: 24.94 MiB

• Auto-cached (documentation): Yes

• Splits:

• Examples (tfds.as_dataframe):

beir/nq

• Download size: 475.22 MiB

• Dataset size: 1.66 GiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/hotpotqa

• Download size: 623.73 MiB

• Dataset size: 2.64 GiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/fiqa

• Download size: 17.12 MiB

• Dataset size: 73.39 MiB

• Auto-cached (documentation): Yes

• Splits:

• Examples (tfds.as_dataframe):

beir/arguana

• Download size: 3.60 MiB

• Dataset size: 15.08 MiB

• Auto-cached (documentation): Yes

• Splits:

• Examples (tfds.as_dataframe):

beir/webis_touche2020

• Download size: 216.61 MiB

• Dataset size: 747.57 MiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/cqadupstack.android

• Download size: 4.98 GiB

• Dataset size: 883.49 MiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/cqadupstack.english

• Download size: 4.98 GiB

• Dataset size: 3.78 GiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/cqadupstack.gaming

• Download size: 4.98 GiB

• Dataset size: 2.64 GiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/cqadupstack.gis

• Download size: 4.98 GiB

• Dataset size: 1.47 GiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/cqadupstack.mathematica

• Download size: 4.98 GiB

• Dataset size: 1.58 GiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/cqadupstack.physics

• Download size: 4.98 GiB

• Dataset size: 2.02 GiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/cqadupstack.programmers

• Download size: 4.98 GiB

• Dataset size: 2.14 GiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/cqadupstack.stats

• Download size: 4.98 GiB

• Dataset size: 941.07 MiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/cqadupstack.tex

• Download size: 4.98 GiB

• Dataset size: 21.75 GiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/cqadupstack.unix

• Download size: 4.98 GiB

• Dataset size: 2.04 GiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/cqadupstack.webmasters

• Download size: 4.98 GiB

• Dataset size: 650.08 MiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/cqadupstack.wordpress

• Download size: 4.98 GiB

• Dataset size: 726.15 MiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/quora

• Download size: 15.12 MiB

• Dataset size: 121.33 MiB

• Auto-cached (documentation): Yes

• Splits:

• Examples (tfds.as_dataframe):

beir/dbpedia_entity

• Download size: 609.67 MiB

• Dataset size: 2.47 GiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/scidocs

• Download size: 135.87 MiB

• Dataset size: 573.04 MiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/fever

• Download size: 1.15 GiB

• Dataset size: 3.70 GiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/climate_fever

• Download size: 1.14 GiB

• Dataset size: 3.64 GiB

• Auto-cached (documentation): No

• Splits:

• Examples (tfds.as_dataframe):

beir/scifact

• Download size: 2.69 MiB

• Dataset size: 10.16 MiB

• Auto-cached (documentation): Yes

• Splits:

• Examples (tfds.as_dataframe):