<br />

- **Description**:

The Answer Equivalence Dataset contains human ratings on model predictions from
several models on the SQuAD dataset. The ratings establish whether the predicted
answer is 'equivalent' to the gold answer (taking into account both question and
context).

More specifically, by 'equivalent' we mean that the predicted answer contains at
least the same information as the gold answer and does not add superfluous
information. The dataset contains annotations for: \* predictions from BiDAF on
SQuAD dev \* predictions from XLNet on SQuAD dev \* predictions from Luke on SQuAD
dev \* predictions from Albert on SQuAD training, dev and test examples

- **Homepage** :
  <https://github.com/google-research-datasets/answer-equivalence-dataset>

- **Source code** :
  [`tfds.datasets.answer_equivalence.Builder`](https://github.com/tensorflow/datasets/tree/master/tensorflow_datasets/datasets/answer_equivalence/answer_equivalence_dataset_builder.py)

- **Versions**:

  - **`1.0.0`** (default): Initial release.
- **Download size** : `45.86 MiB`

- **Dataset size** : `47.24 MiB`

- **Auto-cached**
  ([documentation](https://www.tensorflow.org/datasets/performances#auto-caching)):
  Yes

- **Splits**:

| Split | Examples |
|---|---|
| `'ae_dev'` | 4,446 |
| `'ae_test'` | 9,724 |
| `'dev_bidaf'` | 7,522 |
| `'dev_luke'` | 4,590 |
| `'dev_xlnet'` | 7,932 |
| `'train'` | 9,090 |

- **Feature structure**:

    FeaturesDict({
        'candidate': Text(shape=(), dtype=string),
        'context': Text(shape=(), dtype=string),
        'gold_index': int32,
        'qid': Text(shape=(), dtype=string),
        'question': Text(shape=(), dtype=string),
        'question_1': ClassLabel(shape=(), dtype=int64, num_classes=3),
        'question_2': ClassLabel(shape=(), dtype=int64, num_classes=3),
        'question_3': ClassLabel(shape=(), dtype=int64, num_classes=3),
        'question_4': ClassLabel(shape=(), dtype=int64, num_classes=3),
        'reference': Text(shape=(), dtype=string),
        'score': float32,
    })

- **Feature documentation**:

| Feature | Class | Shape | Dtype | Description |
|---|---|---|---|---|
|   | FeaturesDict |   |   |   |
| candidate | Text |   | string |   |
| context | Text |   | string |   |
| gold_index | Tensor |   | int32 |   |
| qid | Text |   | string |   |
| question | Text |   | string |   |
| question_1 | ClassLabel |   | int64 |   |
| question_2 | ClassLabel |   | int64 |   |
| question_3 | ClassLabel |   | int64 |   |
| question_4 | ClassLabel |   | int64 |   |
| reference | Text |   | string |   |
| score | Tensor |   | float32 |   |

- **Supervised keys** (See
  [`as_supervised` doc](https://www.tensorflow.org/datasets/api_docs/python/tfds/load#args)):
  `None`

- **Figure**
  ([tfds.show_examples](https://www.tensorflow.org/datasets/api_docs/python/tfds/visualization/show_examples)):
  Not supported.

- **Examples**
  ([tfds.as_dataframe](https://www.tensorflow.org/datasets/api_docs/python/tfds/as_dataframe)):

<iframe src="https:///frame/datasets/catalog/answer_equivalence_61a16df1f9fd4cf549c12d29c4d1d86509e57c3986815078b82b5d16ab5ce06f.frame" class="framebox inherit-locale " allow="clipboard-write https://" allowfullscreen is-upgraded></iframe>

- **Citation**:

    @article{bulian-etal-2022-tomayto,
          title={Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation},
          author={Jannis Bulian and Christian Buck and Wojciech Gajewski and Benjamin Boerschinger and Tal Schuster},
          year={2022},
          eprint={2202.07654},
          archivePrefix={arXiv},
          primaryClass={cs.CL}
    }