- Description:
Translate dataset based on the data from statmt.org.
Versions exists for the different years using a combination of multiple data
sources. The base wmt_translate
allows you to create your own config to choose
your own data/language pair by creating a custom tfds.translate.wmt.WmtConfig
.
config = tfds.translate.wmt.WmtConfig(
version="0.0.1",
language_pair=("fr", "de"),
subsets={
tfds.Split.TRAIN: ["commoncrawl_frde"],
tfds.Split.VALIDATION: ["euelections_dev2019"],
},
)
builder = tfds.builder("wmt_translate", config=config)
Source code:
tfds.translate.Wmt18Translate
Versions:
1.0.0
(default): No release notes.
Download size:
Unknown size
Dataset size:
Unknown size
Manual download instructions: This dataset requires you to download the source data manually into
download_config.manual_dir
(defaults to~/tensorflow_datasets/downloads/manual/
):
Some of the wmt configs here, require a manual download. Please look into wmt.py to see the exact path (and file name) that has to be downloaded.Auto-cached (documentation): Unknown
Splits:
Split | Examples |
---|
Figure (tfds.show_examples): Not supported.
Examples (tfds.as_dataframe): Missing.
Citation:
@InProceedings{bojar-EtAl:2018:WMT1,
author = {Bojar, Ond
{r}ej and Federmann, Christian and Fishel, Mark
and Graham, Yvette and Haddow, Barry and Huck, Matthias and
Koehn, Philipp and Monz, Christof},
title = {Findings of the 2018 Conference on Machine Translation (WMT18)},
booktitle = {Proceedings of the Third Conference on Machine Translation,
Volume 2: Shared Task Papers},
month = {October},
year = {2018},
address = {Belgium, Brussels},
publisher = {Association for Computational Linguistics},
pages = {272--307},
url = {http://www.aclweb.org/anthology/W18-6401}
}
wmt18_translate/cs-en (default config)
Config description: WMT 2018 cs-en translation task dataset.
Features:
Translation({
'cs': Text(shape=(), dtype=tf.string),
'en': Text(shape=(), dtype=tf.string),
})
- Supervised keys (See
as_supervised
doc):('cs', 'en')
wmt18_translate/de-en
Config description: WMT 2018 de-en translation task dataset.
Features:
Translation({
'de': Text(shape=(), dtype=tf.string),
'en': Text(shape=(), dtype=tf.string),
})
- Supervised keys (See
as_supervised
doc):('de', 'en')
wmt18_translate/et-en
Config description: WMT 2018 et-en translation task dataset.
Features:
Translation({
'en': Text(shape=(), dtype=tf.string),
'et': Text(shape=(), dtype=tf.string),
})
- Supervised keys (See
as_supervised
doc):('et', 'en')
wmt18_translate/fi-en
Config description: WMT 2018 fi-en translation task dataset.
Features:
Translation({
'en': Text(shape=(), dtype=tf.string),
'fi': Text(shape=(), dtype=tf.string),
})
- Supervised keys (See
as_supervised
doc):('fi', 'en')
wmt18_translate/kk-en
Config description: WMT 2018 kk-en translation task dataset.
Features:
Translation({
'en': Text(shape=(), dtype=tf.string),
'kk': Text(shape=(), dtype=tf.string),
})
- Supervised keys (See
as_supervised
doc):('kk', 'en')
wmt18_translate/ru-en
Config description: WMT 2018 ru-en translation task dataset.
Features:
Translation({
'en': Text(shape=(), dtype=tf.string),
'ru': Text(shape=(), dtype=tf.string),
})
- Supervised keys (See
as_supervised
doc):('ru', 'en')
wmt18_translate/tr-en
Config description: WMT 2018 tr-en translation task dataset.
Features:
Translation({
'en': Text(shape=(), dtype=tf.string),
'tr': Text(shape=(), dtype=tf.string),
})
- Supervised keys (See
as_supervised
doc):('tr', 'en')
wmt18_translate/zh-en
Config description: WMT 2018 zh-en translation task dataset.
Features:
Translation({
'en': Text(shape=(), dtype=tf.string),
'zh': Text(shape=(), dtype=tf.string),
})
- Supervised keys (See
as_supervised
doc):('zh', 'en')