- Description:
Translate dataset based on the data from statmt.org.
Versions exists for the different years using a combination of multiple data
sources. The base wmt_translate
allows you to create your own config to choose
your own data/language pair by creating a custom tfds.translate.wmt.WmtConfig
.
config = tfds.translate.wmt.WmtConfig(
version="0.0.1",
language_pair=("fr", "de"),
subsets={
tfds.Split.TRAIN: ["commoncrawl_frde"],
tfds.Split.VALIDATION: ["euelections_dev2019"],
},
)
builder = tfds.builder("wmt_translate", config=config)
Additional Documentation: Explore on Papers With Code
Source code:
tfds.translate.Wmt14Translate
Versions:
1.0.0
(default): No release notes.
Manual download instructions: This dataset requires you to download the source data manually into
download_config.manual_dir
(defaults to~/tensorflow_datasets/downloads/manual/
):
Some of the wmt configs here, require a manual download. Please look into wmt.py to see the exact path (and file name) that has to be downloaded.Figure (tfds.show_examples): Not supported.
Citation:
@InProceedings{bojar-EtAl:2014:W14-33,
author = {Bojar, Ondrej and Buck, Christian and Federmann, Christian and Haddow, Barry and Koehn, Philipp and Leveling, Johannes and Monz, Christof and Pecina, Pavel and Post, Matt and Saint-Amand, Herve and Soricut, Radu and Specia, Lucia and Tamchyna, Ale
{s} },
title = {Findings of the 2014 Workshop on Statistical Machine Translation},
booktitle = {Proceedings of the Ninth Workshop on Statistical Machine Translation},
month = {June},
year = {2014},
address = {Baltimore, Maryland, USA},
publisher = {Association for Computational Linguistics},
pages = {12--58},
url = {http://www.aclweb.org/anthology/W/W14/W14-3302}
}
wmt14_translate/cs-en (default config)
Config description: WMT 2014 cs-en translation task dataset.
Download size:
1.58 GiB
Dataset size:
2.89 GiB
Auto-cached (documentation): No
Splits:
Split | Examples |
---|---|
'test' |
3,003 |
'train' |
15,786,979 |
'validation' |
3,000 |
- Feature structure:
Translation({
'cs': Text(shape=(), dtype=string),
'en': Text(shape=(), dtype=string),
})
- Feature documentation:
Feature | Class | Shape | Dtype | Description |
---|---|---|---|---|
Translation | ||||
cs | Text | string | ||
en | Text | string |
Supervised keys (See
as_supervised
doc):('cs', 'en')
Examples (tfds.as_dataframe):
wmt14_translate/de-en
Config description: WMT 2014 de-en translation task dataset.
Download size:
1.58 GiB
Dataset size:
1.36 GiB
Auto-cached (documentation): No
Splits:
Split | Examples |
---|---|
'test' |
3,003 |
'train' |
4,508,785 |
'validation' |
3,000 |
- Feature structure:
Translation({
'de': Text(shape=(), dtype=string),
'en': Text(shape=(), dtype=string),
})
- Feature documentation:
Feature | Class | Shape | Dtype | Description |
---|---|---|---|---|
Translation | ||||
de | Text | string | ||
en | Text | string |
Supervised keys (See
as_supervised
doc):('de', 'en')
Examples (tfds.as_dataframe):
wmt14_translate/fr-en
Config description: WMT 2014 fr-en translation task dataset.
Download size:
6.20 GiB
Dataset size:
14.65 GiB
Auto-cached (documentation): No
Splits:
Split | Examples |
---|---|
'test' |
3,003 |
'train' |
40,836,876 |
'validation' |
3,000 |
- Feature structure:
Translation({
'en': Text(shape=(), dtype=string),
'fr': Text(shape=(), dtype=string),
})
- Feature documentation:
Feature | Class | Shape | Dtype | Description |
---|---|---|---|---|
Translation | ||||
en | Text | string | ||
fr | Text | string |
Supervised keys (See
as_supervised
doc):('fr', 'en')
Examples (tfds.as_dataframe):
wmt14_translate/hi-en
Config description: WMT 2014 hi-en translation task dataset.
Download size:
44.65 MiB
Dataset size:
77.53 MiB
Auto-cached (documentation): Yes
Splits:
Split | Examples |
---|---|
'test' |
2,507 |
'train' |
313,748 |
'validation' |
520 |
- Feature structure:
Translation({
'en': Text(shape=(), dtype=string),
'hi': Text(shape=(), dtype=string),
})
- Feature documentation:
Feature | Class | Shape | Dtype | Description |
---|---|---|---|---|
Translation | ||||
en | Text | string | ||
hi | Text | string |
Supervised keys (See
as_supervised
doc):('hi', 'en')
Examples (tfds.as_dataframe):
wmt14_translate/ru-en
Config description: WMT 2014 ru-en translation task dataset.
Download size:
998.38 MiB
Dataset size:
839.40 MiB
Auto-cached (documentation): No
Splits:
Split | Examples |
---|---|
'test' |
3,003 |
'train' |
2,486,965 |
'validation' |
3,000 |
- Feature structure:
Translation({
'en': Text(shape=(), dtype=string),
'ru': Text(shape=(), dtype=string),
})
- Feature documentation:
Feature | Class | Shape | Dtype | Description |
---|---|---|---|---|
Translation | ||||
en | Text | string | ||
ru | Text | string |
Supervised keys (See
as_supervised
doc):('ru', 'en')
Examples (tfds.as_dataframe):