scb_mt_enth_2020

Références:

enth

Utilisez la commande suivante pour charger cet ensemble de données dans TFDS :

ds = tfds.load('huggingface:scb_mt_enth_2020/enth')
  • Descriptif :
scb-mt-en-th-2020: A Large English-Thai Parallel Corpus
The primary objective of our work is to build a large-scale English-Thai dataset for machine translation.
We construct an English-Thai machine translation dataset with over 1 million segment pairs, curated from various sources,
namely news, Wikipedia articles, SMS messages, task-based dialogs, web-crawled data and government documents.
Methodology for gathering data, building parallel texts and removing noisy sentence pairs are presented in a reproducible manner.
We train machine translation models based on this dataset. Our models' performance are comparable to that of
Google Translation API (as of May 2020) for Thai-English and outperform Google when the Open Parallel Corpus (OPUS) is
included in the training data for both Thai-English and English-Thai translation.
The dataset, pre-trained models, and source code to reproduce our work are available for public use.
  • Licence : Aucune licence connue
  • Version : 1.0.0
  • Fractionnements :
Diviser Exemples
'test' 100177
'train' 801402
'validation' 100173
  • Caractéristiques :
{
    "translation": {
        "languages": [
            "en",
            "th"
        ],
        "id": null,
        "_type": "Translation"
    },
    "subdataset": {
        "dtype": "string",
        "id": null,
        "_type": "Value"
    }
}

alors

Utilisez la commande suivante pour charger cet ensemble de données dans TFDS :

ds = tfds.load('huggingface:scb_mt_enth_2020/then')
  • Descriptif :
scb-mt-en-th-2020: A Large English-Thai Parallel Corpus
The primary objective of our work is to build a large-scale English-Thai dataset for machine translation.
We construct an English-Thai machine translation dataset with over 1 million segment pairs, curated from various sources,
namely news, Wikipedia articles, SMS messages, task-based dialogs, web-crawled data and government documents.
Methodology for gathering data, building parallel texts and removing noisy sentence pairs are presented in a reproducible manner.
We train machine translation models based on this dataset. Our models' performance are comparable to that of
Google Translation API (as of May 2020) for Thai-English and outperform Google when the Open Parallel Corpus (OPUS) is
included in the training data for both Thai-English and English-Thai translation.
The dataset, pre-trained models, and source code to reproduce our work are available for public use.
  • Licence : Aucune licence connue
  • Version : 1.0.0
  • Fractionnements :
Diviser Exemples
'test' 100177
'train' 801402
'validation' 100173
  • Caractéristiques :
{
    "translation": {
        "languages": [
            "th",
            "en"
        ],
        "id": null,
        "_type": "Translation"
    },
    "subdataset": {
        "dtype": "string",
        "id": null,
        "_type": "Value"
    }
}