- Description:
BLiMP is a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English. BLiMP consists of 67 sub-datasets, each containing 1000 minimal pairs isolating specific contrasts in syntax, morphology, or semantics. The data is automatically generated according to expert-crafted grammars.
Additional Documentation: Explore on Papers With Code
Homepage: https://github.com/alexwarstadt/blimp/tree/master/
Source code:
tfds.datasets.blimp.Builder
Versions:
0.1.0
(default): No release notes.
Auto-cached (documentation): Yes
Splits:
Split | Examples |
---|---|
'train' |
1,000 |
- Feature structure:
FeaturesDict({
'UID': Text(shape=(), dtype=string),
'field': Text(shape=(), dtype=string),
'lexically_identical': bool,
'linguistics_term': Text(shape=(), dtype=string),
'one_prefix_method': bool,
'pair_id': int32,
'sentence_bad': Text(shape=(), dtype=string),
'sentence_good': Text(shape=(), dtype=string),
'simple_LM_method': bool,
'two_prefix_method': bool,
})
- Feature documentation:
Feature | Class | Shape | Dtype | Description |
---|---|---|---|---|
FeaturesDict | ||||
UID | Text | string | ||
field | Text | string | ||
lexically_identical | Tensor | bool | ||
linguistics_term | Text | string | ||
one_prefix_method | Tensor | bool | ||
pair_id | Tensor | int32 | ||
sentence_bad | Text | string | ||
sentence_good | Text | string | ||
simple_LM_method | Tensor | bool | ||
two_prefix_method | Tensor | bool |
Supervised keys (See
as_supervised
doc):None
Figure (tfds.show_examples): Not supported.
Citation:
@article{warstadt2019blimp,
title={BLiMP: A Benchmark of Linguistic Minimal Pairs for English},
author={Warstadt, Alex and Parrish, Alicia and Liu, Haokun and Mohananey, Anhad and Peng, Wei, and Wang, Sheng-Fu and Bowman, Samuel R},
journal={arXiv preprint arXiv:1912.00582},
year={2019}
}
blimp/adjunct_island (default config)
Config description: This configuration includes the paradigm adjunct_island.
Download size:
350.86 KiB
Dataset size:
366.47 KiB
Examples (tfds.as_dataframe):
blimp/anaphor_gender_agreement
Config description: This configuration includes the paradigm anaphor_gender_agreement.
Download size:
426.51 KiB
Dataset size:
332.31 KiB
Examples (tfds.as_dataframe):
blimp/anaphor_number_agreement
Config description: This configuration includes the paradigm anaphor_number_agreement.
Download size:
440.29 KiB
Dataset size:
341.07 KiB
Examples (tfds.as_dataframe):
blimp/animate_subject_passive
Config description: This configuration includes the paradigm animate_subject_passive.
Download size:
451.46 KiB
Dataset size:
345.50 KiB
Examples (tfds.as_dataframe):
blimp/animate_subject_trans
Config description: This configuration includes the paradigm animate_subject_trans.
Download size:
422.95 KiB
Dataset size:
329.27 KiB
Examples (tfds.as_dataframe):
blimp/causative
Config description: This configuration includes the paradigm causative.
Download size:
309.73 KiB
Dataset size:
324.36 KiB
Examples (tfds.as_dataframe):
blimp/complex_NP_island
Config description: This configuration includes the paradigm complex_NP_island.
Download size:
383.17 KiB
Dataset size:
398.78 KiB
Examples (tfds.as_dataframe):
blimp/coordinate_structure_constraint_complex_left_branch
Config description: This configuration includes the paradigm coordinate_structure_constraint_complex_left_branch.
Download size:
558.30 KiB
Dataset size:
410.43 KiB
Examples (tfds.as_dataframe):
blimp/coordinate_structure_constraint_object_extraction
Config description: This configuration includes the paradigm coordinate_structure_constraint_object_extraction.
Download size:
357.47 KiB
Dataset size:
372.10 KiB
Examples (tfds.as_dataframe):
blimp/determiner_noun_agreement_1
Config description: This configuration includes the paradigm determiner_noun_agreement_1.
Download size:
457.66 KiB
Dataset size:
356.93 KiB
Examples (tfds.as_dataframe):
blimp/determiner_noun_agreement_2
Config description: This configuration includes the paradigm determiner_noun_agreement_2.
Download size:
477.40 KiB
Dataset size:
357.01 KiB
Examples (tfds.as_dataframe):
blimp/determiner_noun_agreement_irregular_1
Config description: This configuration includes the paradigm determiner_noun_agreement_irregular_1.
Download size:
463.80 KiB
Dataset size:
365.08 KiB
Examples (tfds.as_dataframe):
blimp/determiner_noun_agreement_irregular_2
Config description: This configuration includes the paradigm determiner_noun_agreement_irregular_2.
Download size:
479.35 KiB
Dataset size:
361.76 KiB
Examples (tfds.as_dataframe):
blimp/determiner_noun_agreement_with_adj_2
Config description: This configuration includes the paradigm determiner_noun_agreement_with_adj_2.
Download size:
514.46 KiB
Dataset size:
379.92 KiB
Examples (tfds.as_dataframe):
blimp/determiner_noun_agreement_with_adj_irregular_1
Config description: This configuration includes the paradigm determiner_noun_agreement_with_adj_irregular_1.
Download size:
487.95 KiB
Dataset size:
384.67 KiB
Examples (tfds.as_dataframe):
blimp/determiner_noun_agreement_with_adj_irregular_2
Config description: This configuration includes the paradigm determiner_noun_agreement_with_adj_irregular_2.
Download size:
516.14 KiB
Dataset size:
384.54 KiB
Examples (tfds.as_dataframe):
blimp/determiner_noun_agreement_with_adjective_1
Config description: This configuration includes the paradigm determiner_noun_agreement_with_adjective_1.
Download size:
492.85 KiB
Dataset size:
385.25 KiB
Examples (tfds.as_dataframe):
blimp/distractor_agreement_relational_noun
Config description: This configuration includes the paradigm distractor_agreement_relational_noun.
Download size:
513.33 KiB
Dataset size:
391.45 KiB
Examples (tfds.as_dataframe):
blimp/distractor_agreement_relative_clause
Config description: This configuration includes the paradigm distractor_agreement_relative_clause.
Download size:
551.53 KiB
Dataset size:
416.14 KiB
Examples (tfds.as_dataframe):
blimp/drop_argument
Config description: This configuration includes the paradigm drop_argument.
Download size:
297.07 KiB
Dataset size:
311.70 KiB
Examples (tfds.as_dataframe):
blimp/ellipsis_n_bar_1
Config description: This configuration includes the paradigm ellipsis_n_bar_1.
Download size:
402.32 KiB
Dataset size:
417.06 KiB
Examples (tfds.as_dataframe):
blimp/ellipsis_n_bar_2
Config description: This configuration includes the paradigm ellipsis_n_bar_2.
Download size:
417.53 KiB
Dataset size:
432.47 KiB
Examples (tfds.as_dataframe):
blimp/existential_there_object_raising
Config description: This configuration includes the paradigm existential_there_object_raising.
Download size:
537.77 KiB
Dataset size:
423.00 KiB
Examples (tfds.as_dataframe):
blimp/existential_there_quantifiers_1
Config description: This configuration includes the paradigm existential_there_quantifiers_1.
Download size:
348.95 KiB
Dataset size:
363.58 KiB
Examples (tfds.as_dataframe):
blimp/existential_there_quantifiers_2
Config description: This configuration includes the paradigm existential_there_quantifiers_2.
Download size:
350.80 KiB
Dataset size:
365.43 KiB
Examples (tfds.as_dataframe):
blimp/existential_there_subject_raising
Config description: This configuration includes the paradigm existential_there_subject_raising.
Download size:
385.21 KiB
Dataset size:
399.84 KiB
Examples (tfds.as_dataframe):
blimp/expletive_it_object_raising
Config description: This configuration includes the paradigm expletive_it_object_raising.
Download size:
573.88 KiB
Dataset size:
437.58 KiB
Examples (tfds.as_dataframe):
blimp/inchoative
Config description: This configuration includes the paradigm inchoative.
Download size:
291.71 KiB
Dataset size:
306.34 KiB
Examples (tfds.as_dataframe):
blimp/intransitive
Config description: This configuration includes the paradigm intransitive.
Download size:
298.33 KiB
Dataset size:
312.96 KiB
Examples (tfds.as_dataframe):
blimp/irregular_past_participle_adjectives
Config description: This configuration includes the paradigm irregular_past_participle_adjectives.
Download size:
434.10 KiB
Dataset size:
345.74 KiB
Examples (tfds.as_dataframe):
blimp/irregular_past_participle_verbs
Config description: This configuration includes the paradigm irregular_past_participle_verbs.
Download size:
410.27 KiB
Dataset size:
327.21 KiB
Examples (tfds.as_dataframe):
blimp/irregular_plural_subject_verb_agreement_1
Config description: This configuration includes the paradigm irregular_plural_subject_verb_agreement_1.
Download size:
449.91 KiB
Dataset size:
366.17 KiB
Examples (tfds.as_dataframe):
blimp/irregular_plural_subject_verb_agreement_2
Config description: This configuration includes the paradigm irregular_plural_subject_verb_agreement_2.
Download size:
442.75 KiB
Dataset size:
354.70 KiB
Examples (tfds.as_dataframe):
blimp/left_branch_island_echo_question
Config description: This configuration includes the paradigm left_branch_island_echo_question.
Download size:
471.31 KiB
Dataset size:
348.84 KiB
Examples (tfds.as_dataframe):
blimp/left_branch_island_simple_question
Config description: This configuration includes the paradigm left_branch_island_simple_question.
Download size:
335.40 KiB
Dataset size:
351.01 KiB
Examples (tfds.as_dataframe):
blimp/matrix_question_npi_licensor_present
Config description: This configuration includes the paradigm matrix_question_npi_licensor_present.
Download size:
447.08 KiB
Dataset size:
354.14 KiB
Examples (tfds.as_dataframe):
blimp/npi_present_1
Config description: This configuration includes the paradigm npi_present_1.
Download size:
427.75 KiB
Dataset size:
339.68 KiB
Examples (tfds.as_dataframe):
blimp/npi_present_2
Config description: This configuration includes the paradigm npi_present_2.
Download size:
412.24 KiB
Dataset size:
329.11 KiB
Examples (tfds.as_dataframe):
blimp/only_npi_licensor_present
Config description: This configuration includes the paradigm only_npi_licensor_present.
Download size:
448.41 KiB
Dataset size:
349.50 KiB
Examples (tfds.as_dataframe):
blimp/only_npi_scope
Config description: This configuration includes the paradigm only_npi_scope.
Download size:
570.04 KiB
Dataset size:
408.52 KiB
Examples (tfds.as_dataframe):
blimp/passive_1
Config description: This configuration includes the paradigm passive_1.
Download size:
332.30 KiB
Dataset size:
346.93 KiB
Examples (tfds.as_dataframe):
blimp/passive_2
Config description: This configuration includes the paradigm passive_2.
Download size:
301.12 KiB
Dataset size:
315.75 KiB
Examples (tfds.as_dataframe):
blimp/principle_A_c_command
Config description: This configuration includes the paradigm principle_A_c_command.
Download size:
515.32 KiB
Dataset size:
388.54 KiB
Examples (tfds.as_dataframe):
blimp/principle_A_case_1
Config description: This configuration includes the paradigm principle_A_case_1.
Download size:
466.05 KiB
Dataset size:
370.87 KiB
Examples (tfds.as_dataframe):
blimp/principle_A_case_2
Config description: This configuration includes the paradigm principle_A_case_2.
Download size:
481.42 KiB
Dataset size:
370.88 KiB
Examples (tfds.as_dataframe):
blimp/principle_A_domain_1
Config description: This configuration includes the paradigm principle_A_domain_1.
Download size:
488.15 KiB
Dataset size:
371.62 KiB
Examples (tfds.as_dataframe):
blimp/principle_A_domain_2
Config description: This configuration includes the paradigm principle_A_domain_2.
Download size:
481.63 KiB
Dataset size:
365.92 KiB
Examples (tfds.as_dataframe):
blimp/principle_A_domain_3
Config description: This configuration includes the paradigm principle_A_domain_3.
Download size:
501.84 KiB
Dataset size:
359.74 KiB
Examples (tfds.as_dataframe):
blimp/principle_A_reconstruction
Config description: This configuration includes the paradigm principle_A_reconstruction.
Download size:
337.40 KiB
Dataset size:
353.00 KiB
Examples (tfds.as_dataframe):
blimp/regular_plural_subject_verb_agreement_1
Config description: This configuration includes the paradigm regular_plural_subject_verb_agreement_1.
Download size:
441.26 KiB
Dataset size:
359.56 KiB
Examples (tfds.as_dataframe):
blimp/regular_plural_subject_verb_agreement_2
Config description: This configuration includes the paradigm regular_plural_subject_verb_agreement_2.
Download size:
445.78 KiB
Dataset size:
354.47 KiB
Examples (tfds.as_dataframe):
blimp/sentential_negation_npi_licensor_present
Config description: This configuration includes the paradigm sentential_negation_npi_licensor_present.
Download size:
479.49 KiB
Dataset size:
372.30 KiB
Examples (tfds.as_dataframe):
blimp/sentential_negation_npi_scope
Config description: This configuration includes the paradigm sentential_negation_npi_scope.
Download size:
600.52 KiB
Dataset size:
431.18 KiB
Examples (tfds.as_dataframe):
blimp/sentential_subject_island
Config description: This configuration includes the paradigm sentential_subject_island.
Download size:
357.25 KiB
Dataset size:
372.86 KiB
Examples (tfds.as_dataframe):
blimp/superlative_quantifiers_1
Config description: This configuration includes the paradigm superlative_quantifiers_1.
Download size:
372.25 KiB
Dataset size:
360.02 KiB
Examples (tfds.as_dataframe):
blimp/superlative_quantifiers_2
Config description: This configuration includes the paradigm superlative_quantifiers_2.
Download size:
504.02 KiB
Dataset size:
360.07 KiB
Examples (tfds.as_dataframe):
blimp/tough_vs_raising_1
Config description: This configuration includes the paradigm tough_vs_raising_1.
Download size:
334.99 KiB
Dataset size:
349.62 KiB
Examples (tfds.as_dataframe):
blimp/tough_vs_raising_2
Config description: This configuration includes the paradigm tough_vs_raising_2.
Download size:
355.54 KiB
Dataset size:
370.17 KiB
Examples (tfds.as_dataframe):
blimp/transitive
Config description: This configuration includes the paradigm transitive.
Download size:
449.50 KiB
Dataset size:
334.45 KiB
Examples (tfds.as_dataframe):
blimp/wh_island
Config description: This configuration includes the paradigm wh_island.
Download size:
438.12 KiB
Dataset size:
343.47 KiB
Examples (tfds.as_dataframe):
blimp/wh_questions_object_gap
Config description: This configuration includes the paradigm wh_questions_object_gap.
Download size:
378.35 KiB
Dataset size:
392.99 KiB
Examples (tfds.as_dataframe):
blimp/wh_questions_subject_gap
Config description: This configuration includes the paradigm wh_questions_subject_gap.
Download size:
380.84 KiB
Dataset size:
395.47 KiB
Examples (tfds.as_dataframe):
blimp/wh_questions_subject_gap_long_distance
Config description: This configuration includes the paradigm wh_questions_subject_gap_long_distance.
Download size:
451.82 KiB
Dataset size:
466.53 KiB
Examples (tfds.as_dataframe):
blimp/wh_vs_that_no_gap
Config description: This configuration includes the paradigm wh_vs_that_no_gap.
Download size:
374.28 KiB
Dataset size:
388.91 KiB
Examples (tfds.as_dataframe):
blimp/wh_vs_that_no_gap_long_distance
Config description: This configuration includes the paradigm wh_vs_that_no_gap_long_distance.
Download size:
431.08 KiB
Dataset size:
445.74 KiB
Examples (tfds.as_dataframe):
blimp/wh_vs_that_with_gap
Config description: This configuration includes the paradigm wh_vs_that_with_gap.
Download size:
359.16 KiB
Dataset size:
373.79 KiB
Examples (tfds.as_dataframe):
blimp/wh_vs_that_with_gap_long_distance
Config description: This configuration includes the paradigm wh_vs_that_with_gap_long_distance.
Download size:
416.00 KiB
Dataset size:
430.63 KiB
Examples (tfds.as_dataframe):