siscore

SI-Score (Synthetic Interventions on Scenes for Robustness Evaluation) is a dataset to evaluate robustness of image classification models to changes in object size, location and rotation angle.

In SI-SCORE, we take objects and backgrounds and systematically vary object size, location and rotation angle so we can study the effect of changing these factors on model performance. The image label space is the ImageNet label space (1k classes) for easy evaluation of models.

More information about the dataset can be found at https://github.com/google-research/si-score

FeaturesDict({
    'dataset_label': ClassLabel(shape=(), dtype=int64, num_classes=1000),
    'image': Image(shape=(None, None, 3), dtype=uint8),
    'image_id': int64,
    'label': ClassLabel(shape=(), dtype=int64, num_classes=1000),
})
  • Feature documentation:
Feature Class Shape Dtype Description
FeaturesDict
dataset_label ClassLabel int64
image Image (None, None, 3) uint8
image_id Tensor int64
label ClassLabel int64
@misc{djolonga2020robustness,
      title={On Robustness and Transferability of Convolutional Neural Networks},
      author={Josip Djolonga and Jessica Yung and Michael Tschannen and Rob Romijnders and Lucas Beyer and Alexander Kolesnikov and Joan Puigcerver and Matthias Minderer and Alexander D'Amour and Dan Moldovan and Sylvain Gelly and Neil Houlsby and Xiaohua Zhai and Mario Lucic},
      year={2020},
      eprint={2007.08558},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

siscore/rotation (default config)

  • Config description: factor of variation: rotation

  • Download size: 1.40 GiB

  • Dataset size: 1.40 GiB

  • Splits:

Split Examples
'test' 39,540

Visualization

siscore/size

  • Config description: factor of variation: size

  • Download size: 3.25 GiB

  • Dataset size: 3.27 GiB

  • Splits:

Split Examples
'test' 92,884

Visualization

siscore/location

  • Config description: factor of variation: location

  • Download size: 18.21 GiB

  • Dataset size: 18.31 GiB

  • Splits:

Split Examples
'test' 541,548

Visualization