Google I/O is a wrap! Catch up on TensorFlow sessions View sessions

tflite_model_maker.object_detector.DataLoader

DataLoader for object detector.

tflite_model_maker.object_detector.DataLoader(
    tfrecord_file_patten, size, label_map, annotations_json_file=None
)

Used in the notebooks

Used in the tutorials

tfrecord_file_patten Glob for tfrecord files. e.g. "/tmp/coco*.tfrecord".
size The size of the dataset.
label_map Variable shows mapping label integers ids to string label names. 0 is the reserved key for background and doesn't need to be included in label_map. Label names can't be duplicated. Supported formats are:

  1. Dict, map label integers ids to string label names, such as {1: 'person', 2: 'notperson'}. 2. List, a list of label names such as ['person', 'notperson'] which is the same as setting label_map={1: 'person', 2: 'notperson'}.
  2. String, name for certain dataset. Accepted values are: 'coco', 'voc' and 'waymo'. 4. String, yaml filename that stores label_map.
annotations_json_file JSON with COCO data format containing golden bounding boxes. Used for validation. If None, use the ground truth from the dataloader. Refer to https://towardsdatascience.com/coco-data-format-for-object-detection-a4c5eaf518c5 for the description of COCO data format.

size Returns the size of the dataset.

Note that this function may return None becuase the exact size of the dataset isn't a necessary parameter to create an instance of this class, and tf.data.Dataset donesn't support a function to get the length directly since it's lazy-loaded and may be infinite. In most cases, however, when an instance of this class is created by helper functions like 'from_folder', the size of the dataset will be preprocessed, and this function can return an int representing the size of the dataset.

Methods

from_cache

View source

@classmethod
from_cache(
    cache_prefix
)

Loads the data from cache.

Args
cache_prefix The cache prefix including the cache directory and the cache prefix filename, e.g: '/tmp/cache/train'.

Returns
ObjectDetectorDataLoader object.

from_csv

View source

@classmethod
from_csv(
    filename: str,
    images_dir: Optional[str] = None,
    delimiter: str = ',',
    quotechar: str = '\'",
    num_shards: int = 10,
    max_num_images: Optional[int] = None,
    cache_dir: Optional[str] = None,
    cache_prefix_filename: Optional[str] = None
) -> List[Optional[DetectorDataLoader]]

Loads the data from the csv file.

The csv format is shown in https://cloud.google.com/vision/automl/object-detection/docs/csv-format We supports bounding box with 2 vertices for now. We support the files in the local machine as well.

Args
filename Name of the csv file.
images_dir Path to directory that store raw images. If None, the image path in the csv file is the path to Google Cloud Storage or the absolute path in the local machine.
delimiter Character used to separate fields.
quotechar Character used to quote fields containing special characters.
num_shards Number of shards for output file.
max_num_images Max number of imags to process.
cache_dir The cache directory to save TFRecord, metadata and json file. When cache_dir is None, a temporary folder will be created and will not be removed automatically after training which makes it can be used later.
cache_prefix_filename The cache prefix filename. If None, will automatically generate it based on filename.

Returns
train_data, validation_data, test_data which are ObjectDetectorDataLoader objects. Can be None if without such data.

from_pascal_voc

View source

@classmethod
from_pascal_voc(
    images_dir: str,
    annotations_dir: str,
    label_map: Union[List[str], Dict[int, str], str],
    annotation_filenames: Optional[Collection[str]] = None,
    ignore_difficult_instances: bool = False,
    num_shards: int = 100,
    max_num_images: Optional[int] = None,
    cache_dir: Optional[str] = None,
    cache_prefix_filename: Optional[str] = None
) -> DetectorDataLoader

Loads from dataset with PASCAL VOC format.

Refer to https://towardsdatascience.com/coco-data-format-for-object-detection-a4c5eaf518c5 for the description of PASCAL VOC data format.

LabelImg Tool (https://github.com/tzutalin/labelImg) can annotate the image and save annotations as XML files in PASCAL VOC data format.

Annotations are in the folder: annotations_dir. Raw images are in the foloder: images_dir.

Args
images_dir Path to directory that store raw images.
annotations_dir Path to the annotations directory.
label_map Variable shows mapping label integers ids to string label names. 0 is the reserved key for background. Label names can't be duplicated. Supported format: 1. Dict, map label integers ids to string label names, e.g. {1: 'person', 2: 'notperson'}. 2. List, a list of label names. e.g. ['person', 'notperson'] which is the same as setting label_map={1: 'person', 2: 'notperson'}.

  1. String, name for certain dataset. Accepted values are: 'coco', 'voc' and 'waymo'. 4. String, yaml filename that stores label_map.
annotation_filenames Collection of annotation filenames (strings) to be loaded. For instance, if there're 3 annotation files [0.xml, 1.xml, 2.xml] in annotations_dir, setting annotation_filenames=['0', '1'] makes this method only load [0.xml, 1.xml].
ignore_difficult_instances Whether to ignore difficult instances. difficult can be set inside object item in the annotation xml file.
num_shards Number of shards for output file.
max_num_images Max number of imags to process.
cache_dir The cache directory to save TFRecord, metadata and json file. When cache_dir is not set, a temporary folder will be created and will not be removed automatically after training which makes it can be used later.
cache_prefix_filename The cache prefix filename. If not set, will automatically generate it based on image_dir, annotations_dir and annotation_filenames.

Returns
ObjectDetectorDataLoader object.

gen_dataset

View source

gen_dataset(
    model_spec, batch_size=None, is_training=False, use_fake_data=False
)

Generate a batched tf.data.Dataset for training/evaluation.

Args
model_spec Specification for the model.
batch_size A integer, the returned dataset will be batched by this size.
is_training A boolean, when True, the returned dataset will be optionally shuffled and repeated as an endless dataset.
use_fake_data Use fake input.

Returns
A TF dataset ready to be consumed by Keras model.

split

View source

split(
    fraction
)

This function isn't implemented for the object detection task.

__len__

View source

__len__()