tfds.file_adapter.TFRecordExampleAdapter

View source on GitHub

Class TFRecordExampleAdapter

Writes/Reads serialized Examples protos to/from TFRecord files.

Inherits From: FileFormatAdapter

Constraints on generators:

  • The generator must yield feature dictionaries (dict<str feature_name, feature_value>).
  • The allowed feature types are int, float, and str (or bytes in Python 3; unicode strings will be encoded in utf-8), or lists thereof.

__init__

View source

__init__(example_specs)

Constructor.

Args:

Properties

filetype_suffix

Returns a str file type suffix (e.g. "tfrecord").

Methods

dataset_from_filename

View source

dataset_from_filename(filename)

Returns a tf.data.Dataset whose elements are dicts given a filename.

write_from_generator

View source

write_from_generator(
    generator,
    output_files
)

Write to files from generators_and_filenames.

Args:

  • generator: generator yielding dictionaries of feature name to value.
  • output_files: list<str>, output files to write files to.

write_from_pcollection

View source

write_from_pcollection(
    pcollection,
    file_path_prefix,
    num_shards
)

Write the PCollection to file.

Args:

  • pcollection: beam.PCollection, the PCollection containing the examples to write.
  • file_path_prefix: str, output files to write files to.
  • num_shards: int,