tfds.file_adapter.FileFormatAdapter

View source on GitHub

Class FileFormatAdapter

Provides writing and reading methods for a file format.

__init__

View source

__init__(example_specs)

Constructor.

Args:

Properties

filetype_suffix

Returns a str file type suffix (e.g. "tfrecord").

Methods

dataset_from_filename

View source

dataset_from_filename(filename)

Returns a tf.data.Dataset whose elements are dicts given a filename.

write_from_generator

View source

write_from_generator(
    generator,
    output_files
)

Write to files from generators_and_filenames.

Args:

  • generator: generator yielding dictionaries of feature name to value.
  • output_files: list<str>, output files to write files to.

write_from_pcollection

View source

write_from_pcollection(
    pcollection,
    file_path_prefix=None,
    num_shards=None
)

Write the PCollection to file.

Args:

  • pcollection: beam.PCollection, the PCollection containing the examples to write.
  • file_path_prefix: str, output files to write files to.
  • num_shards: int,