|---|---|---|---|
| [View on TensorFlow.org](https://www.tensorflow.org/tutorials/load_data/tfrecord) | [Run in Google Colab](https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/load_data/tfrecord.ipynb) | [View source on GitHub](https://github.com/tensorflow/docs/blob/master/site/en/tutorials/load_data/tfrecord.ipynb) | [Download notebook](https://storage.googleapis.com/tensorflow_docs/docs/site/en/tutorials/load_data/tfrecord.ipynb) |
The TFRecord format is a simple format for storing a sequence of binary records.
[Protocol buffers](https://developers.google.com/protocol-buffers/) are a cross-platform, cross-language library for efficient serialization of structured data.
Protocol messages are defined by `.proto` files, these are often the easiest way to understand a message type.
The [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) message (or protobuf) is a flexible message type that represents a `{"string": value}` mapping. It is designed for use with TensorFlow and is used throughout the higher-level APIs such as [TFX](https://www.tensorflow.org/tfx/).
This notebook demonstrates how to create, parse, and use the [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) message, and then serialize, write, and read [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) messages to and from `.tfrecord` files.
> [!NOTE]
> **Note:** While useful, these structures are optional. There is no need to convert existing code to use TFRecords, unless you are [using tf.data](https://www.tensorflow.org/guide/data) and reading data is still the bottleneck to training. You can refer to [Better performance with the tf.data API](https://www.tensorflow.org/guide/data_performance) for dataset performance tips.
> [!NOTE]
> **Note:** In general, you should shard your data across multiple files so that you can parallelize I/O (within a single host or across multiple hosts). The rule of thumb is to have at least 10 times as many files as there will be hosts reading data. At the same time, each file should be large enough (at least 10 MB+ and ideally 100 MB+) so that you can benefit from I/O prefetching. For example, say you have `X` GB of data and you plan to train on up to `N` hosts. Ideally, you should shard the data to \~`10*N` files, as long as \~`X/(10*N)` is 10 MB+ (and ideally 100 MB+). If it is less than that, you might need to create fewer shards to trade off parallelism benefits and I/O prefetching benefits.
## Setup
import tensorflow as tf
import numpy as np
import IPython.display as display
## [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example)
### Data types for [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example)
Fundamentally, a [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) is a `{"string": tf.train.Feature}` mapping.
The [`tf.train.Feature`](https://www.tensorflow.org/api_docs/python/tf/train/Feature) message type can accept one of the following three types (See the [`.proto` file](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/example/feature.proto) for reference). Most other generic types can be coerced into one of these:
1. [`tf.train.BytesList`](https://www.tensorflow.org/api_docs/python/tf/train/BytesList) (the following types can be coerced)
- `string`
- `byte`
2. [`tf.train.FloatList`](https://www.tensorflow.org/api_docs/python/tf/train/FloatList) (the following types can be coerced)
- `float` (`float32`)
- `double` (`float64`)
3. [`tf.train.Int64List`](https://www.tensorflow.org/api_docs/python/tf/train/Int64List) (the following types can be coerced)
- `bool`
- `enum`
- `int32`
- `uint32`
- `int64`
- `uint64`
In order to convert a standard TensorFlow type to a [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example)-compatible [`tf.train.Feature`](https://www.tensorflow.org/api_docs/python/tf/train/Feature), you can use the shortcut functions below. Note that each function takes a scalar input value and returns a [`tf.train.Feature`](https://www.tensorflow.org/api_docs/python/tf/train/Feature) containing one of the three `list` types above:
# The following functions can be used to convert a value to a type compatible
# with tf.train.Example.
def _bytes_feature(value):
"""Returns a bytes_list from a string / byte."""
if isinstance(value, type(tf.constant(0))):
value = value.numpy() # BytesList won't unpack a string from an EagerTensor.
return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))
def _float_feature(value):
"""Returns a float_list from a float / double."""
return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))
def _int64_feature(value):
"""Returns an int64_list from a bool / enum / int / uint."""
if isinstance(value, type(tf.constant(0))):
value = value.numpy()
if isinstance(value, np.generic):
value = value.item()
return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))
> [!NOTE]
> **Note:** To stay simple, this example only uses scalar inputs. The simplest way to handle non-scalar features is to use [`tf.io.serialize_tensor`](https://www.tensorflow.org/api_docs/python/tf/io/serialize_tensor) to convert tensors to binary-strings. Strings are scalars in TensorFlow. Use [`tf.io.parse_tensor`](https://www.tensorflow.org/api_docs/python/tf/io/parse_tensor) to convert the binary-string back to a tensor.
Below are some examples of how these functions work. Note the varying input types and the standardized output types. If the input type for a function does not match one of the coercible types stated above, the function will raise an exception (e.g. `_int64_feature(1.0)` will error out because `1.0` is a float---therefore, it should be used with the `_float_feature` function instead):
print(_bytes_feature(b'test_string'))
print(_bytes_feature(u'test_bytes'.encode('utf-8')))
print(_float_feature(np.exp(1)))
print(_int64_feature(True))
print(_int64_feature(1))
```
bytes_list {
value: "test_string"
}
bytes_list {
value: "test_bytes"
}
float_list {
value: 2.71828175
}
int64_list {
value: 1
}
int64_list {
value: 1
}
```
All proto messages can be serialized to a binary-string using the `.SerializeToString` method:
feature = _float_feature(np.exp(1))
feature.SerializeToString()
```
b'\x12\x06\n\x04T\xf8-@'
```
### Creating a [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) message
Suppose you want to create a [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) message from existing data. In practice, the dataset may come from anywhere, but the procedure of creating the [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) message from a single observation will be the same:
1. Within each observation, each value needs to be converted to a [`tf.train.Feature`](https://www.tensorflow.org/api_docs/python/tf/train/Feature) containing one of the 3 compatible types, using one of the functions above.
2. You create a map (dictionary) from the feature name string to the encoded feature value produced in #1.
3. The map produced in step 2 is converted to a [`Features` message](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/example/feature.proto#L85).
In this notebook, you will create a dataset using NumPy.
This dataset will have 4 features:
- a boolean feature, `False` or `True` with equal probability
- an integer feature uniformly randomly chosen from `[0, 5]`
- a string feature generated from a string table by using the integer feature as an index
- a float feature from a standard normal distribution
Consider a sample consisting of 10,000 independently and identically distributed observations from each of the above distributions:
# The number of observations in the dataset.
n_observations = int(1e4)
# Boolean feature, encoded as False or True.
feature0 = np.random.choice([False, True], n_observations)
# Integer feature, random from 0 to 4.
feature1 = np.random.randint(0, 5, n_observations)
# String feature.
strings = np.array([b'cat', b'dog', b'chicken', b'horse', b'goat'])
feature2 = strings[feature1]
# Float feature, from a standard normal distribution.
feature3 = np.random.randn(n_observations)
Each of these features can be coerced into a [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example)-compatible type using one of `_bytes_feature`, `_float_feature`, `_int64_feature`. You can then create a [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) message from these encoded features:
@tf.py_function(Tout=tf.string)
def serialize_example(feature0, feature1, feature2, feature3):
"""
Creates a tf.train.Example message ready to be written to a file.
"""
# Create a dictionary mapping the feature name to the tf.train.Example-compatible
# data type.
feature = {
'feature0': _int64_feature(feature0),
'feature1': _int64_feature(feature1),
'feature2': _bytes_feature(feature2),
'feature3': _float_feature(feature3),
}
# Create a Features message using tf.train.Example.
example_proto = tf.train.Example(features=tf.train.Features(feature=feature))
return example_proto.SerializeToString()
For example, suppose you have a single observation from the dataset, `[False, 4, bytes('goat'), 0.9876]`. You can create and print the `tf.train.Example` message for this observation using `serialize_example()`. Each single observation will be written as a `Features` message as per the above. Note that the `tf.train.Example` [message](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/example/example.proto#L88) is just a wrapper around the `Features` message:
# This is an example observation from the dataset.
example_observation = [False, 4, b'goat', 0.9876]
serialized_example = serialize_example(*example_observation)
serialized_example
```
```
To decode the message use the [`tf.train.Example.FromString`](https://www.tensorflow.org/api_docs/python/tf/train/Example#FromString) method.
example_proto = tf.train.Example.FromString(serialized_example.numpy())
example_proto
```
features {
feature {
key: "feature3"
value {
float_list {
value: 0.9876
}
}
}
feature {
key: "feature2"
value {
bytes_list {
value: "goat"
}
}
}
feature {
key: "feature1"
value {
int64_list {
value: 4
}
}
}
feature {
key: "feature0"
value {
int64_list {
value: 0
}
}
}
}
```
## TFRecords format details
A TFRecord file contains a sequence of records. The file can only be read sequentially.
Each record contains a byte-string, for the data-payload, plus the data-length, and CRC-32C ([32-bit CRC](https://en.wikipedia.org/wiki/Cyclic_redundancy_check#CRC-32_algorithm) using the [Castagnoli polynomial](https://en.wikipedia.org/wiki/Cyclic_redundancy_check#Standards_and_common_use)) hashes for integrity checking.
Each record is stored in the following formats:
uint64 length
uint32 masked_crc32_of_length
byte data[length]
uint32 masked_crc32_of_data
The records are concatenated together to produce the file. CRCs are
[described here](https://en.wikipedia.org/wiki/Cyclic_redundancy_check), and
the mask of a CRC is:
masked_crc = ((crc >> 15) | (crc << 17)) + 0xa282ead8ul
> [!NOTE]
> **Note:** There is no requirement to use [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) in TFRecord files. [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) is just a method of serializing dictionaries to byte-strings. Any byte-string that can be decoded in TensorFlow could be stored in a TFRecord file. Examples include: lines of text, JSON (using [`tf.io.decode_json_example`](https://www.tensorflow.org/api_docs/python/tf/io/decode_json_example)), encoded image data, or serialized `tf.Tensors` (using [`tf.io.serialize_tensor`](https://www.tensorflow.org/api_docs/python/tf/io/serialize_tensor)/[`tf.io.parse_tensor`](https://www.tensorflow.org/api_docs/python/tf/io/parse_tensor)). See the [`tf.io`](https://www.tensorflow.org/api_docs/python/tf/io) module for more options.
## Reading and writing TFRecord files
The [`tf.io`](https://www.tensorflow.org/api_docs/python/tf/io) module also contains pure-Python functions for reading and writing TFRecord files.
### Writing a TFRecord file
Next, write the 10,000 observations to the file `test.tfrecord`. Each observation is converted to a [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) message, then written to file. You can then verify that the file `test.tfrecord` has been created:
filename = 'test.tfrecord'
# Write the `tf.train.Example` observations to the file.
with tf.io.TFRecordWriter(filename) as writer:
for i in range(n_observations):
example = serialize_example(feature0[i], feature1[i], feature2[i], feature3[i])
writer.write(example.numpy())
du -sh {filename}
```
984K test.tfrecord
```
### Reading a TFRecord file in python
These serialized tensors can be easily parsed using [`tf.train.Example.ParseFromString`](https://www.tensorflow.org/api_docs/python/tf/train/Example#ParseFromString):
filenames = [filename]
raw_dataset = tf.data.TFRecordDataset(filenames)
raw_dataset
```
```
for raw_record in raw_dataset.take(1):
example = tf.train.Example()
example.ParseFromString(raw_record.numpy())
print(example)
```
features {
feature {
key: "feature3"
value {
float_list {
value: -0.70169878
}
}
}
feature {
key: "feature2"
value {
bytes_list {
value: "chicken"
}
}
}
feature {
key: "feature1"
value {
int64_list {
value: 2
}
}
}
feature {
key: "feature0"
value {
int64_list {
value: 1
}
}
}
}
```
That returns a [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) proto which is dificult to use as is, but it's fundamentally a representation of a:
Dict[str,
Union[List[float],
List[int],
List[str]]]
The following code manually converts the `Example` to a dictionary of NumPy arrays, without using TensorFlow Ops. Refer to [the PROTO file](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/example/feature.proto) for details.
result = {}
# example.features.feature is the dictionary
for key, feature in example.features.feature.items():
# The values are the Feature objects which contain a `kind` which contains:
# one of three fields: bytes_list, float_list, int64_list
kind = feature.WhichOneof('kind')
result[key] = np.array(getattr(feature, kind).value)
result
```
{'feature2': array([b'chicken'], dtype='|S7'),
'feature0': array([1]),
'feature1': array([2]),
'feature3': array([-0.70169878])}
```
### Reading a TFRecord file Using tf.data
You can also read the TFRecord file using the [`tf.data.TFRecordDataset`](https://www.tensorflow.org/api_docs/python/tf/data/TFRecordDataset) class.
More information on consuming TFRecord files using [`tf.data`](https://www.tensorflow.org/api_docs/python/tf/data) can be found in the [tf.data: Build TensorFlow input pipelines](https://www.tensorflow.org/guide/data#consuming_tfrecord_data) guide.
Using `TFRecordDataset`s can be useful for standardizing input data and optimizing performance.
filenames = [filename]
raw_dataset = tf.data.TFRecordDataset(filenames)
raw_dataset
```
```
At this point the dataset contains serialized [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) messages. When iterated over it returns these as scalar string tensors.
Use the `.take` method to only show the first 10 records.
> [!NOTE]
> **Note:** iterating over a [`tf.data.Dataset`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) only works with eager execution enabled.
for raw_record in raw_dataset.take(10):
print(repr(raw_record))
```
\n\x11\n\x08feature1\x12\x05\x1a\x03\n\x01\x03\n\x11\n\x08feature0\x12\x05\x1a\x03\n\x01\x00\n\x15\n\x08feature2\x12\t\n\x07\n\x05horse'>
\n\x11\n\x08feature1\x12\x05\x1a\x03\n\x01\x03\n\x11\n\x08feature0\x12\x05\x1a\x03\n\x01\x00\n\x15\n\x08feature2\x12\t\n\x07\n\x05horse'>
```
These tensors can be parsed using the function below. Note that the `feature_description` is necessary here because [`tf.data.Dataset`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset)s use graph-execution, and need this description to build their shape and type signature:
# Create a description of the features.
feature_description = {
'feature0': tf.io.FixedLenFeature([], tf.int64, default_value=0),
'feature1': tf.io.FixedLenFeature([], tf.int64, default_value=0),
'feature2': tf.io.FixedLenFeature([], tf.string, default_value=''),
'feature3': tf.io.FixedLenFeature([], tf.float32, default_value=0.0),
}
def _parse_function(example_proto):
# Parse the input `tf.train.Example` proto using the dictionary above.
return tf.io.parse_single_example(example_proto, feature_description)
Alternatively, use `tf.parse_example` to parse the whole batch at once. Apply this function to each item in the dataset using the [`tf.data.Dataset.map`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map) method:
parsed_dataset = raw_dataset.map(_parse_function)
parsed_dataset
```
<_MapDataset element_spec={'feature0': TensorSpec(shape=(), dtype=tf.int64, name=None), 'feature1': TensorSpec(shape=(), dtype=tf.int64, name=None), 'feature2': TensorSpec(shape=(), dtype=tf.string, name=None), 'feature3': TensorSpec(shape=(), dtype=tf.float32, name=None)}>
```
Use eager execution to display the observations in the dataset. There are 10,000 observations in this dataset, but you will only display the first 10. The data is displayed as a dictionary of features. Each item is a [`tf.Tensor`](https://www.tensorflow.org/api_docs/python/tf/Tensor), and the `numpy` element of this tensor displays the value of the feature:
for parsed_record in parsed_dataset.take(10):
print(repr(parsed_record))
```
{'feature0': , 'feature1': , 'feature2': , 'feature3': }
{'feature0': , 'feature1': , 'feature2': , 'feature3': }
{'feature0': , 'feature1': , 'feature2': , 'feature3': }
{'feature0': , 'feature1': , 'feature2': , 'feature3': }
{'feature0': , 'feature1': , 'feature2': , 'feature3': }
{'feature0': , 'feature1': , 'feature2': , 'feature3': }
{'feature0': , 'feature1': , 'feature2': , 'feature3': }
{'feature0': , 'feature1': , 'feature2': , 'feature3': }
{'feature0': , 'feature1': , 'feature2': , 'feature3': }
{'feature0': , 'feature1': , 'feature2': , 'feature3': }
```
Here, the `tf.parse_example` function unpacks the [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) fields into standard tensors.
## Walkthrough: Reading and writing image data
This is an end-to-end example of how to read and write image data using TFRecords. Using an image as input data, you will write the data as a TFRecord file, then read the file back and display the image.
This can be useful if, for example, you want to use several models on the same input dataset. Instead of storing the image data raw, it can be preprocessed into the TFRecords format, and that can be used in all further processing and modelling.
First, let's download [this image](https://commons.wikimedia.org/wiki/File:Felis_catus-cat_on_snow.jpg) of a cat in the snow and [this photo](https://upload.wikimedia.org/wikipedia/commons/f/fe/New_East_River_Bridge_from_Brooklyn_det.4a09796u.jpg) of the Williamsburg Bridge, NYC under construction.
### Fetch the images
cat_in_snow = tf.keras.utils.get_file(
'320px-Felis_catus-cat_on_snow.jpg',
'https://storage.googleapis.com/download.tensorflow.org/example_images/320px-Felis_catus-cat_on_snow.jpg')
williamsburg_bridge = tf.keras.utils.get_file(
'194px-New_East_River_Bridge_from_Brooklyn_det.4a09796u.jpg',
'https://storage.googleapis.com/download.tensorflow.org/example_images/194px-New_East_River_Bridge_from_Brooklyn_det.4a09796u.jpg')
```
Downloading data from https://storage.googleapis.com/download.tensorflow.org/example_images/320px-Felis_catus-cat_on_snow.jpg
17858/17858 [==============================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/download.tensorflow.org/example_images/194px-New_East_River_Bridge_from_Brooklyn_det.4a09796u.jpg
15477/15477 [==============================] - 0s 0us/step
```
display.display(display.Image(filename=cat_in_snow))
display.display(display.HTML('Image cc-by: Von.grzanka'))
display.display(display.Image(filename=williamsburg_bridge))
display.display(display.HTML('From Wikimedia'))
### Write the TFRecord file
As before, encode the features as types compatible with [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example). This stores the raw image string feature, as well as the height, width, depth, and arbitrary `label` feature. The latter is used when you write the file to distinguish between the cat image and the bridge image. Use `0` for the cat image, and `1` for the bridge image:
image_labels = {
cat_in_snow : 0,
williamsburg_bridge : 1,
}
# This is an example, just using the cat image.
image_string = open(cat_in_snow, 'rb').read()
label = image_labels[cat_in_snow]
# Create a dictionary with features that may be relevant.
def image_example(image_string, label):
image_shape = tf.io.decode_jpeg(image_string).shape
feature = {
'height': _int64_feature(image_shape[0]),
'width': _int64_feature(image_shape[1]),
'depth': _int64_feature(image_shape[2]),
'label': _int64_feature(label),
'image_raw': _bytes_feature(image_string),
}
return tf.train.Example(features=tf.train.Features(feature=feature))
for line in str(image_example(image_string, label)).split('\n')[:15]:
print(line)
print('...')
```
features {
feature {
key: "width"
value {
int64_list {
value: 320
}
}
}
feature {
key: "label"
value {
int64_list {
value: 0
}
...
```
Notice that all of the features are now stored in the [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) message. Next, functionalize the code above and write the example messages to a file named `images.tfrecords`:
# Write the raw image files to `images.tfrecords`.
# First, process the two images into `tf.train.Example` messages.
# Then, write to a `.tfrecords` file.
record_file = 'images.tfrecords'
with tf.io.TFRecordWriter(record_file) as writer:
for filename, label in image_labels.items():
image_string = open(filename, 'rb').read()
tf_example = image_example(image_string, label)
writer.write(tf_example.SerializeToString())
du -sh {record_file}
```
36K images.tfrecords
```
### Read the TFRecord file
You now have the file---`images.tfrecords`---and can now iterate over the records in it to read back what you wrote. Given that in this example you will only reproduce the image, the only feature you will need is the raw image string. Extract it using the getters described above, namely `example.features.feature['image_raw'].bytes_list.value[0]`. You can also use the labels to determine which record is the cat and which one is the bridge:
raw_image_dataset = tf.data.TFRecordDataset('images.tfrecords')
# Create a dictionary describing the features.
image_feature_description = {
'height': tf.io.FixedLenFeature([], tf.int64),
'width': tf.io.FixedLenFeature([], tf.int64),
'depth': tf.io.FixedLenFeature([], tf.int64),
'label': tf.io.FixedLenFeature([], tf.int64),
'image_raw': tf.io.FixedLenFeature([], tf.string),
}
def _parse_image_function(example_proto):
# Parse the input tf.train.Example proto using the dictionary above.
return tf.io.parse_single_example(example_proto, image_feature_description)
parsed_image_dataset = raw_image_dataset.map(_parse_image_function)
parsed_image_dataset
```
<_MapDataset element_spec={'depth': TensorSpec(shape=(), dtype=tf.int64, name=None), 'height': TensorSpec(shape=(), dtype=tf.int64, name=None), 'image_raw': TensorSpec(shape=(), dtype=tf.string, name=None), 'label': TensorSpec(shape=(), dtype=tf.int64, name=None), 'width': TensorSpec(shape=(), dtype=tf.int64, name=None)}>
```
Recover the images from the TFRecord file:
for image_features in parsed_image_dataset:
image_raw = image_features['image_raw'].numpy()
display.display(display.Image(data=image_raw))
