tf.io.gfile.GFile

File I/O wrappers without thread locking.

Used in the notebooks

Used in the tutorials

The main roles of the tf.io.gfile module are:

  1. To provide an API that is close to Python's file I/O objects, and
  2. To provide an implementation based on TensorFlow's C++ FileSystem API.

The C++ FileSystem API supports multiple file system implementations, including local files, Google Cloud Storage (using a gs:// prefix, and HDFS (using an hdfs:// prefix). TensorFlow exports these as tf.io.gfile, so that you can use these implementations for saving and loading checkpoints, writing to TensorBoard logs, and accessing training data (among other uses). However, if all your files are local, you can use the regular Python file API without any problem.

Once you obtain a GFile object, you can use it in most ways as you would any Python's file object:

with open("/tmp/x", "w") as f:
  f.write("asdf")
4
with tf.io.gfile.GFile("/tmp/x") as f:
  f.read()
'asdf'

The difference is that you can specify URI schemes to use other filesystems (e.g., gs:// for GCS, s3:// for S3, etc.), if they are supported. Using file:// as an example, we have:

with tf.io.gfile.GFile("file:///tmp/x", "w") as f:
  f.write("qwert")
  f.write("asdf")
tf.io.gfile.GFile("file:///tmp/x").read()
'qwertasdf'

You can also read all lines of a file directly:

with tf.io.gfile.GFile("file:///tmp/x", "w") as f:
  f.write("asdf\n")
  f.write("qwer\n")
tf.io.gfile.GFile("/tmp/x").readlines()
['asdf\n', 'qwer\n']

You can iterate over the lines:

with tf.io.gfile.GFile("file:///tmp/x", "w") as f:
  f.write("asdf\n")
  f.write("qwer\n")
for line in tf.io.gfile.GFile("/tmp/x"):
  print(line[:-1]) # removes the end of line character
asdf
qwer

Random access read is possible if the underlying filesystem supports it:

with open("/tmp/x", "w") as f:
  f.write("asdfqwer")
f = tf.io.gfile.GFile("/tmp/x")
f.read(3)
'asd'
f.seek(4)
f.tell()
4
f.read(3)
'qwe'
f.tell()
7
f.close()

mode Returns the mode in which the file was opened.
name Returns the file name.

Methods

close

View source

Closes the file.

Should be called for the WritableFile to be flushed.

In general, if you use the context manager pattern, you don't need to call this directly.

with tf.io.gfile.GFile("/tmp/x", "w") as f:
  f.write("asdf\n")
  f.write("qwer\n")
# implicit f.close() at the end of the block

For cloud filesystems, forgetting to call close() might result in data loss as last write might not have been replicated.

flush

View source

Flushes the Writable file.

This only ensures that the data has made its way out of the process without any guarantees on whether it's written to disk. This means that the data would survive an application crash but not necessarily an OS crash.

next

View source

read

View source

Returns the contents of a file as a string.

Starts reading from current position in file.

Args
n Read n bytes if n != -1. If n = -1, reads to end of file.

Returns
n bytes of the file (or whole file) in bytes mode or n bytes of the string if in string (regular) mode.

readline