|---|---|---|---|
| [View on TensorFlow.org](https://www.tensorflow.org/io/tutorials/genome) | [Run in Google Colab](https://colab.research.google.com/github/tensorflow/io/blob/master/docs/tutorials/genome.ipynb) | [View source on GitHub](https://github.com/tensorflow/io/blob/master/docs/tutorials/genome.ipynb) | [Download notebook](https://storage.googleapis.com/tensorflow_docs/io/docs/tutorials/genome.ipynb) |
## Overview
This tutorial demonstrates the [`tfio.genome`](https://www.tensorflow.org/io/api_docs/python/tfio/genome) package that provides commonly used genomics IO functionality--namely reading several genomics file formats and also providing some common operations for preparing the data (for example--one hot encoding or parsing Phred quality into probabilities).
This package uses the [Google Nucleus](https://github.com/google/nucleus) library to provide some of the core functionality.
## Setup
try:
%tensorflow_version 2.x
except Exception:
pass
!pip install -q tensorflow-io
import tensorflow_io as tfio
import tensorflow as tf
## FASTQ Data
FASTQ is a common genomics file format that stores both sequence information in addition to base quality information.
First, let's download a sample `fastq` file.
# Download some sample data:
curl -OL https://raw.githubusercontent.com/tensorflow/io/master/tests/test_genome/test.fastq
```
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 407 100 407 0 0 2035 0 --:--:-- --:--:-- --:--:-- 2035
```
### Read FASTQ Data
Now, let's use [`tfio.genome.read_fastq`](https://www.tensorflow.org/io/api_docs/python/tfio/genome/read_fastq) to read this file (note a [`tf.data`](https://www.tensorflow.org/api_docs/python/tf/data) API coming soon).
fastq_data = tfio.genome.read_fastq(filename="test.fastq")
print(fastq_data.sequences)
print(fastq_data.raw_quality)
```
tf.Tensor(
[b'GATTACA'
b'CGTTAGCGCAGGGGGCATCTTCACACTGGTGACAGGTAACCGCCGTAGTAAAGGTTCCGCCTTTCACT'
b'CGGCTGGTCAGGCTGACATCGCCGCCGGCCTGCAGCGAGCCGCTGC' b'CGG'], shape=(4,), dtype=string)
tf.Tensor(
[b'BB>B@FA'
b'AAAAABF@BBBDGGGG?FFGFGHBFBFBFABBBHGGGFHHCEFGGGGG?FGFFHEDG3EFGGGHEGHG'
b'FAFAF;F/9;.:/;999B/9A.DFFF;-->.AAB/FC;9-@-=;=.' b'FAD'], shape=(4,), dtype=string)
```
As you see, the returned `fastq_data` has `fastq_data.sequences` which is a string tensor of all sequences in the fastq file (which can each be a different size) along with `fastq_data.raw_quality` which includes Phred encoded quality information about the quality of each base read in the sequence.
### Quality
You can use a helper op to convert this quality information into probabilities if you are interested.
quality = tfio.genome.phred_sequences_to_probability(fastq_data.raw_quality)
print(quality.shape)
print(quality.row_lengths().numpy())
print(quality)
```
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py:574: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
(4, None, 1)
[ 7 68 46 3]
```
### One hot encodings
You may also want to encode the genome sequence data (which consists of `A` `T` `C` `G` bases) using a one hot encoder. There's a built in operation that can help with this.
one_hot = tfio.genome.sequences_to_onehot(fastq_data.sequences)
print(one_hot)
print(one_hot.shape)
```
(4, None, 4)
```
print(tfio.genome.sequences_to_onehot.__doc__)
```
Convert DNA sequences into a one hot nucleotide encoding.
Each nucleotide in each sequence is mapped as follows:
A -> [1, 0, 0, 0]
C -> [0, 1, 0, 0]
G -> [0 ,0 ,1, 0]
T -> [0, 0, 0, 1]
If for some reason a non (A, T, C, G) character exists in the string, it is
currently mapped to a error one hot encoding [1, 1, 1, 1].
Args:
sequences: A tf.string tensor where each string represents a DNA sequence
Returns:
tf.RaggedTensor: The output sequences with nucleotides one hot encoded.
```