Have a question? Connect with the community at the TensorFlow Forum Visit Forum

Fine tuning models for plant disease detection

View on TensorFlow.org Run in Google Colab View on GitHub Download notebook See TF Hub models

This notebook shows you how to fine-tune CropNet models from TensorFlow Hub on a dataset from TFDS or your own crop disease detection dataset.

You will:

  • Load the TFDS cassava dataset or your own data
  • Enrich the data with unknown (negative) examples to get a more robust model
  • Apply image augmentations to the data
  • Load and fine tune a CropNet model from TF Hub
  • Export a TFLite model, ready to be deployed on your app with Task Library, MLKit or TFLite directly

Imports and Dependencies

Before starting, you'll need to install some of the dependencies that will be needed like Model Maker and the latest version of TensorFlow Datasets.

pip install -q tflite-model-maker
pip install -q -U tensorflow-datasets
import matplotlib.pyplot as plt
import os
import seaborn as sns

import tensorflow as tf
import tensorflow_datasets as tfds

from tensorflow_examples.lite.model_maker.core.export_format import ExportFormat
from tensorflow_examples.lite.model_maker.core.task import image_preprocessing

from tflite_model_maker import image_classifier
from tflite_model_maker import ImageClassifierDataLoader
from tflite_model_maker.image_classifier import ModelSpec
/tmpfs/src/tf_docs_env/lib/python3.7/site-packages/numba/core/errors.py:154: UserWarning: Insufficiently recent colorama version found. Numba requires colorama >= 0.3.9

Load a TFDS dataset to fine-tune on

Lets use the publicly available Cassava Leaf Disease dataset from TFDS.

tfds_name = 'cassava'
(ds_train, ds_validation, ds_test), ds_info = tfds.load(
    split=['train', 'validation', 'test'],

Or alternatively load your own data to fine-tune on

Instead of using a TFDS dataset, you can also train on your own data. This code snippet shows how to load your own custom dataset. See this link for the supported structure of the data. An example is provided here using the publicly available Cassava Leaf Disease dataset.

# data_root_dir = tf.keras.utils.get_file(
#     'cassavaleafdata.zip',
#     'https://storage.googleapis.com/emcassavadata/cassavaleafdata.zip',
#     extract=True)
# data_root_dir = os.path.splitext(data_root_dir)[0]  # Remove the .zip extension

# builder = tfds.ImageFolder(data_root_dir)

# ds_info = builder.info
# ds_train = builder.as_dataset(split='train', as_supervised=True)
# ds_validation = builder.as_dataset(split='validation', as_supervised=True)
# ds_test = builder.as_dataset(split='test', as_supervised=True)

Visualize samples from train split

Let's take a look at some examples from the dataset including the class id and the class name for the image samples and their labels.

_ = tfds.show_examples(ds_train, ds_info)


Add images to be used as Unknown examples from TFDS datasets

Add additional unknown (negative) examples to the training dataset and assign a new unknown class label number to them. The goal is to have a model that, when used in practice (e.g. in the field), has the option of predicting "Unknown" when it sees something unexpected.

Below you can see a list of datasets that will be used to sample the additional unknown imagery. It includes 3 completely different datasets to increase diversity. One of them is a beans leaf disease dataset, so that the model has exposure to diseased plants other than cassava.

    'tfds_name': 'imagenet_v2/matched-frequency',
    'train_split': 'test[:80%]',
    'test_split': 'test[80%:]',
    'num_examples_ratio_to_normal': 1.0,
}, {
    'tfds_name': 'oxford_flowers102',
    'train_split': 'train',
    'test_split': 'test',
    'num_examples_ratio_to_normal': 1.0,
}, {
    'tfds_name': 'beans',
    'train_split': 'train',
    'test_split': 'test',
    'num_examples_ratio_to_normal': 1.0,

The UNKNOWN datasets are also loaded from TFDS.

# Load unknown datasets.
weights = [
    spec['num_examples_ratio_to_normal'] for spec in UNKNOWN_TFDS_DATASETS
num_unknown_train_examples = sum(
    int(w * ds_train.cardinality().numpy()) for w in weights)
ds_unknown_train = tf.data.experimental.sample_from_datasets([
        name=spec['tfds_name'], split=spec['train_split'],
        as_supervised=True).repeat(-1) for spec in UNKNOWN_TFDS_DATASETS
], weights).take(num_unknown_train_examples)
ds_unknown_train = ds_unknown_train.apply(
ds_unknown_tests = [
        name=spec['tfds_name'], split=spec['test_split'], as_supervised=True)
ds_unknown_test = ds_unknown_tests[0]
for ds in ds_unknown_tests[1:]:
  ds_unknown_test = ds_unknown_test.concatenate(ds)

# All examples from the unknown datasets will get a new class label number.
num_normal_classes = len(ds_info.features['label'].names)
unknown_label_value = tf.convert_to_tensor(num_normal_classes, tf.int64)
ds_unknown_train = ds_unknown_train.map(lambda image, _:
                                        (image, unknown_label_value))
ds_unknown_test = ds_unknown_test.map(lambda image, _:
                                      (image, unknown_label_value))

# Merge the normal train dataset with the unknown train dataset.
weights = [
ds_train_with_unknown = tf.data.experimental.sample_from_datasets(
    [ds_train, ds_unknown_train], [float(w) for w in weights])
ds_train_with_unknown = ds_train_with_unknown.apply(

print((f"Added {ds_unknown_train.cardinality().numpy()} negative examples."
       f"Training dataset has now {ds_train_with_unknown.cardinality().numpy()}"
       ' examples in total.'))
Added 16968 negative examples.Training dataset has now 22624 examples in total.

Apply augmentations

For all the images, to make them more diverse, you'll apply some augmentation, like changes in:

  • Brightness
  • Contrast
  • Saturation
  • Hue
  • Crop

These types of augmentations help make the model more robust to variations in image inputs.

def random_crop_and_random_augmentations_fn(image):
  # preprocess_for_train does random crop and resize internally.
  image = image_preprocessing.preprocess_for_train(image)
  image = tf.image.random_brightness(image, 0.2)
  image = tf.image.random_contrast(image, 0.5, 2.0)
  image = tf.image.random_saturation(image, 0.75, 1.25)
  image = tf.image.random_hue(image, 0.1)
  return image

def random_crop_fn(image):
  # preprocess_for_train does random crop and resize internally.
  image = image_preprocessing.preprocess_for_train(image)
  return image

def resize_and_center_crop_fn(image):
  image = tf.image.resize(image, (256, 256))
  image = image[16:240, 16:240]
  return image

no_augment_fn = lambda image: image

train_augment_fn = lambda image, label: (
    random_crop_and_random_augmentations_fn(image), label)
eval_augment_fn = lambda image, label: (resize_and_center_crop_fn(image), label)

To apply the augmentation, it uses the map method from the Dataset class.

ds_train_with_unknown = ds_train_with_unknown.map(train_augment_fn)
ds_validation = ds_validation.map(eval_augment_fn)
ds_test = ds_test.map(eval_augment_fn)
ds_unknown_test = ds_unknown_test.map(eval_augment_fn)
INFO:tensorflow:Use default resize_bicubic.
INFO:tensorflow:Use default resize_bicubic.
INFO:tensorflow:Use customized resize method bilinear
INFO:tensorflow:Use customized resize method bilinear

Wrap the data into Model Maker friendly format

To use these dataset with Model Maker, they need to be in a ImageClassifierDataLoader class.

label_names = ds_info.features['label'].names + ['UNKNOWN']

train_data = ImageClassifierDataLoader(ds_train_with_unknown,
validation_data = ImageClassifierDataLoader(ds_validation,
test_data = ImageClassifierDataLoader(ds_test, ds_test.cardinality(),
unknown_test_data = ImageClassifierDataLoader(ds_unknown_test,

Run training

TensorFlow Hub has multiple models available for Tranfer Learning.

Here you can choose one and you can also keep experimenting with other ones to try to get better results.

If you want even more models to try, you can add them from this collection.

Choose a base model

To fine tune the model, you will use Model Maker. This makes the overall solution easier since after the training of the model, it'll also convert it to TFLite.

Model Maker makes this conversion be the best one possible and with all the necessary information to easily deploy the model on-device later.

The model spec is how you tell Model Maker which base model you'd like to use.

image_model_spec = ModelSpec(uri=model_handle)

One important detail here is setting train_whole_model which will make the base model fine tuned during training. This makes the process slower but the final model has a higher accuracy. Setting shuffle will make sure the model sees the data in a random shuffled order which is a best practice for model learning.

model = image_classifier.create(
INFO:tensorflow:Retraining the models...
INFO:tensorflow:Retraining the models...
WARNING:tensorflow:Please add `keras.layers.InputLayer` instead of `keras.Input` to Sequential model. `keras.Input` is intended to be used by Functional model.
WARNING:tensorflow:Please add `keras.layers.InputLayer` instead of `keras.Input` to Sequential model. `keras.Input` is intended to be used by Functional model.
Model: "sequential"
Layer (type)                 Output Shape              Param #   
hub_keras_layer_v1v2 (HubKer (None, 1280)              4226432   
dropout (Dropout)            (None, 1280)              0         
dense (Dense)                (None, 6)                 7686      
Total params: 4,234,118
Trainable params: 4,209,718
Non-trainable params: 24,400
/tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:375: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  "The `lr` argument is deprecated, use `learning_rate` instead.")
Epoch 1/5
176/176 [==============================] - 102s 432ms/step - loss: 0.8815 - accuracy: 0.9181 - val_loss: 1.1452 - val_accuracy: 0.7924
Epoch 2/5
176/176 [==============================] - 74s 422ms/step - loss: 0.7931 - accuracy: 0.9511 - val_loss: 1.1023 - val_accuracy: 0.8164
Epoch 3/5
176/176 [==============================] - 74s 419ms/step - loss: 0.7727 - accuracy: 0.9589 - val_loss: 1.0326 - val_accuracy: 0.8410
Epoch 4/5
176/176 [==============================] - 69s 393ms/step - loss: 0.7612 - accuracy: 0.9635 - val_loss: 0.9958 - val_accuracy: 0.8549
Epoch 5/5
176/176 [==============================] - 69s 392ms/step - loss: 0.7532 - accuracy: 0.9658 - val_loss: 1.0252 - val_accuracy: 0.8482

Evaluate model on test split

59/59 [==============================] - 4s 48ms/step - loss: 1.0142 - accuracy: 0.8541
[1.0142441987991333, 0.8541114330291748]

To have an even better understanding of the fine tuned model, it's good to analyse the confusion matrix. This will show how often one class is predicted as another.

def predict_class_label_number(dataset):
  """Runs inference and returns predictions as class label numbers."""
  rev_label_names = {l: i for i, l in enumerate(label_names)}
  return [
      for o in model.predict_top_k(dataset, batch_size=128)

def show_confusion_matrix(cm, labels):
  plt.figure(figsize=(10, 8))
  sns.heatmap(cm, xticklabels=labels, yticklabels=labels, 
              annot=True, fmt='g')
confusion_mtx = tf.math.confusion_matrix(
    list(ds_test.map(lambda x, y: y)),

show_confusion_matrix(confusion_mtx, label_names)


Evaluate model on unknown test data

In this evaluation we expect the model to have accuracy of almost 1. All images the model is tested on are not related to the normal dataset and hence we expect the model to predict the "Unknown" class label.

259/259 [==============================] - 18s 62ms/step - loss: 0.6782 - accuracy: 0.9996
[0.6781710982322693, 0.9996375441551208]

Print the confusion matrix.

unknown_confusion_mtx = tf.math.confusion_matrix(
    list(ds_unknown_test.map(lambda x, y: y)),

show_confusion_matrix(unknown_confusion_mtx, label_names)


Export the model as TFLite and SavedModel

Now we can export the trained models in TFLite and SavedModel formats for deploying on-device and using for inference in TensorFlow.

tflite_filename = f'{TFLITE_NAME_PREFIX}_model_{model_name}.tflite'
model.export(export_dir='.', tflite_filename=tflite_filename)
INFO:tensorflow:Assets written to: /tmp/tmp9ta6i955/assets
INFO:tensorflow:Assets written to: /tmp/tmp9ta6i955/assets
WARNING:absl:For model inputs containing unsupported operations which cannot be quantized, the `inference_input_type` attribute will default to the original type.
INFO:tensorflow:Label file is inside the TFLite model with metadata.
INFO:tensorflow:Label file is inside the TFLite model with metadata.
INFO:tensorflow:Saving labels in /tmp/tmpvxvhgh61/labels.txt
INFO:tensorflow:Saving labels in /tmp/tmpvxvhgh61/labels.txt
INFO:tensorflow:TensorFlow Lite model exported successfully: ./cassava_model_mobilenet_v3_large_100_224.tflite
INFO:tensorflow:TensorFlow Lite model exported successfully: ./cassava_model_mobilenet_v3_large_100_224.tflite
# Export saved model version.
model.export(export_dir='.', export_format=ExportFormat.SAVED_MODEL)
INFO:tensorflow:Assets written to: ./saved_model/assets
INFO:tensorflow:Assets written to: ./saved_model/assets

Next steps

The model that you've just trained can be used on mobile devices and even deployed in the field!

To download the model, click the folder icon for the Files menu on the left side of the colab, and choose the download option.

The same technique used here could be applied to other plant diseases tasks that might be more suitable for your use case or any other type of image classification task. If you want to follow up and deploy on an Android app, you can continue on this Android quickstart guide.