Weight clustering

Maintained by Arm ML Tooling

This document provides an overview on weight clustering to help you determine how it fits with your use case.

Overview

Clustering, or weight sharing, reduces the number of unique weight values in a model, leading to benefits for deployment. It first groups the weights of each layer into N clusters, then shares the cluster's centroid value for all the weights belonging to the cluster.

This technique brings improvements via model compression. Future framework support can unlock memory footprint improvements that can make a crucial difference for deploying deep learning models on embedded systems with limited resources.

We have experimented with clustering across vision and speech tasks. We've seen up to 5x improvements in model compression with minimal loss of accuracy, as demonstrated by the results presented below.

Please note that clustering will provide reduced benefits for convolution and dense layers that precede a batch normalization layer, as well as in combination with per-axis post-training quantization.

API compatibility matrix

Users can apply clustering with the following APIs:

  • Model building: keras with only Sequential and Functional models
  • TensorFlow versions: TF 1.x for versions 1.14+ and 2.x.
    • tf.compat.v1 with a TF 2.X package and tf.compat.v2 with a TF 1.X package are not supported.
  • TensorFlow execution mode: both graph and eager

Results

Image classification

Model Original Clustered
Top-1 accuracy (%) Size of compressed .tflite (MB) Configuration # of clusters Top-1 accuracy (%) Size of compressed .tflite (MB)
MobileNetV1 70.976 14.97
Selective (last 3 Conv2D layers) 16, 16, 16 70.294 7.69
Selective (last 3 Conv2D layers) 32, 32, 32 70.69 8.22
Full (all Conv2D layers) 32 69.4 4.43
MobileNetV2 71.778 12.38
Selective (last 3 Conv2D layers) 16, 16, 16 70.742 6.68
Selective (last 3 Conv2D layers) 32, 32, 32 70.926 7.03
Full (all Conv2D layers) 32 69.744 4.05

The models were trained and tested on ImageNet.

Keyword spotting

Model Original Clustered
Top-1 accuracy (%) Size of compressed .tflite (MB) Configuration # of clusters Top-1 accuracy (%) Size of compressed .tflite (MB)
DS-CNN-L 95.233 1.46
Full (all Conv2D layers) 32 95.09 0.39
Full (all Conv2D layers) 8 94.272 0.27

The model was trained and tested on SpeechCommands v0.02.

  1. Serialize the Keras model into .h5 file
  2. Convert the .h5 file into .tflite using TFLiteConverter.from_keras_model_file()
  3. Compress the .tflite file into a zip

Examples

In addition to the Weight clustering in Keras example, see the following examples:

  • Cluster the weights of a CNN model trained on the MNIST handwritten digit classification dataset: code

The weight clustering implementation is based on the Deep Compression: Compressing Deep Neural Networks With Pruning, Trained Quantization and Huffman Coding paper. See chapter 3, titled Trained Quantization and Weight Sharing.