Hari Komunitas ML adalah 9 November! Bergabung dengan kami untuk update dari TensorFlow, JAX, dan lebih Pelajari lebih lanjut

Panduan komprehensif pengelompokan berat badan

Lihat di TensorFlow.org Jalankan di Google Colab Lihat sumber di GitHub Unduh buku catatan

Welcome to the comprehensive guide for weight clustering , part of the TensorFlow Model Optimization toolkit.

Halaman ini mendokumentasikan berbagai kasus penggunaan dan menunjukkan cara menggunakan API untuk masing-masing kasus. Once you know which APIs you need, find the parameters and the low-level details in the API docs :

  • If you want to see the benefits of weight clustering and what's supported, check the overview .
  • For a single end-to-end example, see the weight clustering example .

Dalam panduan ini, kasus penggunaan berikut dibahas:

  • Tentukan model berkerumun.
  • Checkpoint dan deserialize model berkerumun.
  • Meningkatkan akurasi model berkerumun.
  • Untuk penerapan saja, Anda harus mengambil langkah-langkah untuk melihat manfaat kompresi.

Mempersiapkan

! pip install -q tensorflow-model-optimization

import tensorflow as tf
import numpy as np
import tempfile
import os
import tensorflow_model_optimization as tfmot

input_dim = 20
output_dim = 20
x_train = np.random.randn(1, input_dim).astype(np.float32)
y_train = tf.keras.utils.to_categorical(np.random.randn(1), num_classes=output_dim)

def setup_model():
  model = tf.keras.Sequential([
      tf.keras.layers.Dense(input_dim, input_shape=[input_dim]),
      tf.keras.layers.Flatten()
  ])
  return model

def train_model(model):
  model.compile(
      loss=tf.keras.losses.categorical_crossentropy,
      optimizer='adam',
      metrics=['accuracy']
  )
  model.summary()
  model.fit(x_train, y_train)
  return model

def save_model_weights(model):
  _, pretrained_weights = tempfile.mkstemp('.h5')
  model.save_weights(pretrained_weights)
  return pretrained_weights

def setup_pretrained_weights():
  model= setup_model()
  model = train_model(model)
  pretrained_weights = save_model_weights(model)
  return pretrained_weights

def setup_pretrained_model():
  model = setup_model()
  pretrained_weights = setup_pretrained_weights()
  model.load_weights(pretrained_weights)
  return model

def save_model_file(model):
  _, keras_file = tempfile.mkstemp('.h5') 
  model.save(keras_file, include_optimizer=False)
  return keras_file

def get_gzipped_model_size(model):
  # It returns the size of the gzipped model in bytes.
  import os
  import zipfile

  keras_file = save_model_file(model)

  _, zipped_file = tempfile.mkstemp('.zip')
  with zipfile.ZipFile(zipped_file, 'w', compression=zipfile.ZIP_DEFLATED) as f:
    f.write(keras_file)
  return os.path.getsize(zipped_file)

setup_model()
pretrained_weights = setup_pretrained_weights()

Tentukan model berkerumun

Cluster seluruh model (berurutan dan fungsional)

Tips for better model accuracy:

  • Anda harus meneruskan model terlatih dengan akurasi yang dapat diterima ke API ini. Melatih model dari awal dengan pengelompokan menghasilkan akurasi di bawah standar.
  • Dalam beberapa kasus, pengelompokan lapisan tertentu memiliki efek merugikan pada akurasi model. Centang "Kelompokkan beberapa lapisan" untuk melihat cara melewati pengelompokan lapisan yang paling memengaruhi akurasi.

To cluster all layers, apply tfmot.clustering.keras.cluster_weights to the model.

import tensorflow_model_optimization as tfmot

cluster_weights = tfmot.clustering.keras.cluster_weights
CentroidInitialization = tfmot.clustering.keras.CentroidInitialization

clustering_params = {
  'number_of_clusters': 3,
  'cluster_centroids_init': CentroidInitialization.KMEANS_PLUS_PLUS
}

model = setup_model()
model.load_weights(pretrained_weights)

clustered_model = cluster_weights(model, **clustering_params)

clustered_model.summary()
Model: "sequential_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
cluster_dense_2 (ClusterWeig (None, 20)                823       
_________________________________________________________________
cluster_flatten_2 (ClusterWe (None, 20)                0         
=================================================================
Total params: 823
Trainable params: 423
Non-trainable params: 400
_________________________________________________________________

Kelompokkan beberapa lapisan (model sekuensial dan fungsional)

Tips for better model accuracy:

  • Anda harus meneruskan model terlatih dengan akurasi yang dapat diterima ke API ini. Melatih model dari awal dengan pengelompokan menghasilkan akurasi di bawah standar.
  • Cluster later layers with more redundant parameters (eg tf.keras.layers.Dense , tf.keras.layers.Conv2D ), as opposed to the early layers.
  • Bekukan lapisan awal sebelum lapisan berkerumun selama fine-tuning. Perlakukan jumlah lapisan beku sebagai hyperparameter. Secara empiris, membekukan sebagian besar lapisan awal sangat ideal untuk API pengelompokan saat ini.
  • Hindari pengelompokan lapisan kritis (misalnya mekanisme perhatian).

More : the tfmot.clustering.keras.cluster_weights API docs provide details on how to vary the clustering configuration per layer.

# Create a base model
base_model = setup_model()
base_model.load_weights(pretrained_weights)

# Helper function uses `cluster_weights` to make only 
# the Dense layers train with clustering
def apply_clustering_to_dense(layer):
  if isinstance(layer, tf.keras.layers.Dense):
    return cluster_weights(layer, **clustering_params)
  return layer

# Use `tf.keras.models.clone_model` to apply `apply_clustering_to_dense` 
# to the layers of the model.
clustered_model = tf.keras.models.clone_model(
    base_model,
    clone_function=apply_clustering_to_dense,
)

clustered_model.summary()
Model: "sequential_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
cluster_dense_3 (ClusterWeig (None, 20)                823       
_________________________________________________________________
flatten_3 (Flatten)          (None, 20)                0         
=================================================================
Total params: 823
Trainable params: 423
Non-trainable params: 400
_________________________________________________________________

Kelompokkan lapisan Keras khusus atau tentukan bobot lapisan mana yang akan dikelompokkan

tfmot.clustering.keras.ClusterableLayer serves two use cases:

  1. Kelompokkan lapisan apa pun yang tidak didukung secara asli, termasuk lapisan Keras khusus.
  2. Tentukan bobot mana dari lapisan yang didukung yang akan dikelompokkan.

For an example, the API defaults to only clustering the kernel of the Dense layer. Contoh di bawah ini menunjukkan cara memodifikasinya untuk juga mengelompokkan bias. Note that when deriving from the keras layer, you need to override the function get_clusterable_weights , where you specify the name of the trainable variable to be clustered and the trainable variable itself. Misalnya, jika Anda mengembalikan daftar kosong [], maka tidak ada bobot yang dapat dikelompokkan.

Common mistake: clustering the bias usually harms model accuracy too much.

class MyDenseLayer(tf.keras.layers.Dense, tfmot.clustering.keras.ClusterableLayer):

  def get_clusterable_weights(self):
   # Cluster kernel and bias. This is just an example, clustering
   # bias usually hurts model accuracy.
   return [('kernel', self.kernel), ('bias', self.bias)]

# Use `cluster_weights` to make the `MyDenseLayer` layer train with clustering as usual.
model_for_clustering = tf.keras.Sequential([
  tfmot.clustering.keras.cluster_weights(MyDenseLayer(20, input_shape=[input_dim]), **clustering_params),
  tf.keras.layers.Flatten()
])

model_for_clustering.summary()
Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
cluster_my_dense_layer (Clus (None, 20)                846       
_________________________________________________________________
flatten_4 (Flatten)          (None, 20)                0         
=================================================================
Total params: 846
Trainable params: 426
Non-trainable params: 420
_________________________________________________________________

You may also use tfmot.clustering.keras.ClusterableLayer to cluster a keras custom layer. To do this, you extend tf.keras.Layer as usual and implement the __init__ , call , and build functions, but you also need to extend the clusterable_layer.ClusterableLayer class and implement:

  1. get_clusterable_weights , where you specify the weight kernel to be clustered, as shown above.
  2. get_clusterable_algorithm , where you specify the clustering algorithm for the weight tensor. Ini karena Anda perlu menentukan bagaimana bobot lapisan khusus dibentuk untuk pengelompokan. The returned clustering algorithm class should be derived from the clustering_algorithm.ClusteringAlgorithm class and the function get_pulling_indices should be overwritten. An example of this function, which supports weights of ranks 1D, 2D, and 3D, can be found here .

An example of this use case can be found here .

Checkpoint dan deserialize model berkerumun

Your use case: this code is only needed for the HDF5 model format (not HDF5 weights or other formats).

# Define the model.
base_model = setup_model()
base_model.load_weights(pretrained_weights)
clustered_model = cluster_weights(base_model, **clustering_params)

# Save or checkpoint the model.
_, keras_model_file = tempfile.mkstemp('.h5')
clustered_model.save(keras_model_file, include_optimizer=True)

# `cluster_scope` is needed for deserializing HDF5 models.
with tfmot.clustering.keras.cluster_scope():
  loaded_model = tf.keras.models.load_model(keras_model_file)

loaded_model.summary()
WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
Model: "sequential_5"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
cluster_dense_4 (ClusterWeig (None, 20)                823       
_________________________________________________________________
cluster_flatten_5 (ClusterWe (None, 20)                0         
=================================================================
Total params: 823
Trainable params: 423
Non-trainable params: 400
_________________________________________________________________

Tingkatkan akurasi model berkerumun

Untuk kasus penggunaan khusus Anda, ada tips yang dapat Anda pertimbangkan:

  • Inisialisasi Centroid memainkan peran kunci dalam akurasi model akhir yang dioptimalkan. Secara umum, inisialisasi kmeans++ mengungguli inisialisasi linier, kepadatan, dan acak. Saat tidak menggunakan kmeans++, inisialisasi linier cenderung mengungguli kepadatan dan inisialisasi acak, karena tidak cenderung kehilangan bobot yang besar. Namun, inisialisasi densitas telah diamati untuk memberikan akurasi yang lebih baik untuk kasus penggunaan kluster yang sangat sedikit pada bobot dengan distribusi bimodal.

  • Tetapkan kecepatan pembelajaran yang lebih rendah dari yang digunakan dalam pelatihan saat menyempurnakan model kluster.

  • Untuk gagasan umum guna meningkatkan akurasi model, cari tip untuk kasus penggunaan Anda di bawah "Tentukan model terklaster".

Penyebaran

Model ekspor dengan kompresi ukuran

Common mistake : both strip_clustering and applying a standard compression algorithm (eg via gzip) are necessary to see the compression benefits of clustering.

model = setup_model()
clustered_model = cluster_weights(model, **clustering_params)

clustered_model.compile(
    loss=tf.keras.losses.categorical_crossentropy,
    optimizer='adam',
    metrics=['accuracy']
)

clustered_model.fit(
    x_train,
    y_train
)

final_model = tfmot.clustering.keras.strip_clustering(clustered_model)

print("final model")
final_model.summary()

print("\n")
print("Size of gzipped clustered model without stripping: %.2f bytes" 
      % (get_gzipped_model_size(clustered_model)))
print("Size of gzipped clustered model with stripping: %.2f bytes" 
      % (get_gzipped_model_size(final_model)))
1/1 [==============================] - 0s 343ms/step - loss: 16.1181 - accuracy: 0.0000e+00
final model
Model: "sequential_6"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_5 (Dense)              (None, 20)                420       
_________________________________________________________________
flatten_6 (Flatten)          (None, 20)                0         
=================================================================
Total params: 420
Trainable params: 420
Non-trainable params: 0
_________________________________________________________________


Size of gzipped clustered model without stripping: 3447.00 bytes
WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.
Size of gzipped clustered model with stripping: 1433.00 bytes