# Generate music with an RNN

<br />

|---|---|---|---|
| [![](https://www.tensorflow.org/images/tf_logo_32px.png)View on TensorFlow.org](https://www.tensorflow.org/tutorials/audio/music_generation) | [![](https://www.tensorflow.org/images/colab_logo_32px.png)Run in Google Colab](https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/audio/music_generation.ipynb) | [![](https://www.tensorflow.org/images/GitHub-Mark-32px.png)View source on GitHub](https://github.com/tensorflow/docs/blob/master/site/en/tutorials/audio/music_generation.ipynb) | [![](https://www.tensorflow.org/images/download_logo_32px.png)Download notebook](https://storage.googleapis.com/tensorflow_docs/docs/site/en/tutorials/audio/music_generation.ipynb) |

This tutorial shows you how to generate musical notes using a simple recurrent neural network (RNN). You will train a model using a collection of piano MIDI files from the [MAESTRO dataset](https://magenta.tensorflow.org/datasets/maestro). Given a sequence of notes, your model will learn to predict the next note in the sequence. You can generate longer sequences of notes by calling the model repeatedly.

This tutorial contains complete code to parse and create MIDI files. You can learn more about how RNNs work by visiting the [Text generation with an RNN](https://www.tensorflow.org/text/tutorials/text_generation) tutorial.

## Setup

This tutorial uses the [`pretty_midi`](https://github.com/craffel/pretty-midi) library to create and parse MIDI files, and [`pyfluidsynth`](https://github.com/nwhitehead/pyfluidsynth) for generating audio playback in Colab.

    sudo apt install -y fluidsynth

    pip install --upgrade pyfluidsynth

    pip install pretty_midi

    import collections
    import datetime
    import fluidsynth
    import glob
    import numpy as np
    import pathlib
    import pandas as pd
    import pretty_midi
    import seaborn as sns
    import tensorflow as tf

    from IPython import display
    from matplotlib import pyplot as plt
    from typing import Optional

    seed = 42
    tf.random.set_seed(seed)
    np.random.seed(seed)

    # Sampling rate for audio playback
    _SAMPLING_RATE = 16000

## Download the Maestro dataset

    data_dir = pathlib.Path('data/maestro-v2.0.0')
    if not data_dir.exists():
      tf.keras.utils.get_file(
          'maestro-v2.0.0-midi.zip',
          origin='https://storage.googleapis.com/magentadata/datasets/maestro/v2.0.0/maestro-v2.0.0-midi.zip',
          extract=True,
          cache_dir='.', cache_subdir='data',
      )

The dataset contains about 1,200 MIDI files.

    filenames = glob.glob(str(data_dir/'**/*.mid*'))
    print('Number of files:', len(filenames))

## Process a MIDI file

First, use `pretty_midi` to parse a single MIDI file and inspect the format of the notes. If you would like to download the MIDI file below to play on your computer, you can do so in colab by writing `files.download(sample_file)`.

    sample_file = filenames[1]
    print(sample_file)

Generate a `PrettyMIDI` object for the sample MIDI file.

    pm = pretty_midi.PrettyMIDI(sample_file)

Play the sample file. The playback widget may take several seconds to load.

    def display_audio(pm: pretty_midi.PrettyMIDI, seconds=30):
      waveform = pm.fluidsynth(fs=_SAMPLING_RATE)
      # Take a sample of the generated waveform to mitigate kernel resets
      waveform_short = waveform[:seconds*_SAMPLING_RATE]
      return display.Audio(waveform_short, rate=_SAMPLING_RATE)

    display_audio(pm)

Do some inspection on the MIDI file. What kinds of instruments are used?

    print('Number of instruments:', len(pm.instruments))
    instrument = pm.instruments[0]
    instrument_name = pretty_midi.program_to_instrument_name(instrument.program)
    print('Instrument name:', instrument_name)

## Extract notes

    for i, note in enumerate(instrument.notes[:10]):
      note_name = pretty_midi.note_number_to_name(note.pitch)
      duration = note.end - note.start
      print(f'{i}: pitch={note.pitch}, note_name={note_name},'
            f' duration={duration:.4f}')

You will use three variables to represent a note when training the model: `pitch`, `step` and `duration`. The pitch is the perceptual quality of the sound as a MIDI note number.
The `step` is the time elapsed from the previous note or start of the track.
The `duration` is how long the note will be playing in seconds and is the difference between the note end and note start times.

Extract the notes from the sample MIDI file.

    def midi_to_notes(midi_file: str) -> pd.DataFrame:
      pm = pretty_midi.PrettyMIDI(midi_file)
      instrument = pm.instruments[0]
      notes = collections.defaultdict(list)

      # Sort the notes by start time
      sorted_notes = sorted(instrument.notes, key=lambda note: note.start)
      prev_start = sorted_notes[0].start

      for note in sorted_notes:
        start = note.start
        end = note.end
        notes['pitch'].append(note.pitch)
        notes['start'].append(start)
        notes['end'].append(end)
        notes['step'].append(start - prev_start)
        notes['duration'].append(end - start)
        prev_start = start

      return pd.DataFrame({name: np.array(value) for name, value in notes.items()})

    raw_notes = midi_to_notes(sample_file)
    raw_notes.head()

It may be easier to interpret the note names rather than the pitches, so you can use the function below to convert from the numeric pitch values to note names.
The note name shows the type of note, accidental and octave number
(e.g. C#4).

    get_note_names = np.vectorize(pretty_midi.note_number_to_name)
    sample_note_names = get_note_names(raw_notes['pitch'])
    sample_note_names[:10]

To visualize the musical piece, plot the note pitch, start and end across the length of the track (i.e. piano roll). Start with the first 100 notes

    def plot_piano_roll(notes: pd.DataFrame, count: Optional[int] = None):
      if count:
        title = f'First {count} notes'
      else:
        title = f'Whole track'
        count = len(notes['pitch'])
      plt.figure(figsize=(20, 4))
      plot_pitch = np.stack([notes['pitch'], notes['pitch']], axis=0)
      plot_start_stop = np.stack([notes['start'], notes['end']], axis=0)
      plt.plot(
          plot_start_stop[:, :count], plot_pitch[:, :count], color="b", marker=".")
      plt.xlabel('Time [s]')
      plt.ylabel('Pitch')
      _ = plt.title(title)

    plot_piano_roll(raw_notes, count=100)

Plot the notes for the entire track.

    plot_piano_roll(raw_notes)

Check the distribution of each note variable.

    def plot_distributions(notes: pd.DataFrame, drop_percentile=2.5):
      plt.figure(figsize=[15, 5])
      plt.subplot(1, 3, 1)
      sns.histplot(notes, x="pitch", bins=20)

      plt.subplot(1, 3, 2)
      max_step = np.percentile(notes['step'], 100 - drop_percentile)
      sns.histplot(notes, x="step", bins=np.linspace(0, max_step, 21))

      plt.subplot(1, 3, 3)
      max_duration = np.percentile(notes['duration'], 100 - drop_percentile)
      sns.histplot(notes, x="duration", bins=np.linspace(0, max_duration, 21))

    plot_distributions(raw_notes)

## Create a MIDI file

You can generate your own MIDI file from a list of notes using the function below.

    def notes_to_midi(
      notes: pd.DataFrame,
      out_file: str, 
      instrument_name: str,
      velocity: int = 100,  # note loudness
    ) -> pretty_midi.PrettyMIDI:

      pm = pretty_midi.PrettyMIDI()
      instrument = pretty_midi.Instrument(
          program=pretty_midi.instrument_name_to_program(
              instrument_name))

      prev_start = 0
      for i, note in notes.iterrows():
        start = float(prev_start + note['step'])
        end = float(start + note['duration'])
        note = pretty_midi.Note(
            velocity=velocity,
            pitch=int(note['pitch']),
            start=start,
            end=end,
        )
        instrument.notes.append(note)
        prev_start = start

      pm.instruments.append(instrument)
      pm.write(out_file)
      return pm

    example_file = 'example.midi'
    example_pm = notes_to_midi(
        raw_notes, out_file=example_file, instrument_name=instrument_name)

Play the generated MIDI file and see if there is any difference.

    display_audio(example_pm)

As before, you can write `files.download(example_file)` to download and play this file.

## Create the training dataset

Create the training dataset by extracting notes from the MIDI files. You can start by using a small number of files, and experiment later with more. This may take a couple minutes.

    num_files = 5
    all_notes = []
    for f in filenames[:num_files]:
      notes = midi_to_notes(f)
      all_notes.append(notes)

    all_notes = pd.concat(all_notes)

    n_notes = len(all_notes)
    print('Number of notes parsed:', n_notes)

Next, create a [`tf.data.Dataset`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) from the parsed notes.

    key_order = ['pitch', 'step', 'duration']
    train_notes = np.stack([all_notes[key] for key in key_order], axis=1)

    notes_ds = tf.data.Dataset.from_tensor_slices(train_notes)
    notes_ds.element_spec

You will train the model on batches of sequences of notes. Each example will consist of a sequence of notes as the input features, and the next note as the label. In this way, the model will be trained to predict the next note in a sequence. You can find a diagram describing this process (and more details) in [Text classification with an RNN](https://www.tensorflow.org/text/tutorials/text_generation).

You can use the handy [window](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#window) function with size `seq_length` to create the features and labels in this format.

    def create_sequences(
        dataset: tf.data.Dataset, 
        seq_length: int,
        vocab_size = 128,
    ) -> tf.data.Dataset:
      """Returns TF Dataset of sequence and label examples."""
      seq_length = seq_length+1

      # Take 1 extra for the labels
      windows = dataset.window(seq_length, shift=1, stride=1,
                                  drop_remainder=True)

      # `flat_map` flattens the" dataset of datasets" into a dataset of tensors
      flatten = lambda x: x.batch(seq_length, drop_remainder=True)
      sequences = windows.flat_map(flatten)

      # Normalize note pitch
      def scale_pitch(x):
        x = x/[vocab_size,1.0,1.0]
        return x

      # Split the labels
      def split_labels(sequences):
        inputs = sequences[:-1]
        labels_dense = sequences[-1]
        labels = {key:labels_dense[i] for i,key in enumerate(key_order)}

        return scale_pitch(inputs), labels

      return sequences.map(split_labels, num_parallel_calls=tf.data.AUTOTUNE)

Set the sequence length for each example. Experiment with different lengths (e.g. 50, 100, 150) to see which one works best for the data, or use [hyperparameter tuning](https://www.tensorflow.org/tutorials/keras/keras_tuner). The size of the vocabulary (`vocab_size`) is set to 128 representing all the pitches supported by `pretty_midi`.

    seq_length = 25
    vocab_size = 128
    seq_ds = create_sequences(notes_ds, seq_length, vocab_size)
    seq_ds.element_spec

The shape of the dataset is `(100,1)`, meaning that the model will take 100 notes as input, and learn to predict the following note as output.

    for seq, target in seq_ds.take(1):
      print('sequence shape:', seq.shape)
      print('sequence elements (first 10):', seq[0: 10])
      print()
      print('target:', target)

Batch the examples, and configure the dataset for performance.

    batch_size = 64
    buffer_size = n_notes - seq_length  # the number of items in the dataset
    train_ds = (seq_ds
                .shuffle(buffer_size)
                .batch(batch_size, drop_remainder=True)
                .cache()
                .prefetch(tf.data.experimental.AUTOTUNE))

    train_ds.element_spec

## Create and train the model

The model will have three outputs, one for each note variable. For `step` and `duration`, you will use a custom loss function based on mean squared error that encourages the model to output non-negative values.

    def mse_with_positive_pressure(y_true: tf.Tensor, y_pred: tf.Tensor):
      mse = (y_true - y_pred) ** 2
      positive_pressure = 10 * tf.maximum(-y_pred, 0.0)
      return tf.reduce_mean(mse + positive_pressure)

    input_shape = (seq_length, 3)
    learning_rate = 0.005

    inputs = tf.keras.Input(input_shape)
    x = tf.keras.layers.LSTM(128)(inputs)

    outputs = {
      'pitch': tf.keras.layers.Dense(128, name='pitch')(x),
      'step': tf.keras.layers.Dense(1, name='step')(x),
      'duration': tf.keras.layers.Dense(1, name='duration')(x),
    }

    model = tf.keras.Model(inputs, outputs)

    loss = {
          'pitch': tf.keras.losses.SparseCategoricalCrossentropy(
              from_logits=True),
          'step': mse_with_positive_pressure,
          'duration': mse_with_positive_pressure,
    }

    optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)

    model.compile(loss=loss, optimizer=optimizer)

    model.summary()

Testing the `model.evaluate` function, you can see that the `pitch` loss is significantly greater than the `step` and `duration` losses.
Note that `loss` is the total loss computed by summing all the other losses and is currently dominated by the `pitch` loss.

    losses = model.evaluate(train_ds, return_dict=True)
    losses

One way balance this is to use the `loss_weights` argument to compile:

    model.compile(
        loss=loss,
        loss_weights={
            'pitch': 0.05,
            'step': 1.0,
            'duration':1.0,
        },
        optimizer=optimizer,
    )

The `loss` then becomes the weighted sum of the individual losses.

    model.evaluate(train_ds, return_dict=True)

Train the model.

    callbacks = [
        tf.keras.callbacks.ModelCheckpoint(
            filepath='./training_checkpoints/ckpt_{epoch}',
            save_weights_only=True),
        tf.keras.callbacks.EarlyStopping(
            monitor='loss',
            patience=5,
            verbose=1,
            restore_best_weights=True),
    ]

    %%time
    epochs = 50

    history = model.fit(
        train_ds,
        epochs=epochs,
        callbacks=callbacks,
    )

    plt.plot(history.epoch, history.history['loss'], label='total loss')
    plt.show()

## Generate notes

To use the model to generate notes, you will first need to provide a starting sequence of notes. The function below generates one note from a sequence of notes.

For note pitch, it draws a sample from the softmax distribution of notes produced by the model, and does not simply pick the note with the highest probability.
Always picking the note with the highest probability would lead to repetitive sequences of notes being generated.

The `temperature` parameter can be used to control the randomness of notes generated. You can find more details on temperature in [Text generation with an RNN](https://www.tensorflow.org/text/tutorials/text_generation).

    def predict_next_note(
        notes: np.ndarray, 
        model: tf.keras.Model, 
        temperature: float = 1.0) -> tuple[int, float, float]:
      """Generates a note as a tuple of (pitch, step, duration), using a trained sequence model."""

      assert temperature > 0

      # Add batch dimension
      inputs = tf.expand_dims(notes, 0)

      predictions = model.predict(inputs)
      pitch_logits = predictions['pitch']
      step = predictions['step']
      duration = predictions['duration']

      pitch_logits /= temperature
      pitch = tf.random.categorical(pitch_logits, num_samples=1)
      pitch = tf.squeeze(pitch, axis=-1)
      duration = tf.squeeze(duration, axis=-1)
      step = tf.squeeze(step, axis=-1)

      # `step` and `duration` values should be non-negative
      step = tf.maximum(0, step)
      duration = tf.maximum(0, duration)

      return int(pitch), float(step), float(duration)

Now generate some notes. You can play around with temperature and the starting sequence in `next_notes` and see what happens.

    temperature = 2.0
    num_predictions = 120

    sample_notes = np.stack([raw_notes[key] for key in key_order], axis=1)

    # The initial sequence of notes; pitch is normalized similar to training
    # sequences
    input_notes = (
        sample_notes[:seq_length] / np.array([vocab_size, 1, 1]))

    generated_notes = []
    prev_start = 0
    for _ in range(num_predictions):
      pitch, step, duration = predict_next_note(input_notes, model, temperature)
      start = prev_start + step
      end = start + duration
      input_note = (pitch, step, duration)
      generated_notes.append((*input_note, start, end))
      input_notes = np.delete(input_notes, 0, axis=0)
      input_notes = np.append(input_notes, np.expand_dims(input_note, 0), axis=0)
      prev_start = start

    generated_notes = pd.DataFrame(
        generated_notes, columns=(*key_order, 'start', 'end'))

    generated_notes.head(10)

    out_file = 'output.mid'
    out_pm = notes_to_midi(
        generated_notes, out_file=out_file, instrument_name=instrument_name)
    display_audio(out_pm)

You can also download the audio file by adding the two lines below:

    from google.colab import files
    files.download(out_file)

Visualize the generated notes.

    plot_piano_roll(generated_notes)

Check the distributions of `pitch`, `step` and `duration`.

    plot_distributions(generated_notes)

In the above plots, you will notice the change in distribution of the note variables.
Since there is a feedback loop between the model's outputs and inputs, the model tends to generate similar sequences of outputs to reduce the loss.
This is particularly relevant for `step` and `duration`, which uses the MSE loss.
For `pitch`, you can increase the randomness by increasing the `temperature` in `predict_next_note`.

## Next steps

This tutorial demonstrated the mechanics of using an RNN to generate sequences of notes from a dataset of MIDI files. To learn more, you can visit the closely related [Text generation with an RNN](https://www.tensorflow.org/text/tutorials/text_generation) tutorial, which contains additional diagrams and explanations.

One of the alternatives to using RNNs for music generation is using GANs. Rather than generating audio, a GAN-based approach can generate an entire sequence in parallel. The Magenta team has done impressive work on this approach with [GANSynth](https://magenta.tensorflow.org/gansynth). You can also find many wonderful music and art projects and open-source code on [Magenta project website](https://magenta.tensorflow.org/).