Classification on imbalanced data

View on TensorFlow.org

Run in Google Colab

View source on GitHub

Download notebook

This tutorial demonstrates how to classify a highly imbalanced dataset in which the number of examples in one class greatly outnumbers the examples in another. You will work with the Credit Card Fraud Detection dataset hosted on Kaggle. The aim is to detect a mere 492 fraudulent transactions from 284,807 transactions in total. You will use Keras to define the model and class weights to help the model learn from the imbalanced data. .

This tutorial contains complete code to:

• Load a CSV file using Pandas.
• Create train, validation, and test sets.
• Define and train a model using Keras (including setting class weights).
• Evaluate the model using various metrics (including precision and recall).
• Try common tecniques for dealing with imbalanced data like:
  • Class weighting
  • Oversampling

Setup

from __future__ import absolute_import, division, print_function, unicode_literals

import tensorflow as tf
from tensorflow import keras

import os
import tempfile

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

import sklearn
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

mpl.rcParams['figure.figsize'] = (12, 10)
colors = plt.rcParams['axes.prop_cycle'].by_key()['color']

Data processing and exploration

Download the Kaggle Credit Card Fraud data set

Pandas is a Python library with many helpful utilities for loading and working with structured data and can be used to download CSVs into a dataframe.

file = tf.keras.utils
raw_df = pd.read_csv('https://storage.googleapis.com/download.tensorflow.org/data/creditcard.csv')
raw_df.head()

raw_df[['Time', 'V1', 'V2', 'V3', 'V4', 'V5', 'V26', 'V27', 'V28', 'Amount', 'Class']].describe()

Examine the class label imbalance

Let's look at the dataset imbalance:

neg, pos = np.bincount(raw_df['Class'])
total = neg + pos
print('Examples:\n    Total: {}\n    Positive: {} ({:.2f}% of total)\n'.format(
    total, pos, 100 * pos / total))

Examples:
    Total: 284807
    Positive: 492 (0.17% of total)

This shows the small fraction of positive samples.

Clean, split and normalize the data

The raw data has a few issues. First the Time and Amount columns are too variable to use directly. Drop the Time column (since it's not clear what it means) and take the log of the Ammount column to reduce its range.

cleaned_df = raw_df.copy()

# You don't want the `Time` column.
cleaned_df.pop('Time')

# The `Amount` column covers a huge range. Convert to log-space.
eps=0.001 # 0 => 0.1¢
cleaned_df['Log Ammount'] = np.log(cleaned_df.pop('Amount')+eps)

Split the dataset into train, validation, and test sets. The validation set is used during the model fitting to evaluate the loss and any metrics, however the model is not fit with this data. The test set is completely unused during the training phase and is only used at the end to evaluate how well the model generalizes to new data. This is especially important with imbalanced datasets where overfitting is a significant concern from the lack of training data.

# Use a utility from sklearn to split and shuffle our dataset.
train_df, test_df = train_test_split(cleaned_df, test_size=0.2)
train_df, val_df = train_test_split(train_df, test_size=0.2)

# Form np arrays of labels and features.
train_labels = np.array(train_df.pop('Class'))
bool_train_labels = train_labels != 0
val_labels = np.array(val_df.pop('Class'))
test_labels = np.array(test_df.pop('Class'))

train_features = np.array(train_df)
val_features = np.array(val_df)
test_features = np.array(test_df)

Normalize the input features using the sklearn StandardScaler. This will set the mean to 0 and standard deviation to 1.

scaler = StandardScaler()
train_features = scaler.fit_transform(train_features)

val_features = scaler.transform(val_features)
test_features = scaler.transform(test_features)

train_features = np.clip(train_features, -5, 5)
val_features = np.clip(val_features, -5, 5)
test_features = np.clip(test_features, -5, 5)


print('Training labels shape:', train_labels.shape)
print('Validation labels shape:', val_labels.shape)
print('Test labels shape:', test_labels.shape)

print('Training features shape:', train_features.shape)
print('Validation features shape:', val_features.shape)
print('Test features shape:', test_features.shape)

Training labels shape: (182276,)
Validation labels shape: (45569,)
Test labels shape: (56962,)
Training features shape: (182276, 29)
Validation features shape: (45569, 29)
Test features shape: (56962, 29)

Look at the data distribution

Next compare the distributions of the positive and negative examples over a few features. Good questions to ask yourdelf at this point are:

• Do these distributions make sense?
  • Yes. You've normalized the input and these are mostly concentrated in the +/- 2 range.
• Can you see the difference between the ditributions?
  • Yes the positive examples contain a much higher rate of extreme values.

pos_df = pd.DataFrame(train_features[ bool_train_labels], columns = train_df.columns)
neg_df = pd.DataFrame(train_features[~bool_train_labels], columns = train_df.columns)

sns.jointplot(pos_df['V5'], pos_df['V6'],
              kind='hex', xlim = (-5,5), ylim = (-5,5))
plt.suptitle("Positive distribution")

sns.jointplot(neg_df['V5'], neg_df['V6'],
              kind='hex', xlim = (-5,5), ylim = (-5,5))
_ = plt.suptitle("Negative distribution")

Define the model and metrics

Define a function that creates a simple neural network with a densly connected hidden layer, a dropout layer to reduce overfitting, and an output sigmoid layer that returns the probability of a transaction being fraudulent:

METRICS = [
      keras.metrics.TruePositives(name='tp'),
      keras.metrics.FalsePositives(name='fp'),
      keras.metrics.TrueNegatives(name='tn'),
      keras.metrics.FalseNegatives(name='fn'), 
      keras.metrics.BinaryAccuracy(name='accuracy'),
      keras.metrics.Precision(name='precision'),
      keras.metrics.Recall(name='recall'),
      keras.metrics.AUC(name='auc'),
]

def make_model(metrics = METRICS, output_bias=None):
  if output_bias is not None:
    output_bias = tf.keras.initializers.Constant(output_bias)
  model = keras.Sequential([
      keras.layers.Dense(
          16, activation='relu',
          input_shape=(train_features.shape[-1],)),
      keras.layers.Dropout(0.5),
      keras.layers.Dense(1, activation='sigmoid',
                         bias_initializer=output_bias),
  ])

  model.compile(
      optimizer=keras.optimizers.Adam(lr=1e-3),
      loss=keras.losses.BinaryCrossentropy(),
      metrics=metrics)

  return model

Understanding useful metrics

Notice that there are a few metrics defined above that can be computed by the model that will be helpful when evaluating the performance.

• False negatives and false positives are samples that were incorrectly classified
• True negatives and true positives are samples that were correctly classified
• Accuracy is the percentage of examples correctly classified > $\frac{\text{true samples}}{\text{total samples}}$
• Precision is the percentage of predicted positives that were correctly classified > $\frac{\text{true positives}}{\text{true positives + false positives}}$
• Recall is the percentage of actual positives that were correctly classified > $\frac{\text{true positives}}{\text{true positives + false negatives}}$
• AUC refers to the Area Under the Curve of a Receiver Operating Characteristic curve (ROC-AUC). This metric is equal to the probability that a classifier will rank a random positive sample higher than than a random negative sample.

Read more: * True vs. False and Positive vs. Negative * Accuracy * Precision and Recall * ROC-AUC

Baseline model

Build the model

Now create and train your model using the function that was defined earlier. Notice that the model is fit using a larger than default batch size of 2048, this is important to ensure that each batch has a decent chance of containing a few positive samples. If the batch size was too small, they would likely have no fraudelent transactions to learn from.

EPOCHS = 100
BATCH_SIZE = 2048

early_stopping = tf.keras.callbacks.EarlyStopping(
    monitor='val_auc', 
    verbose=1,
    patience=10,
    mode='max',
    restore_best_weights=True)

model = make_model()
model.summary()

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 16)                480       
_________________________________________________________________
dropout (Dropout)            (None, 16)                0         
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 17        
=================================================================
Total params: 497
Trainable params: 497
Non-trainable params: 0
_________________________________________________________________

Test run the model:

model.predict(train_features[:10])

array([[0.55185866],
       [0.6231443 ],
       [0.3516847 ],
       [0.70152867],
       [0.39482763],
       [0.444918  ],
       [0.52903605],
       [0.761968  ],
       [0.46861732],
       [0.7532199 ]], dtype=float32)

Optional: Set the correct initial bias.

These are initial guesses are not great. You know the dataset is imbalanced. Set the output layer's bias to reflect that (See: A Recipe for Training Neural Networks: "init well"). This can help with initial convergence.

With the default bias initialization the loss should be about math.log(2) = 0.69314

results = model.evaluate(train_features, train_labels, batch_size=BATCH_SIZE, verbose=0)
print("Loss: {:0.4f}".format(results[0]))

Loss: 0.8846

The correct bias to set can be derived from:

initial_bias = np.log([pos/neg])
initial_bias

array([-6.35935934])

Set that as the initial bias, and the model will give much more reasonable initial guesses.

It should be near: pos/total = 0.0018

model = make_model(output_bias = initial_bias)
model.predict(train_features[:10])

array([[0.00053543],
       [0.00028992],
       [0.00115347],
       [0.00082666],
       [0.00098124],
       [0.00224605],
       [0.00068536],
       [0.00158411],
       [0.00114388],
       [0.00070919]], dtype=float32)

With this initialization the initial loss should be approximately:

results = model.evaluate(train_features, train_labels, batch_size=BATCH_SIZE, verbose=0)
print("Loss: {:0.4f}".format(results[0]))

Loss: 0.0151

This initial loss is about 50 times less than if would have been with naive initilization.

This way the model doesn't need to spend the first few epochs just learning that positive examples are unlikely. This also makes it easier to read plots of the loss during training.

Checkpoint the initial weights

To make the various training runs more comparable, keep this initial model's weights in a checkpoint file, and load them into each model before training.

initial_weights = os.path.join(tempfile.mkdtemp(),'initial_weights')
model.save_weights(initial_weights)

Confirm that the bias fix helps

Before moving on, confirm quick that the careful bias initialization actually helped.

Train the model for 20 epochs, with and without this careful initialization, and compare the losses:

model = make_model()
model.load_weights(initial_weights)
model.layers[-1].bias.assign([0.0])
zero_bias_history = model.fit(
    train_features,
    train_labels,
    batch_size=BATCH_SIZE,
    epochs=20,
    validation_data=(val_features, val_labels), 
    verbose=0)

model = make_model()
model.load_weights(initial_weights)
careful_bias_history = model.fit(
    train_features,
    train_labels,
    batch_size=BATCH_SIZE,
    epochs=20,
    validation_data=(val_features, val_labels), 
    verbose=0)

def plot_loss(history, label, n):
  # Use a log scale to show the wide range of values.
  plt.semilogy(history.epoch,  history.history['loss'],
               color=colors[n], label='Train '+label)
  plt.semilogy(history.epoch,  history.history['val_loss'],
          color=colors[n], label='Val '+label,
          linestyle="--")
  plt.xlabel('Epoch')
  plt.ylabel('Loss')
  
  plt.legend()

plot_loss(zero_bias_history, "Zero Bias", 0)
plot_loss(careful_bias_history, "Careful Bias", 1)

The above figure makes it clear: In terms of validation loss, on this problem, this careful initialization gives a clear advantage.

Train the model

model = make_model()
model.load_weights(initial_weights)
baseline_history = model.fit(
    train_features,
    train_labels,
    batch_size=BATCH_SIZE,
    epochs=EPOCHS,
    callbacks = [early_stopping],
    validation_data=(val_features, val_labels))

Train on 182276 samples, validate on 45569 samples
Epoch 1/100
182276/182276 [==============================] - 3s 14us/sample - loss: 0.0106 - tp: 64.0000 - fp: 37.0000 - tn: 181909.0000 - fn: 266.0000 - accuracy: 0.9983 - precision: 0.6337 - recall: 0.1939 - auc: 0.7688 - val_loss: 0.0054 - val_tp: 25.0000 - val_fp: 7.0000 - val_tn: 45489.0000 - val_fn: 48.0000 - val_accuracy: 0.9988 - val_precision: 0.7812 - val_recall: 0.3425 - val_auc: 0.9039
Epoch 2/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0073 - tp: 147.0000 - fp: 30.0000 - tn: 181916.0000 - fn: 183.0000 - accuracy: 0.9988 - precision: 0.8305 - recall: 0.4455 - auc: 0.8670 - val_loss: 0.0046 - val_tp: 40.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 33.0000 - val_accuracy: 0.9991 - val_precision: 0.8163 - val_recall: 0.5479 - val_auc: 0.9039
Epoch 3/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0062 - tp: 170.0000 - fp: 25.0000 - tn: 181921.0000 - fn: 160.0000 - accuracy: 0.9990 - precision: 0.8718 - recall: 0.5152 - auc: 0.8925 - val_loss: 0.0043 - val_tp: 49.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 24.0000 - val_accuracy: 0.9993 - val_precision: 0.8305 - val_recall: 0.6712 - val_auc: 0.9040
Epoch 4/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0062 - tp: 175.0000 - fp: 29.0000 - tn: 181917.0000 - fn: 155.0000 - accuracy: 0.9990 - precision: 0.8578 - recall: 0.5303 - auc: 0.8963 - val_loss: 0.0040 - val_tp: 49.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 24.0000 - val_accuracy: 0.9993 - val_precision: 0.8305 - val_recall: 0.6712 - val_auc: 0.9039
Epoch 5/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0056 - tp: 175.0000 - fp: 28.0000 - tn: 181918.0000 - fn: 155.0000 - accuracy: 0.9990 - precision: 0.8621 - recall: 0.5303 - auc: 0.9013 - val_loss: 0.0039 - val_tp: 53.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 20.0000 - val_accuracy: 0.9993 - val_precision: 0.8413 - val_recall: 0.7260 - val_auc: 0.9039
Epoch 6/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0050 - tp: 198.0000 - fp: 35.0000 - tn: 181911.0000 - fn: 132.0000 - accuracy: 0.9991 - precision: 0.8498 - recall: 0.6000 - auc: 0.9138 - val_loss: 0.0037 - val_tp: 53.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 20.0000 - val_accuracy: 0.9993 - val_precision: 0.8413 - val_recall: 0.7260 - val_auc: 0.9039
Epoch 7/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0052 - tp: 180.0000 - fp: 26.0000 - tn: 181920.0000 - fn: 150.0000 - accuracy: 0.9990 - precision: 0.8738 - recall: 0.5455 - auc: 0.9047 - val_loss: 0.0036 - val_tp: 56.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 17.0000 - val_accuracy: 0.9994 - val_precision: 0.8485 - val_recall: 0.7671 - val_auc: 0.9039
Epoch 8/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0050 - tp: 197.0000 - fp: 33.0000 - tn: 181913.0000 - fn: 133.0000 - accuracy: 0.9991 - precision: 0.8565 - recall: 0.5970 - auc: 0.9123 - val_loss: 0.0035 - val_tp: 55.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 18.0000 - val_accuracy: 0.9994 - val_precision: 0.8462 - val_recall: 0.7534 - val_auc: 0.9107
Epoch 9/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0052 - tp: 183.0000 - fp: 33.0000 - tn: 181913.0000 - fn: 147.0000 - accuracy: 0.9990 - precision: 0.8472 - recall: 0.5545 - auc: 0.8987 - val_loss: 0.0034 - val_tp: 55.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 18.0000 - val_accuracy: 0.9994 - val_precision: 0.8462 - val_recall: 0.7534 - val_auc: 0.9244
Epoch 10/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0049 - tp: 190.0000 - fp: 34.0000 - tn: 181912.0000 - fn: 140.0000 - accuracy: 0.9990 - precision: 0.8482 - recall: 0.5758 - auc: 0.9124 - val_loss: 0.0033 - val_tp: 56.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 17.0000 - val_accuracy: 0.9994 - val_precision: 0.8485 - val_recall: 0.7671 - val_auc: 0.9243
Epoch 11/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0050 - tp: 190.0000 - fp: 33.0000 - tn: 181913.0000 - fn: 140.0000 - accuracy: 0.9991 - precision: 0.8520 - recall: 0.5758 - auc: 0.9002 - val_loss: 0.0032 - val_tp: 56.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 17.0000 - val_accuracy: 0.9994 - val_precision: 0.8485 - val_recall: 0.7671 - val_auc: 0.9243
Epoch 12/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0047 - tp: 198.0000 - fp: 32.0000 - tn: 181914.0000 - fn: 132.0000 - accuracy: 0.9991 - precision: 0.8609 - recall: 0.6000 - auc: 0.9093 - val_loss: 0.0031 - val_tp: 56.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 17.0000 - val_accuracy: 0.9994 - val_precision: 0.8615 - val_recall: 0.7671 - val_auc: 0.9244
Epoch 13/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0045 - tp: 191.0000 - fp: 38.0000 - tn: 181908.0000 - fn: 139.0000 - accuracy: 0.9990 - precision: 0.8341 - recall: 0.5788 - auc: 0.9111 - val_loss: 0.0031 - val_tp: 56.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 17.0000 - val_accuracy: 0.9994 - val_precision: 0.8615 - val_recall: 0.7671 - val_auc: 0.9243
Epoch 14/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0042 - tp: 201.0000 - fp: 29.0000 - tn: 181917.0000 - fn: 129.0000 - accuracy: 0.9991 - precision: 0.8739 - recall: 0.6091 - auc: 0.9248 - val_loss: 0.0030 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9243
Epoch 15/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0043 - tp: 199.0000 - fp: 33.0000 - tn: 181913.0000 - fn: 131.0000 - accuracy: 0.9991 - precision: 0.8578 - recall: 0.6030 - auc: 0.9293 - val_loss: 0.0032 - val_tp: 58.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8529 - val_recall: 0.7945 - val_auc: 0.9243
Epoch 16/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0044 - tp: 209.0000 - fp: 29.0000 - tn: 181917.0000 - fn: 121.0000 - accuracy: 0.9992 - precision: 0.8782 - recall: 0.6333 - auc: 0.9158 - val_loss: 0.0030 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9244
Epoch 17/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0041 - tp: 216.0000 - fp: 27.0000 - tn: 181919.0000 - fn: 114.0000 - accuracy: 0.9992 - precision: 0.8889 - recall: 0.6545 - auc: 0.9233 - val_loss: 0.0030 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9243
Epoch 18/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0041 - tp: 204.0000 - fp: 30.0000 - tn: 181916.0000 - fn: 126.0000 - accuracy: 0.9991 - precision: 0.8718 - recall: 0.6182 - auc: 0.9309 - val_loss: 0.0030 - val_tp: 58.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8529 - val_recall: 0.7945 - val_auc: 0.9243
Epoch 19/100
172032/182276 [===========================>..] - ETA: 0s - loss: 0.0042 - tp: 179.0000 - fp: 26.0000 - tn: 171698.0000 - fn: 129.0000 - accuracy: 0.9991 - precision: 0.8732 - recall: 0.5812 - auc: 0.9275Restoring model weights from the end of the best epoch.
182276/182276 [==============================] - 1s 3us/sample - loss: 0.0042 - tp: 193.0000 - fp: 28.0000 - tn: 181918.0000 - fn: 137.0000 - accuracy: 0.9991 - precision: 0.8733 - recall: 0.5848 - auc: 0.9278 - val_loss: 0.0031 - val_tp: 58.0000 - val_fp: 11.0000 - val_tn: 45485.0000 - val_fn: 15.0000 - val_accuracy: 0.9994 - val_precision: 0.8406 - val_recall: 0.7945 - val_auc: 0.9243
Epoch 00019: early stopping

Check triaining history

In this section, you will produce plots of your model's accuracy and loss on the training and validation set. These are useful to check for overfitting, which you can learn more about in this tutorial.

Additionally, you can produce these plots for any of the metrics you created above. False negatives are included as an example.

def plot_metrics(history):
  metrics =  ['loss', 'auc', 'precision', 'recall']
  for n, metric in enumerate(metrics):
    name = metric.replace("_"," ").capitalize()
    plt.subplot(2,2,n+1)
    plt.plot(history.epoch,  history.history[metric], color=colors[0], label='Train')
    plt.plot(history.epoch, history.history['val_'+metric],
             color=colors[0], linestyle="--", label='Val')
    plt.xlabel('Epoch')
    plt.ylabel(name)
    if metric == 'loss':
      plt.ylim([0, plt.ylim()[1]])
    elif metric == 'auc':
      plt.ylim([0.8,1])
    else:
      plt.ylim([0,1])

    plt.legend()

plot_metrics(baseline_history)

Evaluate metrics

You can use a confusion matrix to summarize the actual vs. predicted labels where the X axis is the predicted label and the Y axis is the actual label.

train_predictions_baseline = model.predict(train_features, batch_size=BATCH_SIZE)
test_predictions_baseline = model.predict(test_features, batch_size=BATCH_SIZE)

def plot_cm(labels, predictions, p=0.5):
  cm = confusion_matrix(labels, predictions > p)
  plt.figure(figsize=(5,5))
  sns.heatmap(cm, annot=True, fmt="d")
  plt.title('Confusion matrix @{:.2f}'.format(p))
  plt.ylabel('Actual label')
  plt.xlabel('Predicted label')

  print('Legitimate Transactions Detected (True Negatives): ', cm[0][0])
  print('Legitimate Transactions Incorrectly Detected (False Positives): ', cm[0][1])
  print('Fraudulent Transactions Missed (False Negatives): ', cm[1][0])
  print('Fraudulent Transactions Detected (True Positives): ', cm[1][1])
  print('Total Fraudulent Transactions: ', np.sum(cm[1]))

Evaluate your model on the test dataset and display the results for the metrics you created above.

baseline_results = model.evaluate(test_features, test_labels,
                                  batch_size=BATCH_SIZE, verbose=0)
for name, value in zip(model.metrics_names, baseline_results):
  print(name, ': ', value)
print()

plot_cm(test_labels, test_predictions_baseline)

loss :  0.004063335630105151
tp :  57.0
fp :  14.0
tn :  56859.0
fn :  32.0
accuracy :  0.9991924
precision :  0.8028169
recall :  0.64044946
auc :  0.898594

Legitimate Transactions Detected (True Negatives):  56859
Legitimate Transactions Incorrectly Detected (False Positives):  14
Fraudulent Transactions Missed (False Negatives):  32
Fraudulent Transactions Detected (True Positives):  57
Total Fraudulent Transactions:  89

If the model had predicted everything perfectly, this would be a diagonal matrix where values off the main diagonal, indicating incorrect predictions, would be zero. In this case the matrix shows that you have relatively few false positives, meaning that there were relatively few legitimate transactions that were incorrectly flagged. However, you would likely want to have even fewer false negatives despite the cost of increasing the number of false positives. This trade off may be preferable because false negatives would allow fraudulent transactions to go through, whereas false positives may cause an email to be sent to a customer to ask them to verify their card activity.

Plot the ROC

Now plot the ROC. This plot is useful because it shows, at a glance, the range of performance the model can reach just by tuning the output threshold.

def plot_roc(name, labels, predictions, **kwargs):
  fp, tp, _ = sklearn.metrics.roc_curve(labels, predictions)

  plt.plot(100*fp, 100*tp, label=name, linewidth=2, **kwargs)
  plt.xlabel('False positives [%]')
  plt.ylabel('True positives [%]')
  plt.xlim([-0.5,20])
  plt.ylim([80,100.5])
  plt.grid(True)
  ax = plt.gca()
  ax.set_aspect('equal')

plot_roc("Train Baseline", train_labels, train_predictions_baseline, color=colors[0])
plot_roc("Test Baseline", test_labels, test_predictions_baseline, color=colors[0], linestyle='--')
plt.legend(loc='lower right')

<matplotlib.legend.Legend at 0x7f5889cb7f98>

It looks like the precision is relatively high, but the recall and the area under the ROC curve (AUC) aren't as high as you might like. Classifiers often face challenges when trying to maximize both precision and recall, which is especially true when working with imbalanced datasets. It is important to consider the costs of different types of errors in the context of the problem you care about. In this example, a false negative (a fraudulent transaction is missed) may have a financial cost, while a false positive (a transaction is incorrectly flagged as fraudulent) may decrease user happiness.

Class weights

Calculate class weights

The goal is to identify fradulent transactions, but you don't have very many of those positive samples to work with, so you would want to have the classifier heavily weight the few examples that are available. You can do this by passing Keras weights for each class through a parameter. These will cause the model to "pay more attention" to examples from an under-represented class.

# Scaling by total/2 helps keep the loss to a similar magnitude.
# The sum of the weights of all examples stays the same.
weight_for_0 = (1 / neg)*(total)/2.0 
weight_for_1 = (1 / pos)*(total)/2.0

class_weight = {0: weight_for_0, 1: weight_for_1}

print('Weight for class 0: {:.2f}'.format(weight_for_0))
print('Weight for class 1: {:.2f}'.format(weight_for_1))

Weight for class 0: 0.50
Weight for class 1: 289.44

Train a model with class weights

Now try re-training and evaluating the model with class weights to see how that affects the predictions.

weighted_model = make_model()
weighted_model.load_weights(initial_weights)

weighted_history = weighted_model.fit(
    train_features,
    train_labels,
    batch_size=BATCH_SIZE,
    epochs=EPOCHS,
    callbacks = [early_stopping],
    validation_data=(val_features, val_labels),
    # The class weights go here
    class_weight=class_weight) 

Train on 182276 samples, validate on 45569 samples
Epoch 1/100
182276/182276 [==============================] - 3s 15us/sample - loss: 2.3254 - tp: 65.0000 - fp: 100.0000 - tn: 181846.0000 - fn: 265.0000 - accuracy: 0.9980 - precision: 0.3939 - recall: 0.1970 - auc: 0.7660 - val_loss: 0.9735 - val_tp: 34.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 39.0000 - val_accuracy: 0.9989 - val_precision: 0.7907 - val_recall: 0.4658 - val_auc: 0.9013
Epoch 2/100
182276/182276 [==============================] - 1s 3us/sample - loss: 1.1321 - tp: 187.0000 - fp: 160.0000 - tn: 181786.0000 - fn: 143.0000 - accuracy: 0.9983 - precision: 0.5389 - recall: 0.5667 - auc: 0.8754 - val_loss: 0.6411 - val_tp: 58.0000 - val_fp: 13.0000 - val_tn: 45483.0000 - val_fn: 15.0000 - val_accuracy: 0.9994 - val_precision: 0.8169 - val_recall: 0.7945 - val_auc: 0.9030
Epoch 3/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.8921 - tp: 217.0000 - fp: 284.0000 - tn: 181662.0000 - fn: 113.0000 - accuracy: 0.9978 - precision: 0.4331 - recall: 0.6576 - auc: 0.8955 - val_loss: 0.5363 - val_tp: 59.0000 - val_fp: 14.0000 - val_tn: 45482.0000 - val_fn: 14.0000 - val_accuracy: 0.9994 - val_precision: 0.8082 - val_recall: 0.8082 - val_auc: 0.9156
Epoch 4/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.7563 - tp: 233.0000 - fp: 534.0000 - tn: 181412.0000 - fn: 97.0000 - accuracy: 0.9965 - precision: 0.3038 - recall: 0.7061 - auc: 0.9007 - val_loss: 0.4570 - val_tp: 59.0000 - val_fp: 19.0000 - val_tn: 45477.0000 - val_fn: 14.0000 - val_accuracy: 0.9993 - val_precision: 0.7564 - val_recall: 0.8082 - val_auc: 0.9422
Epoch 5/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.6091 - tp: 242.0000 - fp: 1025.0000 - tn: 180921.0000 - fn: 88.0000 - accuracy: 0.9939 - precision: 0.1910 - recall: 0.7333 - auc: 0.9131 - val_loss: 0.3867 - val_tp: 59.0000 - val_fp: 37.0000 - val_tn: 45459.0000 - val_fn: 14.0000 - val_accuracy: 0.9989 - val_precision: 0.6146 - val_recall: 0.8082 - val_auc: 0.9409
Epoch 6/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.5879 - tp: 250.0000 - fp: 1719.0000 - tn: 180227.0000 - fn: 80.0000 - accuracy: 0.9901 - precision: 0.1270 - recall: 0.7576 - auc: 0.9134 - val_loss: 0.3373 - val_tp: 59.0000 - val_fp: 67.0000 - val_tn: 45429.0000 - val_fn: 14.0000 - val_accuracy: 0.9982 - val_precision: 0.4683 - val_recall: 0.8082 - val_auc: 0.9486
Epoch 7/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.4981 - tp: 266.0000 - fp: 2498.0000 - tn: 179448.0000 - fn: 64.0000 - accuracy: 0.9859 - precision: 0.0962 - recall: 0.8061 - auc: 0.9271 - val_loss: 0.2981 - val_tp: 61.0000 - val_fp: 173.0000 - val_tn: 45323.0000 - val_fn: 12.0000 - val_accuracy: 0.9959 - val_precision: 0.2607 - val_recall: 0.8356 - val_auc: 0.9554
Epoch 8/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.4313 - tp: 265.0000 - fp: 3276.0000 - tn: 178670.0000 - fn: 65.0000 - accuracy: 0.9817 - precision: 0.0748 - recall: 0.8030 - auc: 0.9344 - val_loss: 0.2715 - val_tp: 62.0000 - val_fp: 297.0000 - val_tn: 45199.0000 - val_fn: 11.0000 - val_accuracy: 0.9932 - val_precision: 0.1727 - val_recall: 0.8493 - val_auc: 0.9561
Epoch 9/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.3583 - tp: 282.0000 - fp: 3730.0000 - tn: 178216.0000 - fn: 48.0000 - accuracy: 0.9793 - precision: 0.0703 - recall: 0.8545 - auc: 0.9421 - val_loss: 0.2545 - val_tp: 62.0000 - val_fp: 388.0000 - val_tn: 45108.0000 - val_fn: 11.0000 - val_accuracy: 0.9912 - val_precision: 0.1378 - val_recall: 0.8493 - val_auc: 0.9618
Epoch 10/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.3509 - tp: 276.0000 - fp: 4299.0000 - tn: 177647.0000 - fn: 54.0000 - accuracy: 0.9761 - precision: 0.0603 - recall: 0.8364 - auc: 0.9534 - val_loss: 0.2392 - val_tp: 62.0000 - val_fp: 450.0000 - val_tn: 45046.0000 - val_fn: 11.0000 - val_accuracy: 0.9899 - val_precision: 0.1211 - val_recall: 0.8493 - val_auc: 0.9674
Epoch 11/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.3366 - tp: 282.0000 - fp: 4784.0000 - tn: 177162.0000 - fn: 48.0000 - accuracy: 0.9735 - precision: 0.0557 - recall: 0.8545 - auc: 0.9494 - val_loss: 0.2295 - val_tp: 63.0000 - val_fp: 520.0000 - val_tn: 44976.0000 - val_fn: 10.0000 - val_accuracy: 0.9884 - val_precision: 0.1081 - val_recall: 0.8630 - val_auc: 0.9673
Epoch 12/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.3953 - tp: 275.0000 - fp: 5054.0000 - tn: 176892.0000 - fn: 55.0000 - accuracy: 0.9720 - precision: 0.0516 - recall: 0.8333 - auc: 0.9344 - val_loss: 0.2199 - val_tp: 65.0000 - val_fp: 599.0000 - val_tn: 44897.0000 - val_fn: 8.0000 - val_accuracy: 0.9867 - val_precision: 0.0979 - val_recall: 0.8904 - val_auc: 0.9677
Epoch 13/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.3565 - tp: 284.0000 - fp: 5466.0000 - tn: 176480.0000 - fn: 46.0000 - accuracy: 0.9698 - precision: 0.0494 - recall: 0.8606 - auc: 0.9411 - val_loss: 0.2122 - val_tp: 65.0000 - val_fp: 641.0000 - val_tn: 44855.0000 - val_fn: 8.0000 - val_accuracy: 0.9858 - val_precision: 0.0921 - val_recall: 0.8904 - val_auc: 0.9719
Epoch 14/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.3041 - tp: 288.0000 - fp: 5421.0000 - tn: 176525.0000 - fn: 42.0000 - accuracy: 0.9700 - precision: 0.0504 - recall: 0.8727 - auc: 0.9516 - val_loss: 0.2070 - val_tp: 65.0000 - val_fp: 655.0000 - val_tn: 44841.0000 - val_fn: 8.0000 - val_accuracy: 0.9855 - val_precision: 0.0903 - val_recall: 0.8904 - val_auc: 0.9731
Epoch 15/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.3052 - tp: 283.0000 - fp: 5520.0000 - tn: 176426.0000 - fn: 47.0000 - accuracy: 0.9695 - precision: 0.0488 - recall: 0.8576 - auc: 0.9538 - val_loss: 0.1995 - val_tp: 65.0000 - val_fp: 704.0000 - val_tn: 44792.0000 - val_fn: 8.0000 - val_accuracy: 0.9844 - val_precision: 0.0845 - val_recall: 0.8904 - val_auc: 0.9752
Epoch 16/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2664 - tp: 294.0000 - fp: 5772.0000 - tn: 176174.0000 - fn: 36.0000 - accuracy: 0.9681 - precision: 0.0485 - recall: 0.8909 - auc: 0.9611 - val_loss: 0.1954 - val_tp: 65.0000 - val_fp: 708.0000 - val_tn: 44788.0000 - val_fn: 8.0000 - val_accuracy: 0.9843 - val_precision: 0.0841 - val_recall: 0.8904 - val_auc: 0.9766
Epoch 17/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.3308 - tp: 287.0000 - fp: 5670.0000 - tn: 176276.0000 - fn: 43.0000 - accuracy: 0.9687 - precision: 0.0482 - recall: 0.8697 - auc: 0.9422 - val_loss: 0.1933 - val_tp: 65.0000 - val_fp: 683.0000 - val_tn: 44813.0000 - val_fn: 8.0000 - val_accuracy: 0.9848 - val_precision: 0.0869 - val_recall: 0.8904 - val_auc: 0.9770
Epoch 18/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2879 - tp: 289.0000 - fp: 5571.0000 - tn: 176375.0000 - fn: 41.0000 - accuracy: 0.9692 - precision: 0.0493 - recall: 0.8758 - auc: 0.9560 - val_loss: 0.1894 - val_tp: 65.0000 - val_fp: 681.0000 - val_tn: 44815.0000 - val_fn: 8.0000 - val_accuracy: 0.9849 - val_precision: 0.0871 - val_recall: 0.8904 - val_auc: 0.9775
Epoch 19/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2861 - tp: 290.0000 - fp: 5674.0000 - tn: 176272.0000 - fn: 40.0000 - accuracy: 0.9687 - precision: 0.0486 - recall: 0.8788 - auc: 0.9567 - val_loss: 0.1854 - val_tp: 65.0000 - val_fp: 709.0000 - val_tn: 44787.0000 - val_fn: 8.0000 - val_accuracy: 0.9843 - val_precision: 0.0840 - val_recall: 0.8904 - val_auc: 0.9783
Epoch 20/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2619 - tp: 295.0000 - fp: 5809.0000 - tn: 176137.0000 - fn: 35.0000 - accuracy: 0.9679 - precision: 0.0483 - recall: 0.8939 - auc: 0.9583 - val_loss: 0.1817 - val_tp: 65.0000 - val_fp: 709.0000 - val_tn: 44787.0000 - val_fn: 8.0000 - val_accuracy: 0.9843 - val_precision: 0.0840 - val_recall: 0.8904 - val_auc: 0.9792
Epoch 21/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2957 - tp: 288.0000 - fp: 5767.0000 - tn: 176179.0000 - fn: 42.0000 - accuracy: 0.9681 - precision: 0.0476 - recall: 0.8727 - auc: 0.9508 - val_loss: 0.1787 - val_tp: 65.0000 - val_fp: 711.0000 - val_tn: 44785.0000 - val_fn: 8.0000 - val_accuracy: 0.9842 - val_precision: 0.0838 - val_recall: 0.8904 - val_auc: 0.9797
Epoch 22/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2867 - tp: 294.0000 - fp: 5761.0000 - tn: 176185.0000 - fn: 36.0000 - accuracy: 0.9682 - precision: 0.0486 - recall: 0.8909 - auc: 0.9526 - val_loss: 0.1771 - val_tp: 65.0000 - val_fp: 710.0000 - val_tn: 44786.0000 - val_fn: 8.0000 - val_accuracy: 0.9842 - val_precision: 0.0839 - val_recall: 0.8904 - val_auc: 0.9799
Epoch 23/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2946 - tp: 289.0000 - fp: 5887.0000 - tn: 176059.0000 - fn: 41.0000 - accuracy: 0.9675 - precision: 0.0468 - recall: 0.8758 - auc: 0.9550 - val_loss: 0.1734 - val_tp: 65.0000 - val_fp: 750.0000 - val_tn: 44746.0000 - val_fn: 8.0000 - val_accuracy: 0.9834 - val_precision: 0.0798 - val_recall: 0.8904 - val_auc: 0.9800
Epoch 24/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2968 - tp: 286.0000 - fp: 6210.0000 - tn: 175736.0000 - fn: 44.0000 - accuracy: 0.9657 - precision: 0.0440 - recall: 0.8667 - auc: 0.9545 - val_loss: 0.1734 - val_tp: 65.0000 - val_fp: 791.0000 - val_tn: 44705.0000 - val_fn: 8.0000 - val_accuracy: 0.9825 - val_precision: 0.0759 - val_recall: 0.8904 - val_auc: 0.9800
Epoch 25/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2658 - tp: 291.0000 - fp: 6092.0000 - tn: 175854.0000 - fn: 39.0000 - accuracy: 0.9664 - precision: 0.0456 - recall: 0.8818 - auc: 0.9609 - val_loss: 0.1724 - val_tp: 65.0000 - val_fp: 783.0000 - val_tn: 44713.0000 - val_fn: 8.0000 - val_accuracy: 0.9826 - val_precision: 0.0767 - val_recall: 0.8904 - val_auc: 0.9803
Epoch 26/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2755 - tp: 294.0000 - fp: 6039.0000 - tn: 175907.0000 - fn: 36.0000 - accuracy: 0.9667 - precision: 0.0464 - recall: 0.8909 - auc: 0.9562 - val_loss: 0.1703 - val_tp: 65.0000 - val_fp: 747.0000 - val_tn: 44749.0000 - val_fn: 8.0000 - val_accuracy: 0.9834 - val_precision: 0.0800 - val_recall: 0.8904 - val_auc: 0.9810
Epoch 27/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2453 - tp: 293.0000 - fp: 5812.0000 - tn: 176134.0000 - fn: 37.0000 - accuracy: 0.9679 - precision: 0.0480 - recall: 0.8879 - auc: 0.9639 - val_loss: 0.1689 - val_tp: 64.0000 - val_fp: 749.0000 - val_tn: 44747.0000 - val_fn: 9.0000 - val_accuracy: 0.9834 - val_precision: 0.0787 - val_recall: 0.8767 - val_auc: 0.9814
Epoch 28/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2658 - tp: 291.0000 - fp: 5659.0000 - tn: 176287.0000 - fn: 39.0000 - accuracy: 0.9687 - precision: 0.0489 - recall: 0.8818 - auc: 0.9614 - val_loss: 0.1672 - val_tp: 65.0000 - val_fp: 770.0000 - val_tn: 44726.0000 - val_fn: 8.0000 - val_accuracy: 0.9829 - val_precision: 0.0778 - val_recall: 0.8904 - val_auc: 0.9816
Epoch 29/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2779 - tp: 286.0000 - fp: 5922.0000 - tn: 176024.0000 - fn: 44.0000 - accuracy: 0.9673 - precision: 0.0461 - recall: 0.8667 - auc: 0.9584 - val_loss: 0.1656 - val_tp: 65.0000 - val_fp: 774.0000 - val_tn: 44722.0000 - val_fn: 8.0000 - val_accuracy: 0.9828 - val_precision: 0.0775 - val_recall: 0.8904 - val_auc: 0.9816
Epoch 30/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2761 - tp: 289.0000 - fp: 6097.0000 - tn: 175849.0000 - fn: 41.0000 - accuracy: 0.9663 - precision: 0.0453 - recall: 0.8758 - auc: 0.9566 - val_loss: 0.1652 - val_tp: 65.0000 - val_fp: 781.0000 - val_tn: 44715.0000 - val_fn: 8.0000 - val_accuracy: 0.9827 - val_precision: 0.0768 - val_recall: 0.8904 - val_auc: 0.9817
Epoch 31/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2477 - tp: 295.0000 - fp: 5934.0000 - tn: 176012.0000 - fn: 35.0000 - accuracy: 0.9673 - precision: 0.0474 - recall: 0.8939 - auc: 0.9603 - val_loss: 0.1647 - val_tp: 64.0000 - val_fp: 733.0000 - val_tn: 44763.0000 - val_fn: 9.0000 - val_accuracy: 0.9837 - val_precision: 0.0803 - val_recall: 0.8767 - val_auc: 0.9820
Epoch 32/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2510 - tp: 294.0000 - fp: 5458.0000 - tn: 176488.0000 - fn: 36.0000 - accuracy: 0.9699 - precision: 0.0511 - recall: 0.8909 - auc: 0.9605 - val_loss: 0.1637 - val_tp: 64.0000 - val_fp: 725.0000 - val_tn: 44771.0000 - val_fn: 9.0000 - val_accuracy: 0.9839 - val_precision: 0.0811 - val_recall: 0.8767 - val_auc: 0.9823
Epoch 33/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2445 - tp: 295.0000 - fp: 5306.0000 - tn: 176640.0000 - fn: 35.0000 - accuracy: 0.9707 - precision: 0.0527 - recall: 0.8939 - auc: 0.9605 - val_loss: 0.1644 - val_tp: 64.0000 - val_fp: 681.0000 - val_tn: 44815.0000 - val_fn: 9.0000 - val_accuracy: 0.9849 - val_precision: 0.0859 - val_recall: 0.8767 - val_auc: 0.9824
Epoch 34/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2505 - tp: 289.0000 - fp: 5184.0000 - tn: 176762.0000 - fn: 41.0000 - accuracy: 0.9713 - precision: 0.0528 - recall: 0.8758 - auc: 0.9664 - val_loss: 0.1620 - val_tp: 64.0000 - val_fp: 684.0000 - val_tn: 44812.0000 - val_fn: 9.0000 - val_accuracy: 0.9848 - val_precision: 0.0856 - val_recall: 0.8767 - val_auc: 0.9827
Epoch 35/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2521 - tp: 296.0000 - fp: 5146.0000 - tn: 176800.0000 - fn: 34.0000 - accuracy: 0.9716 - precision: 0.0544 - recall: 0.8970 - auc: 0.9535 - val_loss: 0.1631 - val_tp: 64.0000 - val_fp: 610.0000 - val_tn: 44886.0000 - val_fn: 9.0000 - val_accuracy: 0.9864 - val_precision: 0.0950 - val_recall: 0.8767 - val_auc: 0.9830
Epoch 36/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2721 - tp: 287.0000 - fp: 4838.0000 - tn: 177108.0000 - fn: 43.0000 - accuracy: 0.9732 - precision: 0.0560 - recall: 0.8697 - auc: 0.9555 - val_loss: 0.1621 - val_tp: 64.0000 - val_fp: 651.0000 - val_tn: 44845.0000 - val_fn: 9.0000 - val_accuracy: 0.9855 - val_precision: 0.0895 - val_recall: 0.8767 - val_auc: 0.9831
Epoch 37/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2684 - tp: 291.0000 - fp: 4931.0000 - tn: 177015.0000 - fn: 39.0000 - accuracy: 0.9727 - precision: 0.0557 - recall: 0.8818 - auc: 0.9507 - val_loss: 0.1619 - val_tp: 64.0000 - val_fp: 668.0000 - val_tn: 44828.0000 - val_fn: 9.0000 - val_accuracy: 0.9851 - val_precision: 0.0874 - val_recall: 0.8767 - val_auc: 0.9830
Epoch 38/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.1836 - tp: 299.0000 - fp: 5330.0000 - tn: 176616.0000 - fn: 31.0000 - accuracy: 0.9706 - precision: 0.0531 - recall: 0.9061 - auc: 0.9772 - val_loss: 0.1602 - val_tp: 64.0000 - val_fp: 697.0000 - val_tn: 44799.0000 - val_fn: 9.0000 - val_accuracy: 0.9845 - val_precision: 0.0841 - val_recall: 0.8767 - val_auc: 0.9830
Epoch 39/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2514 - tp: 291.0000 - fp: 5371.0000 - tn: 176575.0000 - fn: 39.0000 - accuracy: 0.9703 - precision: 0.0514 - recall: 0.8818 - auc: 0.9590 - val_loss: 0.1604 - val_tp: 64.0000 - val_fp: 700.0000 - val_tn: 44796.0000 - val_fn: 9.0000 - val_accuracy: 0.9844 - val_precision: 0.0838 - val_recall: 0.8767 - val_auc: 0.9830
Epoch 40/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.1874 - tp: 303.0000 - fp: 5355.0000 - tn: 176591.0000 - fn: 27.0000 - accuracy: 0.9705 - precision: 0.0536 - recall: 0.9182 - auc: 0.9772 - val_loss: 0.1609 - val_tp: 64.0000 - val_fp: 667.0000 - val_tn: 44829.0000 - val_fn: 9.0000 - val_accuracy: 0.9852 - val_precision: 0.0876 - val_recall: 0.8767 - val_auc: 0.9832
Epoch 41/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2080 - tp: 303.0000 - fp: 4894.0000 - tn: 177052.0000 - fn: 27.0000 - accuracy: 0.9730 - precision: 0.0583 - recall: 0.9182 - auc: 0.9702 - val_loss: 0.1633 - val_tp: 64.0000 - val_fp: 636.0000 - val_tn: 44860.0000 - val_fn: 9.0000 - val_accuracy: 0.9858 - val_precision: 0.0914 - val_recall: 0.8767 - val_auc: 0.9834
Epoch 42/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2198 - tp: 294.0000 - fp: 4866.0000 - tn: 177080.0000 - fn: 36.0000 - accuracy: 0.9731 - precision: 0.0570 - recall: 0.8909 - auc: 0.9740 - val_loss: 0.1600 - val_tp: 64.0000 - val_fp: 643.0000 - val_tn: 44853.0000 - val_fn: 9.0000 - val_accuracy: 0.9857 - val_precision: 0.0905 - val_recall: 0.8767 - val_auc: 0.9834
Epoch 43/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2130 - tp: 297.0000 - fp: 4916.0000 - tn: 177030.0000 - fn: 33.0000 - accuracy: 0.9728 - precision: 0.0570 - recall: 0.9000 - auc: 0.9738 - val_loss: 0.1583 - val_tp: 65.0000 - val_fp: 628.0000 - val_tn: 44868.0000 - val_fn: 8.0000 - val_accuracy: 0.9860 - val_precision: 0.0938 - val_recall: 0.8904 - val_auc: 0.9837
Epoch 44/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2104 - tp: 296.0000 - fp: 4936.0000 - tn: 177010.0000 - fn: 34.0000 - accuracy: 0.9727 - precision: 0.0566 - recall: 0.8970 - auc: 0.9735 - val_loss: 0.1579 - val_tp: 65.0000 - val_fp: 669.0000 - val_tn: 44827.0000 - val_fn: 8.0000 - val_accuracy: 0.9851 - val_precision: 0.0886 - val_recall: 0.8904 - val_auc: 0.9836
Epoch 45/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2180 - tp: 294.0000 - fp: 5122.0000 - tn: 176824.0000 - fn: 36.0000 - accuracy: 0.9717 - precision: 0.0543 - recall: 0.8909 - auc: 0.9767 - val_loss: 0.1602 - val_tp: 65.0000 - val_fp: 690.0000 - val_tn: 44806.0000 - val_fn: 8.0000 - val_accuracy: 0.9847 - val_precision: 0.0861 - val_recall: 0.8904 - val_auc: 0.9834
Epoch 46/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2394 - tp: 291.0000 - fp: 5501.0000 - tn: 176445.0000 - fn: 39.0000 - accuracy: 0.9696 - precision: 0.0502 - recall: 0.8818 - auc: 0.9699 - val_loss: 0.1564 - val_tp: 65.0000 - val_fp: 759.0000 - val_tn: 44737.0000 - val_fn: 8.0000 - val_accuracy: 0.9832 - val_precision: 0.0789 - val_recall: 0.8904 - val_auc: 0.9835
Epoch 47/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2011 - tp: 297.0000 - fp: 5400.0000 - tn: 176546.0000 - fn: 33.0000 - accuracy: 0.9702 - precision: 0.0521 - recall: 0.9000 - auc: 0.9768 - val_loss: 0.1605 - val_tp: 65.0000 - val_fp: 754.0000 - val_tn: 44742.0000 - val_fn: 8.0000 - val_accuracy: 0.9833 - val_precision: 0.0794 - val_recall: 0.8904 - val_auc: 0.9833
Epoch 48/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.1938 - tp: 301.0000 - fp: 5420.0000 - tn: 176526.0000 - fn: 29.0000 - accuracy: 0.9701 - precision: 0.0526 - recall: 0.9121 - auc: 0.9774 - val_loss: 0.1628 - val_tp: 65.0000 - val_fp: 732.0000 - val_tn: 44764.0000 - val_fn: 8.0000 - val_accuracy: 0.9838 - val_precision: 0.0816 - val_recall: 0.8904 - val_auc: 0.9833
Epoch 49/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.1819 - tp: 301.0000 - fp: 5254.0000 - tn: 176692.0000 - fn: 29.0000 - accuracy: 0.9710 - precision: 0.0542 - recall: 0.9121 - auc: 0.9812 - val_loss: 0.1632 - val_tp: 65.0000 - val_fp: 719.0000 - val_tn: 44777.0000 - val_fn: 8.0000 - val_accuracy: 0.9840 - val_precision: 0.0829 - val_recall: 0.8904 - val_auc: 0.9835
Epoch 50/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2211 - tp: 295.0000 - fp: 5107.0000 - tn: 176839.0000 - fn: 35.0000 - accuracy: 0.9718 - precision: 0.0546 - recall: 0.8939 - auc: 0.9750 - val_loss: 0.1591 - val_tp: 65.0000 - val_fp: 696.0000 - val_tn: 44800.0000 - val_fn: 8.0000 - val_accuracy: 0.9846 - val_precision: 0.0854 - val_recall: 0.8904 - val_auc: 0.9838
Epoch 51/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2527 - tp: 293.0000 - fp: 5253.0000 - tn: 176693.0000 - fn: 37.0000 - accuracy: 0.9710 - precision: 0.0528 - recall: 0.8879 - auc: 0.9664 - val_loss: 0.1572 - val_tp: 65.0000 - val_fp: 701.0000 - val_tn: 44795.0000 - val_fn: 8.0000 - val_accuracy: 0.9844 - val_precision: 0.0849 - val_recall: 0.8904 - val_auc: 0.9840
Epoch 52/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2278 - tp: 290.0000 - fp: 5515.0000 - tn: 176431.0000 - fn: 40.0000 - accuracy: 0.9695 - precision: 0.0500 - recall: 0.8788 - auc: 0.9754 - val_loss: 0.1580 - val_tp: 65.0000 - val_fp: 730.0000 - val_tn: 44766.0000 - val_fn: 8.0000 - val_accuracy: 0.9838 - val_precision: 0.0818 - val_recall: 0.8904 - val_auc: 0.9839
Epoch 53/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.1858 - tp: 302.0000 - fp: 5301.0000 - tn: 176645.0000 - fn: 28.0000 - accuracy: 0.9708 - precision: 0.0539 - recall: 0.9152 - auc: 0.9796 - val_loss: 0.1616 - val_tp: 65.0000 - val_fp: 698.0000 - val_tn: 44798.0000 - val_fn: 8.0000 - val_accuracy: 0.9845 - val_precision: 0.0852 - val_recall: 0.8904 - val_auc: 0.9838
Epoch 54/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.1983 - tp: 297.0000 - fp: 4998.0000 - tn: 176948.0000 - fn: 33.0000 - accuracy: 0.9724 - precision: 0.0561 - recall: 0.9000 - auc: 0.9768 - val_loss: 0.1655 - val_tp: 65.0000 - val_fp: 625.0000 - val_tn: 44871.0000 - val_fn: 8.0000 - val_accuracy: 0.9861 - val_precision: 0.0942 - val_recall: 0.8904 - val_auc: 0.9836
Epoch 55/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2221 - tp: 296.0000 - fp: 4927.0000 - tn: 177019.0000 - fn: 34.0000 - accuracy: 0.9728 - precision: 0.0567 - recall: 0.8970 - auc: 0.9738 - val_loss: 0.1640 - val_tp: 65.0000 - val_fp: 597.0000 - val_tn: 44899.0000 - val_fn: 8.0000 - val_accuracy: 0.9867 - val_precision: 0.0982 - val_recall: 0.8904 - val_auc: 0.9839
Epoch 56/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.1865 - tp: 301.0000 - fp: 4637.0000 - tn: 177309.0000 - fn: 29.0000 - accuracy: 0.9744 - precision: 0.0610 - recall: 0.9121 - auc: 0.9788 - val_loss: 0.1625 - val_tp: 65.0000 - val_fp: 574.0000 - val_tn: 44922.0000 - val_fn: 8.0000 - val_accuracy: 0.9872 - val_precision: 0.1017 - val_recall: 0.8904 - val_auc: 0.9842
Epoch 57/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2066 - tp: 294.0000 - fp: 4826.0000 - tn: 177120.0000 - fn: 36.0000 - accuracy: 0.9733 - precision: 0.0574 - recall: 0.8909 - auc: 0.9771 - val_loss: 0.1635 - val_tp: 64.0000 - val_fp: 553.0000 - val_tn: 44943.0000 - val_fn: 9.0000 - val_accuracy: 0.9877 - val_precision: 0.1037 - val_recall: 0.8767 - val_auc: 0.9844
Epoch 58/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.1880 - tp: 298.0000 - fp: 4652.0000 - tn: 177294.0000 - fn: 32.0000 - accuracy: 0.9743 - precision: 0.0602 - recall: 0.9030 - auc: 0.9802 - val_loss: 0.1662 - val_tp: 65.0000 - val_fp: 624.0000 - val_tn: 44872.0000 - val_fn: 8.0000 - val_accuracy: 0.9861 - val_precision: 0.0943 - val_recall: 0.8904 - val_auc: 0.9840
Epoch 59/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2187 - tp: 299.0000 - fp: 4576.0000 - tn: 177370.0000 - fn: 31.0000 - accuracy: 0.9747 - precision: 0.0613 - recall: 0.9061 - auc: 0.9731 - val_loss: 0.1686 - val_tp: 65.0000 - val_fp: 600.0000 - val_tn: 44896.0000 - val_fn: 8.0000 - val_accuracy: 0.9867 - val_precision: 0.0977 - val_recall: 0.8904 - val_auc: 0.9837
Epoch 60/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.2042 - tp: 293.0000 - fp: 4777.0000 - tn: 177169.0000 - fn: 37.0000 - accuracy: 0.9736 - precision: 0.0578 - recall: 0.8879 - auc: 0.9772 - val_loss: 0.1688 - val_tp: 65.0000 - val_fp: 592.0000 - val_tn: 44904.0000 - val_fn: 8.0000 - val_accuracy: 0.9868 - val_precision: 0.0989 - val_recall: 0.8904 - val_auc: 0.9834
Epoch 61/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.1883 - tp: 303.0000 - fp: 4599.0000 - tn: 177347.0000 - fn: 27.0000 - accuracy: 0.9746 - precision: 0.0618 - recall: 0.9182 - auc: 0.9792 - val_loss: 0.1721 - val_tp: 65.0000 - val_fp: 557.0000 - val_tn: 44939.0000 - val_fn: 8.0000 - val_accuracy: 0.9876 - val_precision: 0.1045 - val_recall: 0.8904 - val_auc: 0.9833
Epoch 62/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.1886 - tp: 299.0000 - fp: 4351.0000 - tn: 177595.0000 - fn: 31.0000 - accuracy: 0.9760 - precision: 0.0643 - recall: 0.9061 - auc: 0.9779 - val_loss: 0.1721 - val_tp: 65.0000 - val_fp: 587.0000 - val_tn: 44909.0000 - val_fn: 8.0000 - val_accuracy: 0.9869 - val_precision: 0.0997 - val_recall: 0.8904 - val_auc: 0.9832
Epoch 63/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.1953 - tp: 300.0000 - fp: 4536.0000 - tn: 177410.0000 - fn: 30.0000 - accuracy: 0.9750 - precision: 0.0620 - recall: 0.9091 - auc: 0.9775 - val_loss: 0.1753 - val_tp: 64.0000 - val_fp: 559.0000 - val_tn: 44937.0000 - val_fn: 9.0000 - val_accuracy: 0.9875 - val_precision: 0.1027 - val_recall: 0.8767 - val_auc: 0.9824
Epoch 64/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.1908 - tp: 300.0000 - fp: 4398.0000 - tn: 177548.0000 - fn: 30.0000 - accuracy: 0.9757 - precision: 0.0639 - recall: 0.9091 - auc: 0.9783 - val_loss: 0.1759 - val_tp: 64.0000 - val_fp: 530.0000 - val_tn: 44966.0000 - val_fn: 9.0000 - val_accuracy: 0.9882 - val_precision: 0.1077 - val_recall: 0.8767 - val_auc: 0.9818
Epoch 65/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.1901 - tp: 298.0000 - fp: 4188.0000 - tn: 177758.0000 - fn: 32.0000 - accuracy: 0.9768 - precision: 0.0664 - recall: 0.9030 - auc: 0.9799 - val_loss: 0.1700 - val_tp: 65.0000 - val_fp: 493.0000 - val_tn: 45003.0000 - val_fn: 8.0000 - val_accuracy: 0.9890 - val_precision: 0.1165 - val_recall: 0.8904 - val_auc: 0.9833
Epoch 66/100
182276/182276 [==============================] - 1s 3us/sample - loss: 0.1668 - tp: 306.0000 - fp: 4292.0000 - tn: 177654.0000 - fn: 24.0000 - accuracy: 0.9763 - precision: 0.0666 - recall: 0.9273 - auc: 0.9815 - val_loss: 0.1724 - val_tp: 65.0000 - val_fp: 523.0000 - val_tn: 44973.0000 - val_fn: 8.0000 - val_accuracy: 0.9883 - val_precision: 0.1105 - val_recall: 0.8904 - val_auc: 0.9830
Epoch 67/100
182272/182276 [============================>.] - ETA: 0s - loss: 0.1834 - tp: 298.0000 - fp: 4613.0000 - tn: 177329.0000 - fn: 32.0000 - accuracy: 0.9745 - precision: 0.0607 - recall: 0.9030 - auc: 0.9831Restoring model weights from the end of the best epoch.
182276/182276 [==============================] - 1s 3us/sample - loss: 0.1834 - tp: 298.0000 - fp: 4613.0000 - tn: 177333.0000 - fn: 32.0000 - accuracy: 0.9745 - precision: 0.0607 - recall: 0.9030 - auc: 0.9831 - val_loss: 0.1690 - val_tp: 65.0000 - val_fp: 530.0000 - val_tn: 44966.0000 - val_fn: 8.0000 - val_accuracy: 0.9882 - val_precision: 0.1092 - val_recall: 0.8904 - val_auc: 0.9835
Epoch 00067: early stopping

Check training history

plot_metrics(weighted_history)

Evaluate metrics

train_predictions_weighted = weighted_model.predict(train_features, batch_size=BATCH_SIZE)
test_predictions_weighted = weighted_model.predict(test_features, batch_size=BATCH_SIZE)

weighted_results = weighted_model.evaluate(test_features, test_labels,
                                           batch_size=BATCH_SIZE, verbose=0)
for name, value in zip(weighted_model.metrics_names, weighted_results):
  print(name, ': ', value)
print()

plot_cm(test_labels, test_predictions_weighted)

loss :  0.05434137444066443
tp :  75.0
fp :  711.0
tn :  56162.0
fn :  14.0
accuracy :  0.9872722
precision :  0.09541985
recall :  0.8426966
auc :  0.9538048

Legitimate Transactions Detected (True Negatives):  56162
Legitimate Transactions Incorrectly Detected (False Positives):  711
Fraudulent Transactions Missed (False Negatives):  14
Fraudulent Transactions Detected (True Positives):  75
Total Fraudulent Transactions:  89

Here you can see that with class weights the accuracy and precision are lower because there are more false positives, but conversely the recall and AUC are higher because the model also found more true positives. Despite having lower accuracy, this model has higher recall (and identifies more fraudulent transactions). Of course, there is a cost to both types of error (you wouldn't want to bug users by flagging too many legitimate transactions as fraudulent, either). Carefully consider the trade offs between these different types of errors for your application.

Plot the ROC

plot_roc("Train Baseline", train_labels, train_predictions_baseline, color=colors[0])
plot_roc("Test Baseline", test_labels, test_predictions_baseline, color=colors[0], linestyle='--')

plot_roc("Train Weighted", train_labels, train_predictions_weighted, color=colors[1])
plot_roc("Test Weighted", test_labels, test_predictions_weighted, color=colors[1], linestyle='--')


plt.legend(loc='lower right')

<matplotlib.legend.Legend at 0x7f587f5b5b38>

Oversampling

Oversample the minority class

A related approach would be to resample the dataset by oversampling the minority class.

pos_features = train_features[bool_train_labels]
neg_features = train_features[~bool_train_labels]

pos_labels = train_labels[bool_train_labels]
neg_labels = train_labels[~bool_train_labels]

Using NumPy

You can balance the dataset manually by choosing the right number randon indices from the positive examples:

ids = np.arange(len(pos_features))
choices = np.random.choice(ids, len(neg_features))

res_pos_features = pos_features[choices]
res_pos_labels = pos_labels[choices]

res_pos_features.shape

(181946, 29)

resampled_features = np.concatenate([res_pos_features, neg_features], axis=0)
resampled_labels = np.concatenate([res_pos_labels, neg_labels], axis=0)

order = np.arange(len(resampled_labels))
np.random.shuffle(order)
resampled_features = resampled_features[order]
resampled_labels = resampled_labels[order]

resampled_features.shape

(363892, 29)

Using tf.data

If you're using tf.data the easiest way to produce balanced examples is to start with a positive and a negative dataset, and merge them. See the tf.data guide for more examples.

BUFFER_SIZE = 100000

def make_ds(features, labels):
  ds = tf.data.Dataset.from_tensor_slices((features, labels))#.cache()
  ds = ds.shuffle(BUFFER_SIZE).repeat()
  return ds

pos_ds = make_ds(pos_features, pos_labels)
neg_ds = make_ds(neg_features, neg_labels)

Each dataset provides (feature, label) pairs:

for features, label in pos_ds.take(1):
  print("Features:\n", features.numpy())
  print()
  print("Label: ", label.numpy())

Features:
 [-5.          5.         -5.          5.         -5.         -1.26142183
 -5.          0.54056746 -5.         -5.          4.72342916 -5.
  1.3276559  -5.          0.90559755 -5.         -5.         -5.
  4.08980539  0.56494691 -2.97617527  0.71521581 -1.25031394  1.0936802
 -1.81798602  0.25177097 -5.         -3.80806863  0.98925276]

Label:  1

Merge the two together using experimental.sample_from_datasets:

resampled_ds = tf.data.experimental.sample_from_datasets([pos_ds, neg_ds], weights=[0.5, 0.5])
resampled_ds = resampled_ds.batch(BATCH_SIZE).prefetch(2)

for features, label in resampled_ds.take(1):
  print(label.numpy().mean())

0.50146484375

To use this dataset, you'll need the number of steps per epoch.

The definition of "epoch" in this case is less clear. Say it's the number of batches required to see each negative example once:

resampled_steps_per_epoch = np.ceil(2.0*neg/BATCH_SIZE)
resampled_steps_per_epoch

278.0

Train on the oversampled data

Now try training the model with the resampled data set instead of using class weights to see how these methods compare.

resampled_model = make_model()
resampled_model.load_weights(initial_weights)

# Reset the bias to zero, since this dataset is balanced.
output_layer = resampled_model.layers[-1] 
output_layer.bias.assign([0])

val_ds = tf.data.Dataset.from_tensor_slices((val_features, val_labels)).cache()
val_ds = val_ds.batch(BATCH_SIZE).prefetch(2) 

resampled_history = resampled_model.fit(
    resampled_ds,
    epochs=EPOCHS,
    steps_per_epoch=resampled_steps_per_epoch,
    callbacks = [early_stopping],
    validation_data=val_ds)

Train for 278.0 steps, validate for 23 steps
Epoch 1/100
278/278 [==============================] - 9s 32ms/step - loss: 0.4046 - tp: 231614.0000 - fp: 46918.0000 - tn: 237546.0000 - fn: 53266.0000 - accuracy: 0.8240 - precision: 0.8316 - recall: 0.8130 - auc: 0.8949 - val_loss: 0.1497 - val_tp: 62.0000 - val_fp: 697.0000 - val_tn: 44799.0000 - val_fn: 11.0000 - val_accuracy: 0.9845 - val_precision: 0.0817 - val_recall: 0.8493 - val_auc: 0.9699
Epoch 2/100
278/278 [==============================] - 7s 26ms/step - loss: 0.1827 - tp: 253830.0000 - fp: 13703.0000 - tn: 271516.0000 - fn: 30295.0000 - accuracy: 0.9227 - precision: 0.9488 - recall: 0.8934 - auc: 0.9777 - val_loss: 0.0841 - val_tp: 63.0000 - val_fp: 612.0000 - val_tn: 44884.0000 - val_fn: 10.0000 - val_accuracy: 0.9864 - val_precision: 0.0933 - val_recall: 0.8630 - val_auc: 0.9821
Epoch 3/100
278/278 [==============================] - 7s 25ms/step - loss: 0.1403 - tp: 259234.0000 - fp: 9278.0000 - tn: 275104.0000 - fn: 25728.0000 - accuracy: 0.9385 - precision: 0.9654 - recall: 0.9097 - auc: 0.9878 - val_loss: 0.0670 - val_tp: 65.0000 - val_fp: 629.0000 - val_tn: 44867.0000 - val_fn: 8.0000 - val_accuracy: 0.9860 - val_precision: 0.0937 - val_recall: 0.8904 - val_auc: 0.9847
Epoch 4/100
278/278 [==============================] - 7s 26ms/step - loss: 0.1218 - tp: 261426.0000 - fp: 8108.0000 - tn: 276737.0000 - fn: 23073.0000 - accuracy: 0.9452 - precision: 0.9699 - recall: 0.9189 - auc: 0.9913 - val_loss: 0.0577 - val_tp: 65.0000 - val_fp: 592.0000 - val_tn: 44904.0000 - val_fn: 8.0000 - val_accuracy: 0.9868 - val_precision: 0.0989 - val_recall: 0.8904 - val_auc: 0.9855
Epoch 5/100
278/278 [==============================] - 7s 25ms/step - loss: 0.1105 - tp: 262979.0000 - fp: 7427.0000 - tn: 277754.0000 - fn: 21184.0000 - accuracy: 0.9497 - precision: 0.9725 - recall: 0.9255 - auc: 0.9930 - val_loss: 0.0525 - val_tp: 65.0000 - val_fp: 575.0000 - val_tn: 44921.0000 - val_fn: 8.0000 - val_accuracy: 0.9872 - val_precision: 0.1016 - val_recall: 0.8904 - val_auc: 0.9858
Epoch 6/100
278/278 [==============================] - 7s 26ms/step - loss: 0.1028 - tp: 265226.0000 - fp: 7354.0000 - tn: 276894.0000 - fn: 19870.0000 - accuracy: 0.9522 - precision: 0.9730 - recall: 0.9303 - auc: 0.9941 - val_loss: 0.0481 - val_tp: 65.0000 - val_fp: 550.0000 - val_tn: 44946.0000 - val_fn: 8.0000 - val_accuracy: 0.9878 - val_precision: 0.1057 - val_recall: 0.8904 - val_auc: 0.9863
Epoch 7/100
278/278 [==============================] - 7s 25ms/step - loss: 0.0961 - tp: 266156.0000 - fp: 7054.0000 - tn: 277483.0000 - fn: 18651.0000 - accuracy: 0.9549 - precision: 0.9742 - recall: 0.9345 - auc: 0.9950 - val_loss: 0.0454 - val_tp: 66.0000 - val_fp: 542.0000 - val_tn: 44954.0000 - val_fn: 7.0000 - val_accuracy: 0.9880 - val_precision: 0.1086 - val_recall: 0.9041 - val_auc: 0.9865
Epoch 8/100
278/278 [==============================] - 7s 27ms/step - loss: 0.0905 - tp: 267257.0000 - fp: 7142.0000 - tn: 277351.0000 - fn: 17594.0000 - accuracy: 0.9566 - precision: 0.9740 - recall: 0.9382 - auc: 0.9955 - val_loss: 0.0430 - val_tp: 66.0000 - val_fp: 540.0000 - val_tn: 44956.0000 - val_fn: 7.0000 - val_accuracy: 0.9880 - val_precision: 0.1089 - val_recall: 0.9041 - val_auc: 0.9869
Epoch 9/100
278/278 [==============================] - 7s 25ms/step - loss: 0.0862 - tp: 268061.0000 - fp: 7158.0000 - tn: 277637.0000 - fn: 16488.0000 - accuracy: 0.9585 - precision: 0.9740 - recall: 0.9421 - auc: 0.9960 - val_loss: 0.0406 - val_tp: 66.0000 - val_fp: 537.0000 - val_tn: 44959.0000 - val_fn: 7.0000 - val_accuracy: 0.9881 - val_precision: 0.1095 - val_recall: 0.9041 - val_auc: 0.9867
Epoch 10/100
278/278 [==============================] - 7s 26ms/step - loss: 0.0815 - tp: 270283.0000 - fp: 7341.0000 - tn: 276624.0000 - fn: 15096.0000 - accuracy: 0.9606 - precision: 0.9736 - recall: 0.9471 - auc: 0.9964 - val_loss: 0.0382 - val_tp: 66.0000 - val_fp: 530.0000 - val_tn: 44966.0000 - val_fn: 7.0000 - val_accuracy: 0.9882 - val_precision: 0.1107 - val_recall: 0.9041 - val_auc: 0.9860
Epoch 11/100
278/278 [==============================] - 7s 26ms/step - loss: 0.0777 - tp: 270595.0000 - fp: 7320.0000 - tn: 276909.0000 - fn: 14520.0000 - accuracy: 0.9616 - precision: 0.9737 - recall: 0.9491 - auc: 0.9967 - val_loss: 0.0357 - val_tp: 65.0000 - val_fp: 503.0000 - val_tn: 44993.0000 - val_fn: 8.0000 - val_accuracy: 0.9888 - val_precision: 0.1144 - val_recall: 0.8904 - val_auc: 0.9805
Epoch 12/100
278/278 [==============================] - 7s 26ms/step - loss: 0.0738 - tp: 271542.0000 - fp: 7375.0000 - tn: 276296.0000 - fn: 14131.0000 - accuracy: 0.9622 - precision: 0.9736 - recall: 0.9505 - auc: 0.9970 - val_loss: 0.0336 - val_tp: 65.0000 - val_fp: 495.0000 - val_tn: 45001.0000 - val_fn: 8.0000 - val_accuracy: 0.9890 - val_precision: 0.1161 - val_recall: 0.8904 - val_auc: 0.9807
Epoch 13/100
278/278 [==============================] - 7s 27ms/step - loss: 0.0705 - tp: 273463.0000 - fp: 8624.0000 - tn: 276671.0000 - fn: 10586.0000 - accuracy: 0.9663 - precision: 0.9694 - recall: 0.9627 - auc: 0.9972 - val_loss: 0.0312 - val_tp: 65.0000 - val_fp: 470.0000 - val_tn: 45026.0000 - val_fn: 8.0000 - val_accuracy: 0.9895 - val_precision: 0.1215 - val_recall: 0.8904 - val_auc: 0.9808
Epoch 14/100
278/278 [==============================] - 7s 26ms/step - loss: 0.0683 - tp: 282785.0000 - fp: 12398.0000 - tn: 272289.0000 - fn: 1872.0000 - accuracy: 0.9749 - precision: 0.9580 - recall: 0.9934 - auc: 0.9973 - val_loss: 0.0299 - val_tp: 65.0000 - val_fp: 463.0000 - val_tn: 45033.0000 - val_fn: 8.0000 - val_accuracy: 0.9897 - val_precision: 0.1231 - val_recall: 0.8904 - val_auc: 0.9809
Epoch 15/100
278/278 [==============================] - 7s 27ms/step - loss: 0.0654 - tp: 284456.0000 - fp: 12448.0000 - tn: 271192.0000 - fn: 1248.0000 - accuracy: 0.9759 - precision: 0.9581 - recall: 0.9956 - auc: 0.9975 - val_loss: 0.0281 - val_tp: 64.0000 - val_fp: 452.0000 - val_tn: 45044.0000 - val_fn: 9.0000 - val_accuracy: 0.9899 - val_precision: 0.1240 - val_recall: 0.8767 - val_auc: 0.9808
Epoch 16/100
278/278 [==============================] - 7s 26ms/step - loss: 0.0628 - tp: 283589.0000 - fp: 12458.0000 - tn: 272517.0000 - fn: 780.0000 - accuracy: 0.9767 - precision: 0.9579 - recall: 0.9973 - auc: 0.9977 - val_loss: 0.0263 - val_tp: 64.0000 - val_fp: 417.0000 - val_tn: 45079.0000 - val_fn: 9.0000 - val_accuracy: 0.9907 - val_precision: 0.1331 - val_recall: 0.8767 - val_auc: 0.9808
Epoch 17/100
278/278 [==============================] - 7s 26ms/step - loss: 0.0606 - tp: 284710.0000 - fp: 12263.0000 - tn: 271998.0000 - fn: 373.0000 - accuracy: 0.9778 - precision: 0.9587 - recall: 0.9987 - auc: 0.9978 - val_loss: 0.0251 - val_tp: 64.0000 - val_fp: 404.0000 - val_tn: 45092.0000 - val_fn: 9.0000 - val_accuracy: 0.9909 - val_precision: 0.1368 - val_recall: 0.8767 - val_auc: 0.9745
Epoch 18/100
276/278 [============================>.] - ETA: 0s - loss: 0.0585 - tp: 282380.0000 - fp: 12079.0000 - tn: 270672.0000 - fn: 117.0000 - accuracy: 0.9784 - precision: 0.9590 - recall: 0.9996 - auc: 0.9979Restoring model weights from the end of the best epoch.
278/278 [==============================] - 7s 26ms/step - loss: 0.0585 - tp: 284484.0000 - fp: 12159.0000 - tn: 272584.0000 - fn: 117.0000 - accuracy: 0.9784 - precision: 0.9590 - recall: 0.9996 - auc: 0.9979 - val_loss: 0.0235 - val_tp: 64.0000 - val_fp: 373.0000 - val_tn: 45123.0000 - val_fn: 9.0000 - val_accuracy: 0.9916 - val_precision: 0.1465 - val_recall: 0.8767 - val_auc: 0.9618
Epoch 00018: early stopping

If the training process were considering the whole dataset on each gradient update, this oversampling would be basically identical to the class weighting.

But when training the model batch-wise, as you did here, the oversampled data provides a smoother gradient signal: Instead of each positive example being shown in one batch with a large weight, they're shown in many different batches each time with a small weight.

This smoother gradient signal makes it easier to train the model.

Check training history

Note that the distributions of metrics will be different here, because the training data has a totally different distribution from the validation and test data.

plot_metrics(resampled_history )

Re-train

Because training is easier on the balanced data, the above training procedure may overfit quickly.

So break up the epochs to give the callbacks.EarlyStopping finer control over when to stop training.

resampled_model = make_model()
resampled_model.load_weights(initial_weights)

# Reset the bias to zero, since this dataset is balanced.
output_layer = resampled_model.layers[-1] 
output_layer.bias.assign([0])

resampled_history = resampled_model.fit(
    resampled_ds,
    # These are not real epochs
    steps_per_epoch = 20,
    epochs=10*EPOCHS,
    callbacks = [early_stopping],
    validation_data=(val_ds))

Train for 20 steps, validate for 23 steps
Epoch 1/1000
20/20 [==============================] - 3s 151ms/step - loss: 1.0161 - tp: 10416.0000 - fp: 6202.0000 - tn: 14337.0000 - fn: 10005.0000 - accuracy: 0.6043 - precision: 0.6268 - recall: 0.5101 - auc: 0.6064 - val_loss: 0.4448 - val_tp: 57.0000 - val_fp: 8713.0000 - val_tn: 36783.0000 - val_fn: 16.0000 - val_accuracy: 0.8084 - val_precision: 0.0065 - val_recall: 0.7808 - val_auc: 0.8380
Epoch 2/1000
20/20 [==============================] - 1s 30ms/step - loss: 0.6505 - tp: 14887.0000 - fp: 6394.0000 - tn: 14122.0000 - fn: 5557.0000 - accuracy: 0.7082 - precision: 0.6995 - recall: 0.7282 - auc: 0.7813 - val_loss: 0.4313 - val_tp: 59.0000 - val_fp: 7529.0000 - val_tn: 37967.0000 - val_fn: 14.0000 - val_accuracy: 0.8345 - val_precision: 0.0078 - val_recall: 0.8082 - val_auc: 0.8623
Epoch 3/1000
20/20 [==============================] - 1s 30ms/step - loss: 0.5292 - tp: 16450.0000 - fp: 5784.0000 - tn: 14464.0000 - fn: 4262.0000 - accuracy: 0.7547 - precision: 0.7399 - recall: 0.7942 - auc: 0.8390 - val_loss: 0.3952 - val_tp: 59.0000 - val_fp: 5168.0000 - val_tn: 40328.0000 - val_fn: 14.0000 - val_accuracy: 0.8863 - val_precision: 0.0113 - val_recall: 0.8082 - val_auc: 0.8771
Epoch 4/1000
20/20 [==============================] - 1s 31ms/step - loss: 0.4616 - tp: 16430.0000 - fp: 4937.0000 - tn: 15570.0000 - fn: 4023.0000 - accuracy: 0.7812 - precision: 0.7689 - recall: 0.8033 - auc: 0.8689 - val_loss: 0.3566 - val_tp: 60.0000 - val_fp: 3313.0000 - val_tn: 42183.0000 - val_fn: 13.0000 - val_accuracy: 0.9270 - val_precision: 0.0178 - val_recall: 0.8219 - val_auc: 0.8943
Epoch 5/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.4148 - tp: 16892.0000 - fp: 4199.0000 - tn: 16167.0000 - fn: 3702.0000 - accuracy: 0.8071 - precision: 0.8009 - recall: 0.8202 - auc: 0.8893 - val_loss: 0.3207 - val_tp: 61.0000 - val_fp: 2036.0000 - val_tn: 43460.0000 - val_fn: 12.0000 - val_accuracy: 0.9551 - val_precision: 0.0291 - val_recall: 0.8356 - val_auc: 0.9107
Epoch 6/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.3792 - tp: 16882.0000 - fp: 3674.0000 - tn: 17026.0000 - fn: 3378.0000 - accuracy: 0.8278 - precision: 0.8213 - recall: 0.8333 - auc: 0.9041 - val_loss: 0.2890 - val_tp: 61.0000 - val_fp: 1372.0000 - val_tn: 44124.0000 - val_fn: 12.0000 - val_accuracy: 0.9696 - val_precision: 0.0426 - val_recall: 0.8356 - val_auc: 0.9228
Epoch 7/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.3427 - tp: 17122.0000 - fp: 3059.0000 - tn: 17589.0000 - fn: 3190.0000 - accuracy: 0.8474 - precision: 0.8484 - recall: 0.8429 - auc: 0.9196 - val_loss: 0.2601 - val_tp: 61.0000 - val_fp: 1049.0000 - val_tn: 44447.0000 - val_fn: 12.0000 - val_accuracy: 0.9767 - val_precision: 0.0550 - val_recall: 0.8356 - val_auc: 0.9310
Epoch 8/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.3155 - tp: 17409.0000 - fp: 2566.0000 - tn: 17906.0000 - fn: 3079.0000 - accuracy: 0.8622 - precision: 0.8715 - recall: 0.8497 - auc: 0.9303 - val_loss: 0.2370 - val_tp: 63.0000 - val_fp: 910.0000 - val_tn: 44586.0000 - val_fn: 10.0000 - val_accuracy: 0.9798 - val_precision: 0.0647 - val_recall: 0.8630 - val_auc: 0.9384
Epoch 9/1000
20/20 [==============================] - 1s 33ms/step - loss: 0.2954 - tp: 17556.0000 - fp: 2260.0000 - tn: 18177.0000 - fn: 2967.0000 - accuracy: 0.8724 - precision: 0.8860 - recall: 0.8554 - auc: 0.9385 - val_loss: 0.2177 - val_tp: 63.0000 - val_fp: 862.0000 - val_tn: 44634.0000 - val_fn: 10.0000 - val_accuracy: 0.9809 - val_precision: 0.0681 - val_recall: 0.8630 - val_auc: 0.9448
Epoch 10/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.2800 - tp: 17687.0000 - fp: 2037.0000 - tn: 18334.0000 - fn: 2902.0000 - accuracy: 0.8794 - precision: 0.8967 - recall: 0.8591 - auc: 0.9445 - val_loss: 0.2007 - val_tp: 63.0000 - val_fp: 818.0000 - val_tn: 44678.0000 - val_fn: 10.0000 - val_accuracy: 0.9818 - val_precision: 0.0715 - val_recall: 0.8630 - val_auc: 0.9521
Epoch 11/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.2627 - tp: 17727.0000 - fp: 1802.0000 - tn: 18709.0000 - fn: 2722.0000 - accuracy: 0.8896 - precision: 0.9077 - recall: 0.8669 - auc: 0.9510 - val_loss: 0.1858 - val_tp: 63.0000 - val_fp: 792.0000 - val_tn: 44704.0000 - val_fn: 10.0000 - val_accuracy: 0.9824 - val_precision: 0.0737 - val_recall: 0.8630 - val_auc: 0.9574
Epoch 12/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.2476 - tp: 17807.0000 - fp: 1661.0000 - tn: 18813.0000 - fn: 2679.0000 - accuracy: 0.8940 - precision: 0.9147 - recall: 0.8692 - auc: 0.9568 - val_loss: 0.1722 - val_tp: 62.0000 - val_fp: 758.0000 - val_tn: 44738.0000 - val_fn: 11.0000 - val_accuracy: 0.9831 - val_precision: 0.0756 - val_recall: 0.8493 - val_auc: 0.9628
Epoch 13/1000
20/20 [==============================] - 1s 30ms/step - loss: 0.2383 - tp: 18065.0000 - fp: 1516.0000 - tn: 18780.0000 - fn: 2599.0000 - accuracy: 0.8995 - precision: 0.9226 - recall: 0.8742 - auc: 0.9604 - val_loss: 0.1604 - val_tp: 62.0000 - val_fp: 742.0000 - val_tn: 44754.0000 - val_fn: 11.0000 - val_accuracy: 0.9835 - val_precision: 0.0771 - val_recall: 0.8493 - val_auc: 0.9663
Epoch 14/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.2296 - tp: 17967.0000 - fp: 1473.0000 - tn: 18972.0000 - fn: 2548.0000 - accuracy: 0.9018 - precision: 0.9242 - recall: 0.8758 - auc: 0.9631 - val_loss: 0.1505 - val_tp: 62.0000 - val_fp: 720.0000 - val_tn: 44776.0000 - val_fn: 11.0000 - val_accuracy: 0.9840 - val_precision: 0.0793 - val_recall: 0.8493 - val_auc: 0.9693
Epoch 15/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.2203 - tp: 17989.0000 - fp: 1339.0000 - tn: 19187.0000 - fn: 2445.0000 - accuracy: 0.9076 - precision: 0.9307 - recall: 0.8803 - auc: 0.9665 - val_loss: 0.1411 - val_tp: 62.0000 - val_fp: 703.0000 - val_tn: 44793.0000 - val_fn: 11.0000 - val_accuracy: 0.9843 - val_precision: 0.0810 - val_recall: 0.8493 - val_auc: 0.9711
Epoch 16/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.2152 - tp: 17871.0000 - fp: 1270.0000 - tn: 19325.0000 - fn: 2494.0000 - accuracy: 0.9081 - precision: 0.9337 - recall: 0.8775 - auc: 0.9682 - val_loss: 0.1326 - val_tp: 62.0000 - val_fp: 689.0000 - val_tn: 44807.0000 - val_fn: 11.0000 - val_accuracy: 0.9846 - val_precision: 0.0826 - val_recall: 0.8493 - val_auc: 0.9735
Epoch 17/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.2060 - tp: 18235.0000 - fp: 1146.0000 - tn: 19174.0000 - fn: 2405.0000 - accuracy: 0.9133 - precision: 0.9409 - recall: 0.8835 - auc: 0.9713 - val_loss: 0.1254 - val_tp: 62.0000 - val_fp: 669.0000 - val_tn: 44827.0000 - val_fn: 11.0000 - val_accuracy: 0.9851 - val_precision: 0.0848 - val_recall: 0.8493 - val_auc: 0.9748
Epoch 18/1000
20/20 [==============================] - 1s 32ms/step - loss: 0.2001 - tp: 18287.0000 - fp: 1079.0000 - tn: 19235.0000 - fn: 2359.0000 - accuracy: 0.9161 - precision: 0.9443 - recall: 0.8857 - auc: 0.9726 - val_loss: 0.1193 - val_tp: 63.0000 - val_fp: 659.0000 - val_tn: 44837.0000 - val_fn: 10.0000 - val_accuracy: 0.9853 - val_precision: 0.0873 - val_recall: 0.8630 - val_auc: 0.9765
Epoch 19/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.1910 - tp: 18194.0000 - fp: 1054.0000 - tn: 19492.0000 - fn: 2220.0000 - accuracy: 0.9201 - precision: 0.9452 - recall: 0.8913 - auc: 0.9756 - val_loss: 0.1141 - val_tp: 63.0000 - val_fp: 652.0000 - val_tn: 44844.0000 - val_fn: 10.0000 - val_accuracy: 0.9855 - val_precision: 0.0881 - val_recall: 0.8630 - val_auc: 0.9777
Epoch 20/1000
20/20 [==============================] - 1s 30ms/step - loss: 0.1859 - tp: 18080.0000 - fp: 997.0000 - tn: 19684.0000 - fn: 2199.0000 - accuracy: 0.9220 - precision: 0.9477 - recall: 0.8916 - auc: 0.9772 - val_loss: 0.1088 - val_tp: 63.0000 - val_fp: 646.0000 - val_tn: 44850.0000 - val_fn: 10.0000 - val_accuracy: 0.9856 - val_precision: 0.0889 - val_recall: 0.8630 - val_auc: 0.9787
Epoch 21/1000
20/20 [==============================] - 1s 31ms/step - loss: 0.1793 - tp: 18319.0000 - fp: 955.0000 - tn: 19587.0000 - fn: 2099.0000 - accuracy: 0.9254 - precision: 0.9505 - recall: 0.8972 - auc: 0.9784 - val_loss: 0.1050 - val_tp: 63.0000 - val_fp: 656.0000 - val_tn: 44840.0000 - val_fn: 10.0000 - val_accuracy: 0.9854 - val_precision: 0.0876 - val_recall: 0.8630 - val_auc: 0.9791
Epoch 22/1000
20/20 [==============================] - 1s 30ms/step - loss: 0.1776 - tp: 18302.0000 - fp: 964.0000 - tn: 19578.0000 - fn: 2116.0000 - accuracy: 0.9248 - precision: 0.9500 - recall: 0.8964 - auc: 0.9791 - val_loss: 0.1007 - val_tp: 63.0000 - val_fp: 647.0000 - val_tn: 44849.0000 - val_fn: 10.0000 - val_accuracy: 0.9856 - val_precision: 0.0887 - val_recall: 0.8630 - val_auc: 0.9797
Epoch 23/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1724 - tp: 18285.0000 - fp: 861.0000 - tn: 19751.0000 - fn: 2063.0000 - accuracy: 0.9286 - precision: 0.9550 - recall: 0.8986 - auc: 0.9805 - val_loss: 0.0967 - val_tp: 63.0000 - val_fp: 639.0000 - val_tn: 44857.0000 - val_fn: 10.0000 - val_accuracy: 0.9858 - val_precision: 0.0897 - val_recall: 0.8630 - val_auc: 0.9804
Epoch 24/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.1653 - tp: 18326.0000 - fp: 840.0000 - tn: 19755.0000 - fn: 2039.0000 - accuracy: 0.9297 - precision: 0.9562 - recall: 0.8999 - auc: 0.9821 - val_loss: 0.0935 - val_tp: 63.0000 - val_fp: 629.0000 - val_tn: 44867.0000 - val_fn: 10.0000 - val_accuracy: 0.9860 - val_precision: 0.0910 - val_recall: 0.8630 - val_auc: 0.9809
Epoch 25/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.1632 - tp: 18468.0000 - fp: 814.0000 - tn: 19666.0000 - fn: 2012.0000 - accuracy: 0.9310 - precision: 0.9578 - recall: 0.9018 - auc: 0.9825 - val_loss: 0.0909 - val_tp: 63.0000 - val_fp: 630.0000 - val_tn: 44866.0000 - val_fn: 10.0000 - val_accuracy: 0.9860 - val_precision: 0.0909 - val_recall: 0.8630 - val_auc: 0.9813
Epoch 26/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1605 - tp: 18546.0000 - fp: 834.0000 - tn: 19590.0000 - fn: 1990.0000 - accuracy: 0.9311 - precision: 0.9570 - recall: 0.9031 - auc: 0.9833 - val_loss: 0.0884 - val_tp: 63.0000 - val_fp: 623.0000 - val_tn: 44873.0000 - val_fn: 10.0000 - val_accuracy: 0.9861 - val_precision: 0.0918 - val_recall: 0.8630 - val_auc: 0.9817
Epoch 27/1000
20/20 [==============================] - 1s 33ms/step - loss: 0.1562 - tp: 18485.0000 - fp: 739.0000 - tn: 19798.0000 - fn: 1938.0000 - accuracy: 0.9346 - precision: 0.9616 - recall: 0.9051 - auc: 0.9845 - val_loss: 0.0859 - val_tp: 63.0000 - val_fp: 618.0000 - val_tn: 44878.0000 - val_fn: 10.0000 - val_accuracy: 0.9862 - val_precision: 0.0925 - val_recall: 0.8630 - val_auc: 0.9819
Epoch 28/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1567 - tp: 18432.0000 - fp: 751.0000 - tn: 19773.0000 - fn: 2004.0000 - accuracy: 0.9327 - precision: 0.9609 - recall: 0.9019 - auc: 0.9841 - val_loss: 0.0837 - val_tp: 63.0000 - val_fp: 626.0000 - val_tn: 44870.0000 - val_fn: 10.0000 - val_accuracy: 0.9860 - val_precision: 0.0914 - val_recall: 0.8630 - val_auc: 0.9822
Epoch 29/1000
20/20 [==============================] - 1s 30ms/step - loss: 0.1502 - tp: 18669.0000 - fp: 736.0000 - tn: 19646.0000 - fn: 1909.0000 - accuracy: 0.9354 - precision: 0.9621 - recall: 0.9072 - auc: 0.9856 - val_loss: 0.0816 - val_tp: 63.0000 - val_fp: 614.0000 - val_tn: 44882.0000 - val_fn: 10.0000 - val_accuracy: 0.9863 - val_precision: 0.0931 - val_recall: 0.8630 - val_auc: 0.9825
Epoch 30/1000
20/20 [==============================] - 1s 30ms/step - loss: 0.1488 - tp: 18722.0000 - fp: 733.0000 - tn: 19591.0000 - fn: 1914.0000 - accuracy: 0.9354 - precision: 0.9623 - recall: 0.9072 - auc: 0.9859 - val_loss: 0.0800 - val_tp: 64.0000 - val_fp: 619.0000 - val_tn: 44877.0000 - val_fn: 9.0000 - val_accuracy: 0.9862 - val_precision: 0.0937 - val_recall: 0.8767 - val_auc: 0.9827
Epoch 31/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.1494 - tp: 18517.0000 - fp: 770.0000 - tn: 19765.0000 - fn: 1908.0000 - accuracy: 0.9346 - precision: 0.9601 - recall: 0.9066 - auc: 0.9860 - val_loss: 0.0784 - val_tp: 65.0000 - val_fp: 615.0000 - val_tn: 44881.0000 - val_fn: 8.0000 - val_accuracy: 0.9863 - val_precision: 0.0956 - val_recall: 0.8904 - val_auc: 0.9829
Epoch 32/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1444 - tp: 18600.0000 - fp: 675.0000 - tn: 19769.0000 - fn: 1916.0000 - accuracy: 0.9367 - precision: 0.9650 - recall: 0.9066 - auc: 0.9869 - val_loss: 0.0768 - val_tp: 65.0000 - val_fp: 610.0000 - val_tn: 44886.0000 - val_fn: 8.0000 - val_accuracy: 0.9864 - val_precision: 0.0963 - val_recall: 0.8904 - val_auc: 0.9831
Epoch 33/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.1423 - tp: 18803.0000 - fp: 661.0000 - tn: 19621.0000 - fn: 1875.0000 - accuracy: 0.9381 - precision: 0.9660 - recall: 0.9093 - auc: 0.9874 - val_loss: 0.0760 - val_tp: 66.0000 - val_fp: 620.0000 - val_tn: 44876.0000 - val_fn: 7.0000 - val_accuracy: 0.9862 - val_precision: 0.0962 - val_recall: 0.9041 - val_auc: 0.9834
Epoch 34/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1431 - tp: 18622.0000 - fp: 665.0000 - tn: 19801.0000 - fn: 1872.0000 - accuracy: 0.9381 - precision: 0.9655 - recall: 0.9087 - auc: 0.9876 - val_loss: 0.0748 - val_tp: 66.0000 - val_fp: 623.0000 - val_tn: 44873.0000 - val_fn: 7.0000 - val_accuracy: 0.9862 - val_precision: 0.0958 - val_recall: 0.9041 - val_auc: 0.9836
Epoch 35/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1383 - tp: 18555.0000 - fp: 684.0000 - tn: 19974.0000 - fn: 1747.0000 - accuracy: 0.9406 - precision: 0.9644 - recall: 0.9139 - auc: 0.9884 - val_loss: 0.0733 - val_tp: 66.0000 - val_fp: 608.0000 - val_tn: 44888.0000 - val_fn: 7.0000 - val_accuracy: 0.9865 - val_precision: 0.0979 - val_recall: 0.9041 - val_auc: 0.9838
Epoch 36/1000
20/20 [==============================] - 1s 33ms/step - loss: 0.1415 - tp: 18492.0000 - fp: 650.0000 - tn: 19940.0000 - fn: 1878.0000 - accuracy: 0.9383 - precision: 0.9660 - recall: 0.9078 - auc: 0.9876 - val_loss: 0.0720 - val_tp: 66.0000 - val_fp: 605.0000 - val_tn: 44891.0000 - val_fn: 7.0000 - val_accuracy: 0.9866 - val_precision: 0.0984 - val_recall: 0.9041 - val_auc: 0.9840
Epoch 37/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.1359 - tp: 18641.0000 - fp: 620.0000 - tn: 19897.0000 - fn: 1802.0000 - accuracy: 0.9409 - precision: 0.9678 - recall: 0.9119 - auc: 0.9887 - val_loss: 0.0706 - val_tp: 66.0000 - val_fp: 596.0000 - val_tn: 44900.0000 - val_fn: 7.0000 - val_accuracy: 0.9868 - val_precision: 0.0997 - val_recall: 0.9041 - val_auc: 0.9842
Epoch 38/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.1355 - tp: 18687.0000 - fp: 647.0000 - tn: 19787.0000 - fn: 1839.0000 - accuracy: 0.9393 - precision: 0.9665 - recall: 0.9104 - auc: 0.9890 - val_loss: 0.0695 - val_tp: 66.0000 - val_fp: 597.0000 - val_tn: 44899.0000 - val_fn: 7.0000 - val_accuracy: 0.9867 - val_precision: 0.0995 - val_recall: 0.9041 - val_auc: 0.9843
Epoch 39/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.1323 - tp: 18674.0000 - fp: 633.0000 - tn: 19835.0000 - fn: 1818.0000 - accuracy: 0.9402 - precision: 0.9672 - recall: 0.9113 - auc: 0.9894 - val_loss: 0.0689 - val_tp: 66.0000 - val_fp: 596.0000 - val_tn: 44900.0000 - val_fn: 7.0000 - val_accuracy: 0.9868 - val_precision: 0.0997 - val_recall: 0.9041 - val_auc: 0.9845
Epoch 40/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1319 - tp: 18661.0000 - fp: 636.0000 - tn: 19899.0000 - fn: 1764.0000 - accuracy: 0.9414 - precision: 0.9670 - recall: 0.9136 - auc: 0.9897 - val_loss: 0.0678 - val_tp: 66.0000 - val_fp: 598.0000 - val_tn: 44898.0000 - val_fn: 7.0000 - val_accuracy: 0.9867 - val_precision: 0.0994 - val_recall: 0.9041 - val_auc: 0.9846
Epoch 41/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.1299 - tp: 18719.0000 - fp: 607.0000 - tn: 19881.0000 - fn: 1753.0000 - accuracy: 0.9424 - precision: 0.9686 - recall: 0.9144 - auc: 0.9895 - val_loss: 0.0672 - val_tp: 66.0000 - val_fp: 620.0000 - val_tn: 44876.0000 - val_fn: 7.0000 - val_accuracy: 0.9862 - val_precision: 0.0962 - val_recall: 0.9041 - val_auc: 0.9848
Epoch 42/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1276 - tp: 18898.0000 - fp: 581.0000 - tn: 19755.0000 - fn: 1726.0000 - accuracy: 0.9437 - precision: 0.9702 - recall: 0.9163 - auc: 0.9903 - val_loss: 0.0669 - val_tp: 65.0000 - val_fp: 643.0000 - val_tn: 44853.0000 - val_fn: 8.0000 - val_accuracy: 0.9857 - val_precision: 0.0918 - val_recall: 0.8904 - val_auc: 0.9849
Epoch 43/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.1276 - tp: 18718.0000 - fp: 620.0000 - tn: 19919.0000 - fn: 1703.0000 - accuracy: 0.9433 - precision: 0.9679 - recall: 0.9166 - auc: 0.9903 - val_loss: 0.0659 - val_tp: 65.0000 - val_fp: 635.0000 - val_tn: 44861.0000 - val_fn: 8.0000 - val_accuracy: 0.9859 - val_precision: 0.0929 - val_recall: 0.8904 - val_auc: 0.9850
Epoch 44/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.1280 - tp: 18777.0000 - fp: 625.0000 - tn: 19843.0000 - fn: 1715.0000 - accuracy: 0.9429 - precision: 0.9678 - recall: 0.9163 - auc: 0.9902 - val_loss: 0.0651 - val_tp: 65.0000 - val_fp: 620.0000 - val_tn: 44876.0000 - val_fn: 8.0000 - val_accuracy: 0.9862 - val_precision: 0.0949 - val_recall: 0.8904 - val_auc: 0.9851
Epoch 45/1000
20/20 [==============================] - 1s 33ms/step - loss: 0.1265 - tp: 18645.0000 - fp: 609.0000 - tn: 19938.0000 - fn: 1768.0000 - accuracy: 0.9420 - precision: 0.9684 - recall: 0.9134 - auc: 0.9904 - val_loss: 0.0640 - val_tp: 65.0000 - val_fp: 612.0000 - val_tn: 44884.0000 - val_fn: 8.0000 - val_accuracy: 0.9864 - val_precision: 0.0960 - val_recall: 0.8904 - val_auc: 0.9852
Epoch 46/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.1255 - tp: 18751.0000 - fp: 607.0000 - tn: 19870.0000 - fn: 1732.0000 - accuracy: 0.9429 - precision: 0.9686 - recall: 0.9154 - auc: 0.9906 - val_loss: 0.0633 - val_tp: 65.0000 - val_fp: 612.0000 - val_tn: 44884.0000 - val_fn: 8.0000 - val_accuracy: 0.9864 - val_precision: 0.0960 - val_recall: 0.8904 - val_auc: 0.9853
Epoch 47/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.1230 - tp: 18835.0000 - fp: 582.0000 - tn: 19844.0000 - fn: 1699.0000 - accuracy: 0.9443 - precision: 0.9700 - recall: 0.9173 - auc: 0.9912 - val_loss: 0.0629 - val_tp: 65.0000 - val_fp: 620.0000 - val_tn: 44876.0000 - val_fn: 8.0000 - val_accuracy: 0.9862 - val_precision: 0.0949 - val_recall: 0.8904 - val_auc: 0.9854
Epoch 48/1000
20/20 [==============================] - 1s 30ms/step - loss: 0.1261 - tp: 18855.0000 - fp: 639.0000 - tn: 19815.0000 - fn: 1651.0000 - accuracy: 0.9441 - precision: 0.9672 - recall: 0.9195 - auc: 0.9906 - val_loss: 0.0619 - val_tp: 65.0000 - val_fp: 605.0000 - val_tn: 44891.0000 - val_fn: 8.0000 - val_accuracy: 0.9865 - val_precision: 0.0970 - val_recall: 0.8904 - val_auc: 0.9854
Epoch 49/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1212 - tp: 18759.0000 - fp: 600.0000 - tn: 19928.0000 - fn: 1673.0000 - accuracy: 0.9445 - precision: 0.9690 - recall: 0.9181 - auc: 0.9914 - val_loss: 0.0613 - val_tp: 65.0000 - val_fp: 612.0000 - val_tn: 44884.0000 - val_fn: 8.0000 - val_accuracy: 0.9864 - val_precision: 0.0960 - val_recall: 0.8904 - val_auc: 0.9855
Epoch 50/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.1222 - tp: 18924.0000 - fp: 614.0000 - tn: 19764.0000 - fn: 1658.0000 - accuracy: 0.9445 - precision: 0.9686 - recall: 0.9194 - auc: 0.9912 - val_loss: 0.0605 - val_tp: 65.0000 - val_fp: 611.0000 - val_tn: 44885.0000 - val_fn: 8.0000 - val_accuracy: 0.9864 - val_precision: 0.0962 - val_recall: 0.8904 - val_auc: 0.9855
Epoch 51/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.1195 - tp: 18670.0000 - fp: 547.0000 - tn: 20059.0000 - fn: 1684.0000 - accuracy: 0.9455 - precision: 0.9715 - recall: 0.9173 - auc: 0.9916 - val_loss: 0.0602 - val_tp: 65.0000 - val_fp: 612.0000 - val_tn: 44884.0000 - val_fn: 8.0000 - val_accuracy: 0.9864 - val_precision: 0.0960 - val_recall: 0.8904 - val_auc: 0.9856
Epoch 52/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.1195 - tp: 18933.0000 - fp: 608.0000 - tn: 19724.0000 - fn: 1695.0000 - accuracy: 0.9438 - precision: 0.9689 - recall: 0.9178 - auc: 0.9915 - val_loss: 0.0598 - val_tp: 65.0000 - val_fp: 612.0000 - val_tn: 44884.0000 - val_fn: 8.0000 - val_accuracy: 0.9864 - val_precision: 0.0960 - val_recall: 0.8904 - val_auc: 0.9856
Epoch 53/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1155 - tp: 18697.0000 - fp: 534.0000 - tn: 20118.0000 - fn: 1611.0000 - accuracy: 0.9476 - precision: 0.9722 - recall: 0.9207 - auc: 0.9923 - val_loss: 0.0591 - val_tp: 65.0000 - val_fp: 599.0000 - val_tn: 44897.0000 - val_fn: 8.0000 - val_accuracy: 0.9867 - val_precision: 0.0979 - val_recall: 0.8904 - val_auc: 0.9856
Epoch 54/1000
20/20 [==============================] - 1s 34ms/step - loss: 0.1168 - tp: 18834.0000 - fp: 580.0000 - tn: 19884.0000 - fn: 1662.0000 - accuracy: 0.9453 - precision: 0.9701 - recall: 0.9189 - auc: 0.9919 - val_loss: 0.0588 - val_tp: 65.0000 - val_fp: 608.0000 - val_tn: 44888.0000 - val_fn: 8.0000 - val_accuracy: 0.9865 - val_precision: 0.0966 - val_recall: 0.8904 - val_auc: 0.9856
Epoch 55/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.1160 - tp: 18849.0000 - fp: 569.0000 - tn: 19893.0000 - fn: 1649.0000 - accuracy: 0.9458 - precision: 0.9707 - recall: 0.9196 - auc: 0.9922 - val_loss: 0.0583 - val_tp: 65.0000 - val_fp: 599.0000 - val_tn: 44897.0000 - val_fn: 8.0000 - val_accuracy: 0.9867 - val_precision: 0.0979 - val_recall: 0.8904 - val_auc: 0.9857
Epoch 56/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.1172 - tp: 18843.0000 - fp: 559.0000 - tn: 19969.0000 - fn: 1589.0000 - accuracy: 0.9476 - precision: 0.9712 - recall: 0.9222 - auc: 0.9921 - val_loss: 0.0577 - val_tp: 65.0000 - val_fp: 601.0000 - val_tn: 44895.0000 - val_fn: 8.0000 - val_accuracy: 0.9866 - val_precision: 0.0976 - val_recall: 0.8904 - val_auc: 0.9856
Epoch 57/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.1162 - tp: 18925.0000 - fp: 562.0000 - tn: 19856.0000 - fn: 1617.0000 - accuracy: 0.9468 - precision: 0.9712 - recall: 0.9213 - auc: 0.9923 - val_loss: 0.0569 - val_tp: 65.0000 - val_fp: 596.0000 - val_tn: 44900.0000 - val_fn: 8.0000 - val_accuracy: 0.9867 - val_precision: 0.0983 - val_recall: 0.8904 - val_auc: 0.9857
Epoch 58/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.1120 - tp: 18958.0000 - fp: 518.0000 - tn: 19863.0000 - fn: 1621.0000 - accuracy: 0.9478 - precision: 0.9734 - recall: 0.9212 - auc: 0.9929 - val_loss: 0.0567 - val_tp: 65.0000 - val_fp: 602.0000 - val_tn: 44894.0000 - val_fn: 8.0000 - val_accuracy: 0.9866 - val_precision: 0.0975 - val_recall: 0.8904 - val_auc: 0.9857
Epoch 59/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1123 - tp: 18793.0000 - fp: 555.0000 - tn: 20066.0000 - fn: 1546.0000 - accuracy: 0.9487 - precision: 0.9713 - recall: 0.9240 - auc: 0.9929 - val_loss: 0.0561 - val_tp: 65.0000 - val_fp: 602.0000 - val_tn: 44894.0000 - val_fn: 8.0000 - val_accuracy: 0.9866 - val_precision: 0.0975 - val_recall: 0.8904 - val_auc: 0.9857
Epoch 60/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.1099 - tp: 18854.0000 - fp: 507.0000 - tn: 20112.0000 - fn: 1487.0000 - accuracy: 0.9513 - precision: 0.9738 - recall: 0.9269 - auc: 0.9932 - val_loss: 0.0562 - val_tp: 65.0000 - val_fp: 603.0000 - val_tn: 44893.0000 - val_fn: 8.0000 - val_accuracy: 0.9866 - val_precision: 0.0973 - val_recall: 0.8904 - val_auc: 0.9858
Epoch 61/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.1143 - tp: 18926.0000 - fp: 567.0000 - tn: 19940.0000 - fn: 1527.0000 - accuracy: 0.9489 - precision: 0.9709 - recall: 0.9253 - auc: 0.9926 - val_loss: 0.0556 - val_tp: 65.0000 - val_fp: 600.0000 - val_tn: 44896.0000 - val_fn: 8.0000 - val_accuracy: 0.9867 - val_precision: 0.0977 - val_recall: 0.8904 - val_auc: 0.9858
Epoch 62/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.1099 - tp: 18997.0000 - fp: 534.0000 - tn: 19952.0000 - fn: 1477.0000 - accuracy: 0.9509 - precision: 0.9727 - recall: 0.9279 - auc: 0.9931 - val_loss: 0.0548 - val_tp: 65.0000 - val_fp: 586.0000 - val_tn: 44910.0000 - val_fn: 8.0000 - val_accuracy: 0.9870 - val_precision: 0.0998 - val_recall: 0.8904 - val_auc: 0.9859
Epoch 63/1000
20/20 [==============================] - 1s 34ms/step - loss: 0.1095 - tp: 18858.0000 - fp: 525.0000 - tn: 20146.0000 - fn: 1431.0000 - accuracy: 0.9522 - precision: 0.9729 - recall: 0.9295 - auc: 0.9932 - val_loss: 0.0538 - val_tp: 65.0000 - val_fp: 571.0000 - val_tn: 44925.0000 - val_fn: 8.0000 - val_accuracy: 0.9873 - val_precision: 0.1022 - val_recall: 0.8904 - val_auc: 0.9860
Epoch 64/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.1092 - tp: 19138.0000 - fp: 552.0000 - tn: 19873.0000 - fn: 1397.0000 - accuracy: 0.9524 - precision: 0.9720 - recall: 0.9320 - auc: 0.9932 - val_loss: 0.0536 - val_tp: 65.0000 - val_fp: 575.0000 - val_tn: 44921.0000 - val_fn: 8.0000 - val_accuracy: 0.9872 - val_precision: 0.1016 - val_recall: 0.8904 - val_auc: 0.9860
Epoch 65/1000
20/20 [==============================] - 1s 30ms/step - loss: 0.1081 - tp: 18986.0000 - fp: 514.0000 - tn: 19997.0000 - fn: 1463.0000 - accuracy: 0.9517 - precision: 0.9736 - recall: 0.9285 - auc: 0.9933 - val_loss: 0.0535 - val_tp: 65.0000 - val_fp: 574.0000 - val_tn: 44922.0000 - val_fn: 8.0000 - val_accuracy: 0.9872 - val_precision: 0.1017 - val_recall: 0.8904 - val_auc: 0.9861
Epoch 66/1000
20/20 [==============================] - 1s 30ms/step - loss: 0.1095 - tp: 19071.0000 - fp: 542.0000 - tn: 19863.0000 - fn: 1484.0000 - accuracy: 0.9505 - precision: 0.9724 - recall: 0.9278 - auc: 0.9932 - val_loss: 0.0530 - val_tp: 65.0000 - val_fp: 570.0000 - val_tn: 44926.0000 - val_fn: 8.0000 - val_accuracy: 0.9873 - val_precision: 0.1024 - val_recall: 0.8904 - val_auc: 0.9861
Epoch 67/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1066 - tp: 19010.0000 - fp: 528.0000 - tn: 20045.0000 - fn: 1377.0000 - accuracy: 0.9535 - precision: 0.9730 - recall: 0.9325 - auc: 0.9936 - val_loss: 0.0527 - val_tp: 65.0000 - val_fp: 570.0000 - val_tn: 44926.0000 - val_fn: 8.0000 - val_accuracy: 0.9873 - val_precision: 0.1024 - val_recall: 0.8904 - val_auc: 0.9861
Epoch 68/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1058 - tp: 19030.0000 - fp: 523.0000 - tn: 19993.0000 - fn: 1414.0000 - accuracy: 0.9527 - precision: 0.9733 - recall: 0.9308 - auc: 0.9937 - val_loss: 0.0527 - val_tp: 65.0000 - val_fp: 577.0000 - val_tn: 44919.0000 - val_fn: 8.0000 - val_accuracy: 0.9872 - val_precision: 0.1012 - val_recall: 0.8904 - val_auc: 0.9862
Epoch 69/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1036 - tp: 19023.0000 - fp: 520.0000 - tn: 19983.0000 - fn: 1434.0000 - accuracy: 0.9523 - precision: 0.9734 - recall: 0.9299 - auc: 0.9940 - val_loss: 0.0526 - val_tp: 65.0000 - val_fp: 585.0000 - val_tn: 44911.0000 - val_fn: 8.0000 - val_accuracy: 0.9870 - val_precision: 0.1000 - val_recall: 0.8904 - val_auc: 0.9862
Epoch 70/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1063 - tp: 18902.0000 - fp: 532.0000 - tn: 20060.0000 - fn: 1466.0000 - accuracy: 0.9512 - precision: 0.9726 - recall: 0.9280 - auc: 0.9936 - val_loss: 0.0521 - val_tp: 65.0000 - val_fp: 578.0000 - val_tn: 44918.0000 - val_fn: 8.0000 - val_accuracy: 0.9871 - val_precision: 0.1011 - val_recall: 0.8904 - val_auc: 0.9862
Epoch 71/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.1057 - tp: 19171.0000 - fp: 554.0000 - tn: 19812.0000 - fn: 1423.0000 - accuracy: 0.9517 - precision: 0.9719 - recall: 0.9309 - auc: 0.9936 - val_loss: 0.0523 - val_tp: 65.0000 - val_fp: 588.0000 - val_tn: 44908.0000 - val_fn: 8.0000 - val_accuracy: 0.9869 - val_precision: 0.0995 - val_recall: 0.8904 - val_auc: 0.9862
Epoch 72/1000
20/20 [==============================] - 1s 31ms/step - loss: 0.1049 - tp: 19158.0000 - fp: 500.0000 - tn: 19919.0000 - fn: 1383.0000 - accuracy: 0.9540 - precision: 0.9746 - recall: 0.9327 - auc: 0.9939 - val_loss: 0.0518 - val_tp: 65.0000 - val_fp: 584.0000 - val_tn: 44912.0000 - val_fn: 8.0000 - val_accuracy: 0.9870 - val_precision: 0.1002 - val_recall: 0.8904 - val_auc: 0.9862
Epoch 73/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.1044 - tp: 19106.0000 - fp: 512.0000 - tn: 19961.0000 - fn: 1381.0000 - accuracy: 0.9538 - precision: 0.9739 - recall: 0.9326 - auc: 0.9939 - val_loss: 0.0513 - val_tp: 65.0000 - val_fp: 578.0000 - val_tn: 44918.0000 - val_fn: 8.0000 - val_accuracy: 0.9871 - val_precision: 0.1011 - val_recall: 0.8904 - val_auc: 0.9863
Epoch 74/1000
20/20 [==============================] - 1s 30ms/step - loss: 0.1042 - tp: 19039.0000 - fp: 564.0000 - tn: 19943.0000 - fn: 1414.0000 - accuracy: 0.9517 - precision: 0.9712 - recall: 0.9309 - auc: 0.9939 - val_loss: 0.0507 - val_tp: 65.0000 - val_fp: 568.0000 - val_tn: 44928.0000 - val_fn: 8.0000 - val_accuracy: 0.9874 - val_precision: 0.1027 - val_recall: 0.8904 - val_auc: 0.9863
Epoch 75/1000
20/20 [==============================] - 1s 31ms/step - loss: 0.1030 - tp: 19078.0000 - fp: 504.0000 - tn: 19970.0000 - fn: 1408.0000 - accuracy: 0.9533 - precision: 0.9743 - recall: 0.9313 - auc: 0.9940 - val_loss: 0.0505 - val_tp: 65.0000 - val_fp: 573.0000 - val_tn: 44923.0000 - val_fn: 8.0000 - val_accuracy: 0.9873 - val_precision: 0.1019 - val_recall: 0.8904 - val_auc: 0.9864
Epoch 76/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.1022 - tp: 19101.0000 - fp: 519.0000 - tn: 19924.0000 - fn: 1416.0000 - accuracy: 0.9528 - precision: 0.9735 - recall: 0.9310 - auc: 0.9941 - val_loss: 0.0506 - val_tp: 65.0000 - val_fp: 582.0000 - val_tn: 44914.0000 - val_fn: 8.0000 - val_accuracy: 0.9871 - val_precision: 0.1005 - val_recall: 0.8904 - val_auc: 0.9863
Epoch 77/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1004 - tp: 19255.0000 - fp: 522.0000 - tn: 19806.0000 - fn: 1377.0000 - accuracy: 0.9536 - precision: 0.9736 - recall: 0.9333 - auc: 0.9943 - val_loss: 0.0504 - val_tp: 65.0000 - val_fp: 582.0000 - val_tn: 44914.0000 - val_fn: 8.0000 - val_accuracy: 0.9871 - val_precision: 0.1005 - val_recall: 0.8904 - val_auc: 0.9864
Epoch 78/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1023 - tp: 19081.0000 - fp: 504.0000 - tn: 19984.0000 - fn: 1391.0000 - accuracy: 0.9537 - precision: 0.9743 - recall: 0.9321 - auc: 0.9943 - val_loss: 0.0503 - val_tp: 65.0000 - val_fp: 581.0000 - val_tn: 44915.0000 - val_fn: 8.0000 - val_accuracy: 0.9871 - val_precision: 0.1006 - val_recall: 0.8904 - val_auc: 0.9865
Epoch 79/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.0992 - tp: 19118.0000 - fp: 539.0000 - tn: 20028.0000 - fn: 1275.0000 - accuracy: 0.9557 - precision: 0.9726 - recall: 0.9375 - auc: 0.9946 - val_loss: 0.0503 - val_tp: 65.0000 - val_fp: 588.0000 - val_tn: 44908.0000 - val_fn: 8.0000 - val_accuracy: 0.9869 - val_precision: 0.0995 - val_recall: 0.8904 - val_auc: 0.9866
Epoch 80/1000
20/20 [==============================] - 1s 32ms/step - loss: 0.1033 - tp: 19303.0000 - fp: 537.0000 - tn: 19754.0000 - fn: 1366.0000 - accuracy: 0.9535 - precision: 0.9729 - recall: 0.9339 - auc: 0.9941 - val_loss: 0.0496 - val_tp: 65.0000 - val_fp: 573.0000 - val_tn: 44923.0000 - val_fn: 8.0000 - val_accuracy: 0.9873 - val_precision: 0.1019 - val_recall: 0.8904 - val_auc: 0.9866
Epoch 81/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.1003 - tp: 19155.0000 - fp: 565.0000 - tn: 19870.0000 - fn: 1370.0000 - accuracy: 0.9528 - precision: 0.9713 - recall: 0.9333 - auc: 0.9943 - val_loss: 0.0492 - val_tp: 65.0000 - val_fp: 567.0000 - val_tn: 44929.0000 - val_fn: 8.0000 - val_accuracy: 0.9874 - val_precision: 0.1028 - val_recall: 0.8904 - val_auc: 0.9867
Epoch 82/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.1015 - tp: 19013.0000 - fp: 548.0000 - tn: 20038.0000 - fn: 1361.0000 - accuracy: 0.9534 - precision: 0.9720 - recall: 0.9332 - auc: 0.9943 - val_loss: 0.0488 - val_tp: 65.0000 - val_fp: 564.0000 - val_tn: 44932.0000 - val_fn: 8.0000 - val_accuracy: 0.9874 - val_precision: 0.1033 - val_recall: 0.8904 - val_auc: 0.9867
Epoch 83/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.0980 - tp: 19079.0000 - fp: 529.0000 - tn: 20002.0000 - fn: 1350.0000 - accuracy: 0.9541 - precision: 0.9730 - recall: 0.9339 - auc: 0.9946 - val_loss: 0.0482 - val_tp: 65.0000 - val_fp: 554.0000 - val_tn: 44942.0000 - val_fn: 8.0000 - val_accuracy: 0.9877 - val_precision: 0.1050 - val_recall: 0.8904 - val_auc: 0.9867
Epoch 84/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.0983 - tp: 19284.0000 - fp: 477.0000 - tn: 19830.0000 - fn: 1369.0000 - accuracy: 0.9549 - precision: 0.9759 - recall: 0.9337 - auc: 0.9948 - val_loss: 0.0484 - val_tp: 65.0000 - val_fp: 565.0000 - val_tn: 44931.0000 - val_fn: 8.0000 - val_accuracy: 0.9874 - val_precision: 0.1032 - val_recall: 0.8904 - val_auc: 0.9867
Epoch 85/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.0984 - tp: 19164.0000 - fp: 526.0000 - tn: 19921.0000 - fn: 1349.0000 - accuracy: 0.9542 - precision: 0.9733 - recall: 0.9342 - auc: 0.9946 - val_loss: 0.0481 - val_tp: 65.0000 - val_fp: 563.0000 - val_tn: 44933.0000 - val_fn: 8.0000 - val_accuracy: 0.9875 - val_precision: 0.1035 - val_recall: 0.8904 - val_auc: 0.9868
Epoch 86/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.0959 - tp: 19086.0000 - fp: 514.0000 - tn: 20069.0000 - fn: 1291.0000 - accuracy: 0.9559 - precision: 0.9738 - recall: 0.9366 - auc: 0.9948 - val_loss: 0.0482 - val_tp: 65.0000 - val_fp: 573.0000 - val_tn: 44923.0000 - val_fn: 8.0000 - val_accuracy: 0.9873 - val_precision: 0.1019 - val_recall: 0.8904 - val_auc: 0.9868
Epoch 87/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.0971 - tp: 19010.0000 - fp: 515.0000 - tn: 20091.0000 - fn: 1344.0000 - accuracy: 0.9546 - precision: 0.9736 - recall: 0.9340 - auc: 0.9949 - val_loss: 0.0478 - val_tp: 65.0000 - val_fp: 561.0000 - val_tn: 44935.0000 - val_fn: 8.0000 - val_accuracy: 0.9875 - val_precision: 0.1038 - val_recall: 0.8904 - val_auc: 0.9869
Epoch 88/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.0970 - tp: 19172.0000 - fp: 518.0000 - tn: 19977.0000 - fn: 1293.0000 - accuracy: 0.9558 - precision: 0.9737 - recall: 0.9368 - auc: 0.9948 - val_loss: 0.0473 - val_tp: 65.0000 - val_fp: 556.0000 - val_tn: 44940.0000 - val_fn: 8.0000 - val_accuracy: 0.9876 - val_precision: 0.1047 - val_recall: 0.8904 - val_auc: 0.9869
Epoch 89/1000
20/20 [==============================] - 1s 32ms/step - loss: 0.0932 - tp: 19131.0000 - fp: 498.0000 - tn: 20047.0000 - fn: 1284.0000 - accuracy: 0.9565 - precision: 0.9746 - recall: 0.9371 - auc: 0.9953 - val_loss: 0.0468 - val_tp: 65.0000 - val_fp: 552.0000 - val_tn: 44944.0000 - val_fn: 8.0000 - val_accuracy: 0.9877 - val_precision: 0.1053 - val_recall: 0.8904 - val_auc: 0.9869
Epoch 90/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.0931 - tp: 19223.0000 - fp: 502.0000 - tn: 19941.0000 - fn: 1294.0000 - accuracy: 0.9562 - precision: 0.9746 - recall: 0.9369 - auc: 0.9951 - val_loss: 0.0470 - val_tp: 65.0000 - val_fp: 558.0000 - val_tn: 44938.0000 - val_fn: 8.0000 - val_accuracy: 0.9876 - val_precision: 0.1043 - val_recall: 0.8904 - val_auc: 0.9869
Epoch 91/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.0950 - tp: 19232.0000 - fp: 531.0000 - tn: 19950.0000 - fn: 1247.0000 - accuracy: 0.9566 - precision: 0.9731 - recall: 0.9391 - auc: 0.9950 - val_loss: 0.0465 - val_tp: 65.0000 - val_fp: 549.0000 - val_tn: 44947.0000 - val_fn: 8.0000 - val_accuracy: 0.9878 - val_precision: 0.1059 - val_recall: 0.8904 - val_auc: 0.9869
Epoch 92/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.0964 - tp: 19262.0000 - fp: 552.0000 - tn: 19839.0000 - fn: 1307.0000 - accuracy: 0.9546 - precision: 0.9721 - recall: 0.9365 - auc: 0.9948 - val_loss: 0.0465 - val_tp: 65.0000 - val_fp: 558.0000 - val_tn: 44938.0000 - val_fn: 8.0000 - val_accuracy: 0.9876 - val_precision: 0.1043 - val_recall: 0.8904 - val_auc: 0.9869
Epoch 93/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.0940 - tp: 19182.0000 - fp: 512.0000 - tn: 20023.0000 - fn: 1243.0000 - accuracy: 0.9572 - precision: 0.9740 - recall: 0.9391 - auc: 0.9952 - val_loss: 0.0461 - val_tp: 65.0000 - val_fp: 551.0000 - val_tn: 44945.0000 - val_fn: 8.0000 - val_accuracy: 0.9877 - val_precision: 0.1055 - val_recall: 0.8904 - val_auc: 0.9869
Epoch 94/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.0940 - tp: 19407.0000 - fp: 522.0000 - tn: 19782.0000 - fn: 1249.0000 - accuracy: 0.9568 - precision: 0.9738 - recall: 0.9395 - auc: 0.9951 - val_loss: 0.0460 - val_tp: 65.0000 - val_fp: 558.0000 - val_tn: 44938.0000 - val_fn: 8.0000 - val_accuracy: 0.9876 - val_precision: 0.1043 - val_recall: 0.8904 - val_auc: 0.9869
Epoch 95/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.0935 - tp: 19294.0000 - fp: 526.0000 - tn: 19855.0000 - fn: 1285.0000 - accuracy: 0.9558 - precision: 0.9735 - recall: 0.9376 - auc: 0.9952 - val_loss: 0.0458 - val_tp: 65.0000 - val_fp: 552.0000 - val_tn: 44944.0000 - val_fn: 8.0000 - val_accuracy: 0.9877 - val_precision: 0.1053 - val_recall: 0.8904 - val_auc: 0.9870
Epoch 96/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.0950 - tp: 19198.0000 - fp: 550.0000 - tn: 19932.0000 - fn: 1280.0000 - accuracy: 0.9553 - precision: 0.9721 - recall: 0.9375 - auc: 0.9951 - val_loss: 0.0451 - val_tp: 65.0000 - val_fp: 540.0000 - val_tn: 44956.0000 - val_fn: 8.0000 - val_accuracy: 0.9880 - val_precision: 0.1074 - val_recall: 0.8904 - val_auc: 0.9871
Epoch 97/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.0911 - tp: 19160.0000 - fp: 494.0000 - tn: 20067.0000 - fn: 1239.0000 - accuracy: 0.9577 - precision: 0.9749 - recall: 0.9393 - auc: 0.9955 - val_loss: 0.0455 - val_tp: 65.0000 - val_fp: 550.0000 - val_tn: 44946.0000 - val_fn: 8.0000 - val_accuracy: 0.9878 - val_precision: 0.1057 - val_recall: 0.8904 - val_auc: 0.9871
Epoch 98/1000
20/20 [==============================] - 1s 32ms/step - loss: 0.0925 - tp: 19372.0000 - fp: 505.0000 - tn: 19784.0000 - fn: 1299.0000 - accuracy: 0.9560 - precision: 0.9746 - recall: 0.9372 - auc: 0.9953 - val_loss: 0.0453 - val_tp: 65.0000 - val_fp: 544.0000 - val_tn: 44952.0000 - val_fn: 8.0000 - val_accuracy: 0.9879 - val_precision: 0.1067 - val_recall: 0.8904 - val_auc: 0.9871
Epoch 99/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.0911 - tp: 19167.0000 - fp: 505.0000 - tn: 20045.0000 - fn: 1243.0000 - accuracy: 0.9573 - precision: 0.9743 - recall: 0.9391 - auc: 0.9955 - val_loss: 0.0450 - val_tp: 65.0000 - val_fp: 543.0000 - val_tn: 44953.0000 - val_fn: 8.0000 - val_accuracy: 0.9879 - val_precision: 0.1069 - val_recall: 0.8904 - val_auc: 0.9871
Epoch 100/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.0928 - tp: 19248.0000 - fp: 523.0000 - tn: 19930.0000 - fn: 1259.0000 - accuracy: 0.9565 - precision: 0.9735 - recall: 0.9386 - auc: 0.9953 - val_loss: 0.0444 - val_tp: 66.0000 - val_fp: 538.0000 - val_tn: 44958.0000 - val_fn: 7.0000 - val_accuracy: 0.9880 - val_precision: 0.1093 - val_recall: 0.9041 - val_auc: 0.9871
Epoch 101/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.0929 - tp: 19334.0000 - fp: 504.0000 - tn: 19824.0000 - fn: 1298.0000 - accuracy: 0.9560 - precision: 0.9746 - recall: 0.9371 - auc: 0.9953 - val_loss: 0.0440 - val_tp: 66.0000 - val_fp: 532.0000 - val_tn: 44964.0000 - val_fn: 7.0000 - val_accuracy: 0.9882 - val_precision: 0.1104 - val_recall: 0.9041 - val_auc: 0.9872
Epoch 102/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.0911 - tp: 19213.0000 - fp: 512.0000 - tn: 20024.0000 - fn: 1211.0000 - accuracy: 0.9579 - precision: 0.9740 - recall: 0.9407 - auc: 0.9954 - val_loss: 0.0440 - val_tp: 65.0000 - val_fp: 531.0000 - val_tn: 44965.0000 - val_fn: 8.0000 - val_accuracy: 0.9882 - val_precision: 0.1091 - val_recall: 0.8904 - val_auc: 0.9872
Epoch 103/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.0892 - tp: 19052.0000 - fp: 520.0000 - tn: 20208.0000 - fn: 1180.0000 - accuracy: 0.9585 - precision: 0.9734 - recall: 0.9417 - auc: 0.9956 - val_loss: 0.0439 - val_tp: 66.0000 - val_fp: 532.0000 - val_tn: 44964.0000 - val_fn: 7.0000 - val_accuracy: 0.9882 - val_precision: 0.1104 - val_recall: 0.9041 - val_auc: 0.9872
Epoch 104/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.0905 - tp: 19211.0000 - fp: 531.0000 - tn: 19974.0000 - fn: 1244.0000 - accuracy: 0.9567 - precision: 0.9731 - recall: 0.9392 - auc: 0.9956 - val_loss: 0.0437 - val_tp: 66.0000 - val_fp: 531.0000 - val_tn: 44965.0000 - val_fn: 7.0000 - val_accuracy: 0.9882 - val_precision: 0.1106 - val_recall: 0.9041 - val_auc: 0.9872
Epoch 105/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.0906 - tp: 19386.0000 - fp: 505.0000 - tn: 19868.0000 - fn: 1201.0000 - accuracy: 0.9583 - precision: 0.9746 - recall: 0.9417 - auc: 0.9956 - val_loss: 0.0432 - val_tp: 65.0000 - val_fp: 525.0000 - val_tn: 44971.0000 - val_fn: 8.0000 - val_accuracy: 0.9883 - val_precision: 0.1102 - val_recall: 0.8904 - val_auc: 0.9872
Epoch 106/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.0890 - tp: 19383.0000 - fp: 512.0000 - tn: 19823.0000 - fn: 1242.0000 - accuracy: 0.9572 - precision: 0.9743 - recall: 0.9398 - auc: 0.9957 - val_loss: 0.0435 - val_tp: 65.0000 - val_fp: 533.0000 - val_tn: 44963.0000 - val_fn: 8.0000 - val_accuracy: 0.9881 - val_precision: 0.1087 - val_recall: 0.8904 - val_auc: 0.9873
Epoch 107/1000
20/20 [==============================] - 1s 32ms/step - loss: 0.0880 - tp: 19371.0000 - fp: 557.0000 - tn: 19873.0000 - fn: 1159.0000 - accuracy: 0.9581 - precision: 0.9720 - recall: 0.9435 - auc: 0.9957 - val_loss: 0.0434 - val_tp: 66.0000 - val_fp: 535.0000 - val_tn: 44961.0000 - val_fn: 7.0000 - val_accuracy: 0.9881 - val_precision: 0.1098 - val_recall: 0.9041 - val_auc: 0.9872
Epoch 108/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.0890 - tp: 19174.0000 - fp: 560.0000 - tn: 20044.0000 - fn: 1182.0000 - accuracy: 0.9575 - precision: 0.9716 - recall: 0.9419 - auc: 0.9956 - val_loss: 0.0425 - val_tp: 66.0000 - val_fp: 525.0000 - val_tn: 44971.0000 - val_fn: 7.0000 - val_accuracy: 0.9883 - val_precision: 0.1117 - val_recall: 0.9041 - val_auc: 0.9872
Epoch 109/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.0905 - tp: 19238.0000 - fp: 499.0000 - tn: 19979.0000 - fn: 1244.0000 - accuracy: 0.9574 - precision: 0.9747 - recall: 0.9393 - auc: 0.9956 - val_loss: 0.0423 - val_tp: 66.0000 - val_fp: 524.0000 - val_tn: 44972.0000 - val_fn: 7.0000 - val_accuracy: 0.9883 - val_precision: 0.1119 - val_recall: 0.9041 - val_auc: 0.9872
Epoch 110/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.0904 - tp: 19481.0000 - fp: 548.0000 - tn: 19743.0000 - fn: 1188.0000 - accuracy: 0.9576 - precision: 0.9726 - recall: 0.9425 - auc: 0.9956 - val_loss: 0.0425 - val_tp: 65.0000 - val_fp: 533.0000 - val_tn: 44963.0000 - val_fn: 8.0000 - val_accuracy: 0.9881 - val_precision: 0.1087 - val_recall: 0.8904 - val_auc: 0.9872
Epoch 111/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.0868 - tp: 19175.0000 - fp: 517.0000 - tn: 20096.0000 - fn: 1172.0000 - accuracy: 0.9588 - precision: 0.9737 - recall: 0.9424 - auc: 0.9960 - val_loss: 0.0423 - val_tp: 65.0000 - val_fp: 532.0000 - val_tn: 44964.0000 - val_fn: 8.0000 - val_accuracy: 0.9881 - val_precision: 0.1089 - val_recall: 0.8904 - val_auc: 0.9872
Epoch 112/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.0866 - tp: 19353.0000 - fp: 495.0000 - tn: 19952.0000 - fn: 1160.0000 - accuracy: 0.9596 - precision: 0.9751 - recall: 0.9435 - auc: 0.9960 - val_loss: 0.0422 - val_tp: 66.0000 - val_fp: 533.0000 - val_tn: 44963.0000 - val_fn: 7.0000 - val_accuracy: 0.9881 - val_precision: 0.1102 - val_recall: 0.9041 - val_auc: 0.9872
Epoch 113/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.0871 - tp: 19381.0000 - fp: 521.0000 - tn: 19929.0000 - fn: 1129.0000 - accuracy: 0.9597 - precision: 0.9738 - recall: 0.9450 - auc: 0.9958 - val_loss: 0.0417 - val_tp: 66.0000 - val_fp: 529.0000 - val_tn: 44967.0000 - val_fn: 7.0000 - val_accuracy: 0.9882 - val_precision: 0.1109 - val_recall: 0.9041 - val_auc: 0.9873
Epoch 114/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.0874 - tp: 19250.0000 - fp: 513.0000 - tn: 19998.0000 - fn: 1199.0000 - accuracy: 0.9582 - precision: 0.9740 - recall: 0.9414 - auc: 0.9958 - val_loss: 0.0418 - val_tp: 66.0000 - val_fp: 535.0000 - val_tn: 44961.0000 - val_fn: 7.0000 - val_accuracy: 0.9881 - val_precision: 0.1098 - val_recall: 0.9041 - val_auc: 0.9872
Epoch 115/1000
20/20 [==============================] - 1s 27ms/step - loss: 0.0859 - tp: 19295.0000 - fp: 491.0000 - tn: 19978.0000 - fn: 1196.0000 - accuracy: 0.9588 - precision: 0.9752 - recall: 0.9416 - auc: 0.9960 - val_loss: 0.0419 - val_tp: 66.0000 - val_fp: 541.0000 - val_tn: 44955.0000 - val_fn: 7.0000 - val_accuracy: 0.9880 - val_precision: 0.1087 - val_recall: 0.9041 - val_auc: 0.9873
Epoch 116/1000
20/20 [==============================] - 1s 33ms/step - loss: 0.0879 - tp: 19277.0000 - fp: 528.0000 - tn: 19998.0000 - fn: 1157.0000 - accuracy: 0.9589 - precision: 0.9733 - recall: 0.9434 - auc: 0.9958 - val_loss: 0.0418 - val_tp: 66.0000 - val_fp: 541.0000 - val_tn: 44955.0000 - val_fn: 7.0000 - val_accuracy: 0.9880 - val_precision: 0.1087 - val_recall: 0.9041 - val_auc: 0.9873
Epoch 117/1000
20/20 [==============================] - 1s 30ms/step - loss: 0.0867 - tp: 19461.0000 - fp: 537.0000 - tn: 19837.0000 - fn: 1125.0000 - accuracy: 0.9594 - precision: 0.9731 - recall: 0.9454 - auc: 0.9959 - val_loss: 0.0416 - val_tp: 66.0000 - val_fp: 542.0000 - val_tn: 44954.0000 - val_fn: 7.0000 - val_accuracy: 0.9880 - val_precision: 0.1086 - val_recall: 0.9041 - val_auc: 0.9873
Epoch 118/1000
20/20 [==============================] - 1s 31ms/step - loss: 0.0878 - tp: 19524.0000 - fp: 554.0000 - tn: 19781.0000 - fn: 1101.0000 - accuracy: 0.9596 - precision: 0.9724 - recall: 0.9466 - auc: 0.9957 - val_loss: 0.0415 - val_tp: 66.0000 - val_fp: 543.0000 - val_tn: 44953.0000 - val_fn: 7.0000 - val_accuracy: 0.9879 - val_precision: 0.1084 - val_recall: 0.9041 - val_auc: 0.9873
Epoch 119/1000
20/20 [==============================] - 1s 31ms/step - loss: 0.0862 - tp: 19269.0000 - fp: 528.0000 - tn: 20009.0000 - fn: 1154.0000 - accuracy: 0.9589 - precision: 0.9733 - recall: 0.9435 - auc: 0.9959 - val_loss: 0.0413 - val_tp: 66.0000 - val_fp: 546.0000 - val_tn: 44950.0000 - val_fn: 7.0000 - val_accuracy: 0.9879 - val_precision: 0.1078 - val_recall: 0.9041 - val_auc: 0.9873
Epoch 120/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.0846 - tp: 19342.0000 - fp: 525.0000 - tn: 19976.0000 - fn: 1117.0000 - accuracy: 0.9599 - precision: 0.9736 - recall: 0.9454 - auc: 0.9961 - val_loss: 0.0411 - val_tp: 66.0000 - val_fp: 538.0000 - val_tn: 44958.0000 - val_fn: 7.0000 - val_accuracy: 0.9880 - val_precision: 0.1093 - val_recall: 0.9041 - val_auc: 0.9872
Epoch 121/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.0835 - tp: 19277.0000 - fp: 505.0000 - tn: 20009.0000 - fn: 1169.0000 - accuracy: 0.9591 - precision: 0.9745 - recall: 0.9428 - auc: 0.9962 - val_loss: 0.0409 - val_tp: 66.0000 - val_fp: 539.0000 - val_tn: 44957.0000 - val_fn: 7.0000 - val_accuracy: 0.9880 - val_precision: 0.1091 - val_recall: 0.9041 - val_auc: 0.9872
Epoch 122/1000
20/20 [==============================] - 1s 29ms/step - loss: 0.0843 - tp: 19233.0000 - fp: 510.0000 - tn: 20053.0000 - fn: 1164.0000 - accuracy: 0.9591 - precision: 0.9742 - recall: 0.9429 - auc: 0.9962 - val_loss: 0.0408 - val_tp: 66.0000 - val_fp: 540.0000 - val_tn: 44956.0000 - val_fn: 7.0000 - val_accuracy: 0.9880 - val_precision: 0.1089 - val_recall: 0.9041 - val_auc: 0.9872
Epoch 123/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.0855 - tp: 19246.0000 - fp: 531.0000 - tn: 20014.0000 - fn: 1169.0000 - accuracy: 0.9585 - precision: 0.9732 - recall: 0.9427 - auc: 0.9960 - val_loss: 0.0406 - val_tp: 66.0000 - val_fp: 533.0000 - val_tn: 44963.0000 - val_fn: 7.0000 - val_accuracy: 0.9881 - val_precision: 0.1102 - val_recall: 0.9041 - val_auc: 0.9872
Epoch 124/1000
20/20 [==============================] - 1s 28ms/step - loss: 0.0830 - tp: 19292.0000 - fp: 500.0000 - tn: 20037.0000 - fn: 1131.0000 - accuracy: 0.9602 - precision: 0.9747 - recall: 0.9446 - auc: 0.9963 - val_loss: 0.0403 - val_tp: 66.0000 - val_fp: 526.0000 - val_tn: 44970.0000 - val_fn: 7.0000 - val_accuracy: 0.9883 - val_precision: 0.1115 - val_recall: 0.9041 - val_auc: 0.9872
Epoch 125/1000
20/20 [==============================] - 1s 34ms/step - loss: 0.0846 - tp: 19329.0000 - fp: 512.0000 - tn: 20029.0000 - fn: 1090.0000 - accuracy: 0.9609 - precision: 0.9742 - recall: 0.9466 - auc: 0.9962 - val_loss: 0.0402 - val_tp: 66.0000 - val_fp: 526.0000 - val_tn: 44970.0000 - val_fn: 7.0000 - val_accuracy: 0.9883 - val_precision: 0.1115 - val_recall: 0.9041 - val_auc: 0.9872
Epoch 126/1000
20/20 [==============================] - 1s 30ms/step - loss: 0.0848 - tp: 19343.0000 - fp: 532.0000 - tn: 19929.0000 - fn: 1156.0000 - accuracy: 0.9588 - precision: 0.9732 - recall: 0.9436 - auc: 0.9961 - val_loss: 0.0407 - val_tp: 66.0000 - val_fp: 540.0000 - val_tn: 44956.0000 - val_fn: 7.0000 - val_accuracy: 0.9880 - val_precision: 0.1089 - val_recall: 0.9041 - val_auc: 0.9871
Epoch 127/1000
18/20 [==========================>...] - ETA: 0s - loss: 0.0836 - tp: 17590.0000 - fp: 513.0000 - tn: 17778.0000 - fn: 983.0000 - accuracy: 0.9594 - precision: 0.9717 - recall: 0.9471 - auc: 0.9961Restoring model weights from the end of the best epoch.
20/20 [==============================] - 1s 30ms/step - loss: 0.0840 - tp: 19583.0000 - fp: 568.0000 - tn: 19703.0000 - fn: 1106.0000 - accuracy: 0.9591 - precision: 0.9718 - recall: 0.9465 - auc: 0.9961 - val_loss: 0.0403 - val_tp: 66.0000 - val_fp: 532.0000 - val_tn: 44964.0000 - val_fn: 7.0000 - val_accuracy: 0.9882 - val_precision: 0.1104 - val_recall: 0.9041 - val_auc: 0.9871
Epoch 00127: early stopping

Re-check training history

plot_metrics(resampled_history)

Evaluate metrics

train_predictions_resampled = resampled_model.predict(train_features, batch_size=BATCH_SIZE)
test_predictions_resampled = resampled_model.predict(test_features, batch_size=BATCH_SIZE)

resampled_results = resampled_model.evaluate(test_features, test_labels,
                                             batch_size=BATCH_SIZE, verbose=0)
for name, value in zip(resampled_model.metrics_names, resampled_results):
  print(name, ': ', value)
print()

plot_cm(test_labels, test_predictions_weighted)

loss :  0.0429724511548488
tp :  76.0
fp :  673.0
tn :  56200.0
fn :  13.0
accuracy :  0.9879569
precision :  0.10146862
recall :  0.85393256
auc :  0.94155157

Legitimate Transactions Detected (True Negatives):  56162
Legitimate Transactions Incorrectly Detected (False Positives):  711
Fraudulent Transactions Missed (False Negatives):  14
Fraudulent Transactions Detected (True Positives):  75
Total Fraudulent Transactions:  89

Plot the ROC

plot_roc("Train Baseline", train_labels, train_predictions_baseline, color=colors[0])
plot_roc("Test Baseline", test_labels, test_predictions_baseline, color=colors[0], linestyle='--')

plot_roc("Train Weighted", train_labels, train_predictions_weighted, color=colors[1])
plot_roc("Test Weighted", test_labels, test_predictions_weighted, color=colors[1], linestyle='--')

plot_roc("Train Resampled", train_labels, train_predictions_resampled,  color=colors[2])
plot_roc("Test Resampled", test_labels, test_predictions_resampled,  color=colors[2], linestyle='--')
plt.legend(loc='lower right')

<matplotlib.legend.Legend at 0x7f5840568d30>

Applying this tutorial to your problem

Imbalanced data classification is an inherantly difficult task since there are so few samples to learn from. You should always start with the data first and do your best to collect as many samples as possible and give substantial thought to what features may be relevant so the model can get the most out of your minority class. At some point your model may struggle to improve and yield the results you want, so it is important to keep in mind the context of your problem and the trade offs between different types of errors.