How to build a simple text classifier with TF-Hub

Licensed under the Apache License, Version 2.0 (the "License");

# Copyright 2018 The TensorFlow Hub Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
Run in Google Colab View source on GitHub

TF-Hub is a platform to share machine learning expertise packaged in reusable resources, notably pre-trained modules. This tutorial is organized into two main parts.

** Introduction: Training a text classifier with TF-Hub**

We will use a TF-Hub text embedding module to train a simple sentiment classifier with a reasonable baseline accuracy. We will then analyze the predictions to make sure our model is reasonable and propose improvements to increase the accuracy.

** Advanced: Transfer learning analysis **

In this section, we will use various TF-Hub modules to compare their effect on the accuracy of the estimator and demonstrate advantages and pitfalls of transfer learning.

Optional prerequisites

Preparing the environment

# Install TF-Hub.
!pip install -q tensorflow==2.1.0
!pip install -q tensorflow-hub
!pip install -q seaborn

More detailed information about installing Tensorflow can be found at https://www.tensorflow.org/install/.

from absl import logging

import tensorflow as tf
import tensorflow_hub as hub
import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import re
import seaborn as sns

Getting started

Data

We will try to solve the Large Movie Review Dataset v1.0 task (Mass et al., 2011). The dataset consists of IMDB movie reviews labeled by positivity from 1 to 10. The task is to label the reviews as negative or positive.

# Load all files from a directory in a DataFrame.
def load_directory_data(directory):
  data = {}
  data["sentence"] = []
  data["sentiment"] = []
  for file_path in os.listdir(directory):
    with tf.io.gfile.GFile(os.path.join(directory, file_path), "r") as f:
      data["sentence"].append(f.read())
      data["sentiment"].append(re.match("\d+_(\d+)\.txt", file_path).group(1))
  return pd.DataFrame.from_dict(data)

# Merge positive and negative examples, add a polarity column and shuffle.
def load_dataset(directory):
  pos_df = load_directory_data(os.path.join(directory, "pos"))
  neg_df = load_directory_data(os.path.join(directory, "neg"))
  pos_df["polarity"] = 1
  neg_df["polarity"] = 0
  return pd.concat([pos_df, neg_df]).sample(frac=1).reset_index(drop=True)

# Download and process the dataset files.
def download_and_load_datasets(force_download=False):
  dataset = tf.keras.utils.get_file(
      fname="aclImdb.tar.gz", 
      origin="http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz", 
      extract=True)
  
  train_df = load_dataset(os.path.join(os.path.dirname(dataset), 
                                       "aclImdb", "train"))
  test_df = load_dataset(os.path.join(os.path.dirname(dataset), 
                                      "aclImdb", "test"))
  
  return train_df, test_df

# Reduce logging output.
logging.set_verbosity(logging.ERROR)

train_df, test_df = download_and_load_datasets()
train_df.head()
Downloading data from http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz
84131840/84125825 [==============================] - 4s 0us/step

Model

Input functions

Estimator framework provides input functions that wrap Pandas dataframes.

# Training input on the whole training set with no limit on training epochs.
train_input_fn = tf.compat.v1.estimator.inputs.pandas_input_fn(
    train_df, train_df["polarity"], num_epochs=None, shuffle=True)

# Prediction on the whole training set.
predict_train_input_fn = tf.compat.v1.estimator.inputs.pandas_input_fn(
    train_df, train_df["polarity"], shuffle=False)
# Prediction on the test set.
predict_test_input_fn = tf.compat.v1.estimator.inputs.pandas_input_fn(
    test_df, test_df["polarity"], shuffle=False)

Feature columns

TF-Hub provides a feature column that applies a module on the given text feature and passes further the outputs of the module. In this tutorial we will be using the nnlm-en-dim128 module. For the purpose of this tutorial, the most important facts are:

  • The module takes a batch of sentences in a 1-D tensor of strings as input.
  • The module is responsible for preprocessing of sentences (e.g. removal of punctuation and splitting on spaces).
  • The module works with any input (e.g. nnlm-en-dim128 hashes words not present in vocabulary into ~20.000 buckets).
embedded_text_feature_column = hub.text_embedding_column(
    key="sentence", 
    module_spec="https://tfhub.dev/google/nnlm-en-dim128/1")

Estimator

For classification we can use a DNN Classifier (note further remarks about different modelling of the label function at the end of the tutorial).

estimator = tf.estimator.DNNClassifier(
    hidden_units=[500, 100],
    feature_columns=[embedded_text_feature_column],
    n_classes=2,
    optimizer=tf.keras.optimizers.Adagrad(lr=0.003))
INFO:tensorflow:Using default config.

INFO:tensorflow:Using default config.

WARNING:tensorflow:Using temporary folder as model directory: /tmpfs/tmp/tmph3gv3f5_

WARNING:tensorflow:Using temporary folder as model directory: /tmpfs/tmp/tmph3gv3f5_

INFO:tensorflow:Using config: {'_is_chief': True, '_num_worker_replicas': 1, '_experimental_distribute': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_evaluation_master': '', '_global_id_in_cluster': 0, '_master': '', '_experimental_max_worker_delay_secs': None, '_cluster_spec': ClusterSpec({}), '_task_id': 0, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_task_type': 'worker', '_model_dir': '/tmpfs/tmp/tmph3gv3f5_', '_service': None, '_session_creation_timeout_secs': 7200, '_eval_distribute': None, '_save_checkpoints_steps': None, '_device_fn': None, '_train_distribute': None, '_keep_checkpoint_max': 5, '_num_ps_replicas': 0, '_protocol': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_tf_random_seed': None}

INFO:tensorflow:Using config: {'_is_chief': True, '_num_worker_replicas': 1, '_experimental_distribute': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_evaluation_master': '', '_global_id_in_cluster': 0, '_master': '', '_experimental_max_worker_delay_secs': None, '_cluster_spec': ClusterSpec({}), '_task_id': 0, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_task_type': 'worker', '_model_dir': '/tmpfs/tmp/tmph3gv3f5_', '_service': None, '_session_creation_timeout_secs': 7200, '_eval_distribute': None, '_save_checkpoints_steps': None, '_device_fn': None, '_train_distribute': None, '_keep_checkpoint_max': 5, '_num_ps_replicas': 0, '_protocol': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_tf_random_seed': None}

Training

Train the estimator for a reasonable amount of steps.

# Training for 5,000 steps means 640,000 training examples with the default
# batch size. This is roughly equivalent to 25 epochs since the training dataset
# contains 25,000 examples.
estimator.train(input_fn=train_input_fn, steps=5000);
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1635: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1635: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/inputs/queues/feeding_queue_runner.py:62: QueueRunner.__init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/inputs/queues/feeding_queue_runner.py:62: QueueRunner.__init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/inputs/queues/feeding_functions.py:500: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/inputs/queues/feeding_functions.py:500: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb086b9978>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb086b9978>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb086b9978>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/keras/optimizer_v2/adagrad.py:103: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/keras/optimizer_v2/adagrad.py:103: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/training/monitored_session.py:906: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/training/monitored_session.py:906: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.

INFO:tensorflow:Saving checkpoints for 0 into /tmpfs/tmp/tmph3gv3f5_/model.ckpt.

INFO:tensorflow:Saving checkpoints for 0 into /tmpfs/tmp/tmph3gv3f5_/model.ckpt.

INFO:tensorflow:loss = 0.7008953, step = 0

INFO:tensorflow:loss = 0.7008953, step = 0

INFO:tensorflow:global_step/sec: 47.4007

INFO:tensorflow:global_step/sec: 47.4007

INFO:tensorflow:loss = 0.68467057, step = 100 (2.112 sec)

INFO:tensorflow:loss = 0.68467057, step = 100 (2.112 sec)

INFO:tensorflow:global_step/sec: 50.1688

INFO:tensorflow:global_step/sec: 50.1688

INFO:tensorflow:loss = 0.68206036, step = 200 (1.993 sec)

INFO:tensorflow:loss = 0.68206036, step = 200 (1.993 sec)

INFO:tensorflow:global_step/sec: 49.7549

INFO:tensorflow:global_step/sec: 49.7549

INFO:tensorflow:loss = 0.65831923, step = 300 (2.010 sec)

INFO:tensorflow:loss = 0.65831923, step = 300 (2.010 sec)

INFO:tensorflow:global_step/sec: 49.7185

INFO:tensorflow:global_step/sec: 49.7185

INFO:tensorflow:loss = 0.6562337, step = 400 (2.012 sec)

INFO:tensorflow:loss = 0.6562337, step = 400 (2.012 sec)

INFO:tensorflow:global_step/sec: 49.681

INFO:tensorflow:global_step/sec: 49.681

INFO:tensorflow:loss = 0.65498376, step = 500 (2.013 sec)

INFO:tensorflow:loss = 0.65498376, step = 500 (2.013 sec)

INFO:tensorflow:global_step/sec: 50.6651

INFO:tensorflow:global_step/sec: 50.6651

INFO:tensorflow:loss = 0.6167938, step = 600 (1.974 sec)

INFO:tensorflow:loss = 0.6167938, step = 600 (1.974 sec)

INFO:tensorflow:global_step/sec: 49.9893

INFO:tensorflow:global_step/sec: 49.9893

INFO:tensorflow:loss = 0.63133186, step = 700 (2.000 sec)

INFO:tensorflow:loss = 0.63133186, step = 700 (2.000 sec)

INFO:tensorflow:global_step/sec: 51.4674

INFO:tensorflow:global_step/sec: 51.4674

INFO:tensorflow:loss = 0.60965127, step = 800 (1.943 sec)

INFO:tensorflow:loss = 0.60965127, step = 800 (1.943 sec)

INFO:tensorflow:global_step/sec: 52.7786

INFO:tensorflow:global_step/sec: 52.7786

INFO:tensorflow:loss = 0.58765227, step = 900 (1.895 sec)

INFO:tensorflow:loss = 0.58765227, step = 900 (1.895 sec)

INFO:tensorflow:global_step/sec: 53.3964

INFO:tensorflow:global_step/sec: 53.3964

INFO:tensorflow:loss = 0.5886106, step = 1000 (1.872 sec)

INFO:tensorflow:loss = 0.5886106, step = 1000 (1.872 sec)

INFO:tensorflow:global_step/sec: 52.537

INFO:tensorflow:global_step/sec: 52.537

INFO:tensorflow:loss = 0.5865797, step = 1100 (1.904 sec)

INFO:tensorflow:loss = 0.5865797, step = 1100 (1.904 sec)

INFO:tensorflow:global_step/sec: 52.7445

INFO:tensorflow:global_step/sec: 52.7445

INFO:tensorflow:loss = 0.58508927, step = 1200 (1.896 sec)

INFO:tensorflow:loss = 0.58508927, step = 1200 (1.896 sec)

INFO:tensorflow:global_step/sec: 52.8678

INFO:tensorflow:global_step/sec: 52.8678

INFO:tensorflow:loss = 0.5036612, step = 1300 (1.892 sec)

INFO:tensorflow:loss = 0.5036612, step = 1300 (1.892 sec)

INFO:tensorflow:global_step/sec: 51.9761

INFO:tensorflow:global_step/sec: 51.9761

INFO:tensorflow:loss = 0.53618413, step = 1400 (1.924 sec)

INFO:tensorflow:loss = 0.53618413, step = 1400 (1.924 sec)

INFO:tensorflow:global_step/sec: 53.2584

INFO:tensorflow:global_step/sec: 53.2584

INFO:tensorflow:loss = 0.57962996, step = 1500 (1.877 sec)

INFO:tensorflow:loss = 0.57962996, step = 1500 (1.877 sec)

INFO:tensorflow:global_step/sec: 53.6031

INFO:tensorflow:global_step/sec: 53.6031

INFO:tensorflow:loss = 0.57044566, step = 1600 (1.866 sec)

INFO:tensorflow:loss = 0.57044566, step = 1600 (1.866 sec)

INFO:tensorflow:global_step/sec: 53.6832

INFO:tensorflow:global_step/sec: 53.6832

INFO:tensorflow:loss = 0.52558756, step = 1700 (1.863 sec)

INFO:tensorflow:loss = 0.52558756, step = 1700 (1.863 sec)

INFO:tensorflow:global_step/sec: 52.6201

INFO:tensorflow:global_step/sec: 52.6201

INFO:tensorflow:loss = 0.59720576, step = 1800 (1.900 sec)

INFO:tensorflow:loss = 0.59720576, step = 1800 (1.900 sec)

INFO:tensorflow:global_step/sec: 51.1092

INFO:tensorflow:global_step/sec: 51.1092

INFO:tensorflow:loss = 0.4651407, step = 1900 (1.956 sec)

INFO:tensorflow:loss = 0.4651407, step = 1900 (1.956 sec)

INFO:tensorflow:global_step/sec: 54.0516

INFO:tensorflow:global_step/sec: 54.0516

INFO:tensorflow:loss = 0.4980325, step = 2000 (1.850 sec)

INFO:tensorflow:loss = 0.4980325, step = 2000 (1.850 sec)

INFO:tensorflow:global_step/sec: 53.8195

INFO:tensorflow:global_step/sec: 53.8195

INFO:tensorflow:loss = 0.55977875, step = 2100 (1.858 sec)

INFO:tensorflow:loss = 0.55977875, step = 2100 (1.858 sec)

INFO:tensorflow:global_step/sec: 54.2267

INFO:tensorflow:global_step/sec: 54.2267

INFO:tensorflow:loss = 0.49089116, step = 2200 (1.844 sec)

INFO:tensorflow:loss = 0.49089116, step = 2200 (1.844 sec)

INFO:tensorflow:global_step/sec: 53.3347

INFO:tensorflow:global_step/sec: 53.3347

INFO:tensorflow:loss = 0.5826153, step = 2300 (1.875 sec)

INFO:tensorflow:loss = 0.5826153, step = 2300 (1.875 sec)

INFO:tensorflow:global_step/sec: 53.8183

INFO:tensorflow:global_step/sec: 53.8183

INFO:tensorflow:loss = 0.4870981, step = 2400 (1.858 sec)

INFO:tensorflow:loss = 0.4870981, step = 2400 (1.858 sec)

INFO:tensorflow:global_step/sec: 53.1715

INFO:tensorflow:global_step/sec: 53.1715

INFO:tensorflow:loss = 0.45260084, step = 2500 (1.880 sec)

INFO:tensorflow:loss = 0.45260084, step = 2500 (1.880 sec)

INFO:tensorflow:global_step/sec: 53.3813

INFO:tensorflow:global_step/sec: 53.3813

INFO:tensorflow:loss = 0.46953064, step = 2600 (1.873 sec)

INFO:tensorflow:loss = 0.46953064, step = 2600 (1.873 sec)

INFO:tensorflow:global_step/sec: 53.2633

INFO:tensorflow:global_step/sec: 53.2633

INFO:tensorflow:loss = 0.4320475, step = 2700 (1.878 sec)

INFO:tensorflow:loss = 0.4320475, step = 2700 (1.878 sec)

INFO:tensorflow:global_step/sec: 53.8371

INFO:tensorflow:global_step/sec: 53.8371

INFO:tensorflow:loss = 0.45182467, step = 2800 (1.857 sec)

INFO:tensorflow:loss = 0.45182467, step = 2800 (1.857 sec)

INFO:tensorflow:global_step/sec: 53.5949

INFO:tensorflow:global_step/sec: 53.5949

INFO:tensorflow:loss = 0.5146018, step = 2900 (1.866 sec)

INFO:tensorflow:loss = 0.5146018, step = 2900 (1.866 sec)

INFO:tensorflow:global_step/sec: 53.8318

INFO:tensorflow:global_step/sec: 53.8318

INFO:tensorflow:loss = 0.47607332, step = 3000 (1.858 sec)

INFO:tensorflow:loss = 0.47607332, step = 3000 (1.858 sec)

INFO:tensorflow:global_step/sec: 53.7416

INFO:tensorflow:global_step/sec: 53.7416

INFO:tensorflow:loss = 0.40731406, step = 3100 (1.861 sec)

INFO:tensorflow:loss = 0.40731406, step = 3100 (1.861 sec)

INFO:tensorflow:global_step/sec: 53.8502

INFO:tensorflow:global_step/sec: 53.8502

INFO:tensorflow:loss = 0.39230305, step = 3200 (1.856 sec)

INFO:tensorflow:loss = 0.39230305, step = 3200 (1.856 sec)

INFO:tensorflow:global_step/sec: 54.2415

INFO:tensorflow:global_step/sec: 54.2415

INFO:tensorflow:loss = 0.47128645, step = 3300 (1.844 sec)

INFO:tensorflow:loss = 0.47128645, step = 3300 (1.844 sec)

INFO:tensorflow:global_step/sec: 54.2846

INFO:tensorflow:global_step/sec: 54.2846

INFO:tensorflow:loss = 0.46608472, step = 3400 (1.842 sec)

INFO:tensorflow:loss = 0.46608472, step = 3400 (1.842 sec)

INFO:tensorflow:global_step/sec: 53.3936

INFO:tensorflow:global_step/sec: 53.3936

INFO:tensorflow:loss = 0.46813703, step = 3500 (1.873 sec)

INFO:tensorflow:loss = 0.46813703, step = 3500 (1.873 sec)

INFO:tensorflow:global_step/sec: 53.352

INFO:tensorflow:global_step/sec: 53.352

INFO:tensorflow:loss = 0.49467272, step = 3600 (1.875 sec)

INFO:tensorflow:loss = 0.49467272, step = 3600 (1.875 sec)

INFO:tensorflow:global_step/sec: 50.5386

INFO:tensorflow:global_step/sec: 50.5386

INFO:tensorflow:loss = 0.50456476, step = 3700 (1.980 sec)

INFO:tensorflow:loss = 0.50456476, step = 3700 (1.980 sec)

INFO:tensorflow:global_step/sec: 49.1338

INFO:tensorflow:global_step/sec: 49.1338

INFO:tensorflow:loss = 0.4356554, step = 3800 (2.034 sec)

INFO:tensorflow:loss = 0.4356554, step = 3800 (2.034 sec)

INFO:tensorflow:global_step/sec: 49.9271

INFO:tensorflow:global_step/sec: 49.9271

INFO:tensorflow:loss = 0.44263726, step = 3900 (2.002 sec)

INFO:tensorflow:loss = 0.44263726, step = 3900 (2.002 sec)

INFO:tensorflow:global_step/sec: 49.7177

INFO:tensorflow:global_step/sec: 49.7177

INFO:tensorflow:loss = 0.44143853, step = 4000 (2.011 sec)

INFO:tensorflow:loss = 0.44143853, step = 4000 (2.011 sec)

INFO:tensorflow:global_step/sec: 50.4059

INFO:tensorflow:global_step/sec: 50.4059

INFO:tensorflow:loss = 0.49029818, step = 4100 (1.984 sec)

INFO:tensorflow:loss = 0.49029818, step = 4100 (1.984 sec)

INFO:tensorflow:global_step/sec: 49.7241

INFO:tensorflow:global_step/sec: 49.7241

INFO:tensorflow:loss = 0.45970577, step = 4200 (2.012 sec)

INFO:tensorflow:loss = 0.45970577, step = 4200 (2.012 sec)

INFO:tensorflow:global_step/sec: 49.4132

INFO:tensorflow:global_step/sec: 49.4132

INFO:tensorflow:loss = 0.4127702, step = 4300 (2.022 sec)

INFO:tensorflow:loss = 0.4127702, step = 4300 (2.022 sec)

INFO:tensorflow:global_step/sec: 49.3703

INFO:tensorflow:global_step/sec: 49.3703

INFO:tensorflow:loss = 0.45625564, step = 4400 (2.026 sec)

INFO:tensorflow:loss = 0.45625564, step = 4400 (2.026 sec)

INFO:tensorflow:global_step/sec: 49.8603

INFO:tensorflow:global_step/sec: 49.8603

INFO:tensorflow:loss = 0.52669656, step = 4500 (2.006 sec)

INFO:tensorflow:loss = 0.52669656, step = 4500 (2.006 sec)

INFO:tensorflow:global_step/sec: 49.5914

INFO:tensorflow:global_step/sec: 49.5914

INFO:tensorflow:loss = 0.4419571, step = 4600 (2.018 sec)

INFO:tensorflow:loss = 0.4419571, step = 4600 (2.018 sec)

INFO:tensorflow:global_step/sec: 48.5683

INFO:tensorflow:global_step/sec: 48.5683

INFO:tensorflow:loss = 0.47491738, step = 4700 (2.057 sec)

INFO:tensorflow:loss = 0.47491738, step = 4700 (2.057 sec)

INFO:tensorflow:global_step/sec: 48.8431

INFO:tensorflow:global_step/sec: 48.8431

INFO:tensorflow:loss = 0.47170496, step = 4800 (2.047 sec)

INFO:tensorflow:loss = 0.47170496, step = 4800 (2.047 sec)

INFO:tensorflow:global_step/sec: 50.6801

INFO:tensorflow:global_step/sec: 50.6801

INFO:tensorflow:loss = 0.52786267, step = 4900 (1.974 sec)

INFO:tensorflow:loss = 0.52786267, step = 4900 (1.974 sec)

INFO:tensorflow:Saving checkpoints for 5000 into /tmpfs/tmp/tmph3gv3f5_/model.ckpt.

INFO:tensorflow:Saving checkpoints for 5000 into /tmpfs/tmp/tmph3gv3f5_/model.ckpt.

INFO:tensorflow:Loss for final step: 0.38206714.

INFO:tensorflow:Loss for final step: 0.38206714.

Prediction

Run predictions for both training and test set.

train_eval_result = estimator.evaluate(input_fn=predict_train_input_fn)
test_eval_result = estimator.evaluate(input_fn=predict_test_input_fn)

print("Training set accuracy: {accuracy}".format(**train_eval_result))
print("Test set accuracy: {accuracy}".format(**test_eval_result))
INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0a41b630>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0a41b630>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0a41b630>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2020-01-16T06:28:24Z

INFO:tensorflow:Starting evaluation at 2020-01-16T06:28:24Z

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmph3gv3f5_/model.ckpt-5000

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmph3gv3f5_/model.ckpt-5000

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Inference Time : 6.13037s

INFO:tensorflow:Inference Time : 6.13037s

INFO:tensorflow:Finished evaluation at 2020-01-16-06:28:30

INFO:tensorflow:Finished evaluation at 2020-01-16-06:28:30

INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.79492, accuracy_baseline = 0.5, auc = 0.8763598, auc_precision_recall = 0.87708974, average_loss = 0.44255307, global_step = 5000, label/mean = 0.5, loss = 0.44249496, precision = 0.79316103, prediction/mean = 0.50677323, recall = 0.79792

INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.79492, accuracy_baseline = 0.5, auc = 0.8763598, auc_precision_recall = 0.87708974, average_loss = 0.44255307, global_step = 5000, label/mean = 0.5, loss = 0.44249496, precision = 0.79316103, prediction/mean = 0.50677323, recall = 0.79792

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmpfs/tmp/tmph3gv3f5_/model.ckpt-5000

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmpfs/tmp/tmph3gv3f5_/model.ckpt-5000

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb083e89b0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb083e89b0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb083e89b0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2020-01-16T06:28:32Z

INFO:tensorflow:Starting evaluation at 2020-01-16T06:28:32Z

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmph3gv3f5_/model.ckpt-5000

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmph3gv3f5_/model.ckpt-5000

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Inference Time : 6.04122s

INFO:tensorflow:Inference Time : 6.04122s

INFO:tensorflow:Finished evaluation at 2020-01-16-06:28:38

INFO:tensorflow:Finished evaluation at 2020-01-16-06:28:38

INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.78836, accuracy_baseline = 0.5, auc = 0.8707019, auc_precision_recall = 0.87288266, average_loss = 0.4507214, global_step = 5000, label/mean = 0.5, loss = 0.45059103, precision = 0.79099053, prediction/mean = 0.50262463, recall = 0.78384

INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.78836, accuracy_baseline = 0.5, auc = 0.8707019, auc_precision_recall = 0.87288266, average_loss = 0.4507214, global_step = 5000, label/mean = 0.5, loss = 0.45059103, precision = 0.79099053, prediction/mean = 0.50262463, recall = 0.78384

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmpfs/tmp/tmph3gv3f5_/model.ckpt-5000

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmpfs/tmp/tmph3gv3f5_/model.ckpt-5000

Training set accuracy: 0.794920027256012
Test set accuracy: 0.7883599996566772

Confusion matrix

We can visually check the confusion matrix to understand the distribution of misclassifications.

def get_predictions(estimator, input_fn):
  return [x["class_ids"][0] for x in estimator.predict(input_fn=input_fn)]

LABELS = [
    "negative", "positive"
]

# Create a confusion matrix on training data.
cm = tf.math.confusion_matrix(train_df["polarity"], 
                              get_predictions(estimator, predict_train_input_fn))

# Normalize the confusion matrix so that each row sums to 1.
cm = tf.cast(cm, dtype=tf.float32)
cm = cm / tf.math.reduce_sum(cm, axis=1)[:, np.newaxis]

sns.heatmap(cm, annot=True, xticklabels=LABELS, yticklabels=LABELS);
plt.xlabel("Predicted");
plt.ylabel("True");
INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb080abef0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb080abef0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb080abef0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmph3gv3f5_/model.ckpt-5000

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmph3gv3f5_/model.ckpt-5000

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

png

Further improvements

  1. Regression on sentiment: we used a classifier to assign each example into a polarity class. But we actually have another categorical feature at our disposal - sentiment. Here classes actually represent a scale and the underlying value (positive/negative) could be well mapped into a continuous range. We could make use of this property by computing a regression (DNN Regressor) instead of a classification (DNN Classifier).
  2. Larger module: for the purposes of this tutorial we used a small module to restrict the memory use. There are modules with larger vocabularies and larger embedding space that could give additional accuracy points.
  3. Parameter tuning: we can improve the accuracy by tuning the meta-parameters like the learning rate or the number of steps, especially if we use a different module. A validation set is very important if we want to get any reasonable results, because it is very easy to set-up a model that learns to predict the training data without generalizing well to the test set.
  4. More complex model: we used a module that computes a sentence embedding by embedding each individual word and then combining them with average. One could also use a sequential module (e.g. Universal Sentence Encoder module) to better capture the nature of sentences. Or an ensemble of two or more TF-Hub modules.
  5. Regularization: to prevent overfitting, we could try to use an optimizer that does some sort of regularization, for example Proximal Adagrad Optimizer.

Advanced: Transfer learning analysis

Transfer learning makes it possible to save training resources and to achieve good model generalization even when training on a small dataset. In this part, we will demonstrate this by training with two different TF-Hub modules:

  • nnlm-en-dim128 - pretrained text embedding module,
  • random-nnlm-en-dim128 - text embedding module that has same vocabulary and network as nnlm-en-dim128, but the weights were just randomly initialized and never trained on real data.

And by training in two modes:

  • training only the classifier (i.e. freezing the module), and
  • training the classifier together with the module.

Let's run a couple of trainings and evaluations to see how using a various modules can affect the accuracy.

def train_and_evaluate_with_module(hub_module, train_module=False):
  embedded_text_feature_column = hub.text_embedding_column(
      key="sentence", module_spec=hub_module, trainable=train_module)

  estimator = tf.estimator.DNNClassifier(
      hidden_units=[500, 100],
      feature_columns=[embedded_text_feature_column],
      n_classes=2,
      optimizer=tf.keras.optimizers.Adagrad(learning_rate=0.003))

  estimator.train(input_fn=train_input_fn, steps=1000)

  train_eval_result = estimator.evaluate(input_fn=predict_train_input_fn)
  test_eval_result = estimator.evaluate(input_fn=predict_test_input_fn)

  training_set_accuracy = train_eval_result["accuracy"]
  test_set_accuracy = test_eval_result["accuracy"]

  return {
      "Training accuracy": training_set_accuracy,
      "Test accuracy": test_set_accuracy
  }


results = {}
results["nnlm-en-dim128"] = train_and_evaluate_with_module(
    "https://tfhub.dev/google/nnlm-en-dim128/1")
results["nnlm-en-dim128-with-module-training"] = train_and_evaluate_with_module(
    "https://tfhub.dev/google/nnlm-en-dim128/1", True)
results["random-nnlm-en-dim128"] = train_and_evaluate_with_module(
    "https://tfhub.dev/google/random-nnlm-en-dim128/1")
results["random-nnlm-en-dim128-with-module-training"] = train_and_evaluate_with_module(
    "https://tfhub.dev/google/random-nnlm-en-dim128/1", True)
INFO:tensorflow:Using default config.

INFO:tensorflow:Using default config.

WARNING:tensorflow:Using temporary folder as model directory: /tmpfs/tmp/tmpm5f4wofc

WARNING:tensorflow:Using temporary folder as model directory: /tmpfs/tmp/tmpm5f4wofc

INFO:tensorflow:Using config: {'_is_chief': True, '_num_worker_replicas': 1, '_experimental_distribute': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_evaluation_master': '', '_global_id_in_cluster': 0, '_master': '', '_experimental_max_worker_delay_secs': None, '_cluster_spec': ClusterSpec({}), '_task_id': 0, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_task_type': 'worker', '_model_dir': '/tmpfs/tmp/tmpm5f4wofc', '_service': None, '_session_creation_timeout_secs': 7200, '_eval_distribute': None, '_save_checkpoints_steps': None, '_device_fn': None, '_train_distribute': None, '_keep_checkpoint_max': 5, '_num_ps_replicas': 0, '_protocol': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_tf_random_seed': None}

INFO:tensorflow:Using config: {'_is_chief': True, '_num_worker_replicas': 1, '_experimental_distribute': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_evaluation_master': '', '_global_id_in_cluster': 0, '_master': '', '_experimental_max_worker_delay_secs': None, '_cluster_spec': ClusterSpec({}), '_task_id': 0, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_task_type': 'worker', '_model_dir': '/tmpfs/tmp/tmpm5f4wofc', '_service': None, '_session_creation_timeout_secs': 7200, '_eval_distribute': None, '_save_checkpoints_steps': None, '_device_fn': None, '_train_distribute': None, '_keep_checkpoint_max': 5, '_num_ps_replicas': 0, '_protocol': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_tf_random_seed': None}

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb081fda58>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb081fda58>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb081fda58>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Saving checkpoints for 0 into /tmpfs/tmp/tmpm5f4wofc/model.ckpt.

INFO:tensorflow:Saving checkpoints for 0 into /tmpfs/tmp/tmpm5f4wofc/model.ckpt.

INFO:tensorflow:loss = 0.7046465, step = 0

INFO:tensorflow:loss = 0.7046465, step = 0

INFO:tensorflow:global_step/sec: 47.9173

INFO:tensorflow:global_step/sec: 47.9173

INFO:tensorflow:loss = 0.6803429, step = 100 (2.090 sec)

INFO:tensorflow:loss = 0.6803429, step = 100 (2.090 sec)

INFO:tensorflow:global_step/sec: 47.5791

INFO:tensorflow:global_step/sec: 47.5791

INFO:tensorflow:loss = 0.6779837, step = 200 (2.101 sec)

INFO:tensorflow:loss = 0.6779837, step = 200 (2.101 sec)

INFO:tensorflow:global_step/sec: 49.3587

INFO:tensorflow:global_step/sec: 49.3587

INFO:tensorflow:loss = 0.66039574, step = 300 (2.026 sec)

INFO:tensorflow:loss = 0.66039574, step = 300 (2.026 sec)

INFO:tensorflow:global_step/sec: 49.8901

INFO:tensorflow:global_step/sec: 49.8901

INFO:tensorflow:loss = 0.66115904, step = 400 (2.005 sec)

INFO:tensorflow:loss = 0.66115904, step = 400 (2.005 sec)

INFO:tensorflow:global_step/sec: 48.9894

INFO:tensorflow:global_step/sec: 48.9894

INFO:tensorflow:loss = 0.6484684, step = 500 (2.040 sec)

INFO:tensorflow:loss = 0.6484684, step = 500 (2.040 sec)

INFO:tensorflow:global_step/sec: 49.55

INFO:tensorflow:global_step/sec: 49.55

INFO:tensorflow:loss = 0.64258116, step = 600 (2.018 sec)

INFO:tensorflow:loss = 0.64258116, step = 600 (2.018 sec)

INFO:tensorflow:global_step/sec: 49.2344

INFO:tensorflow:global_step/sec: 49.2344

INFO:tensorflow:loss = 0.6248789, step = 700 (2.031 sec)

INFO:tensorflow:loss = 0.6248789, step = 700 (2.031 sec)

INFO:tensorflow:global_step/sec: 50.4988

INFO:tensorflow:global_step/sec: 50.4988

INFO:tensorflow:loss = 0.63769805, step = 800 (1.980 sec)

INFO:tensorflow:loss = 0.63769805, step = 800 (1.980 sec)

INFO:tensorflow:global_step/sec: 49.6485

INFO:tensorflow:global_step/sec: 49.6485

INFO:tensorflow:loss = 0.6133951, step = 900 (2.014 sec)

INFO:tensorflow:loss = 0.6133951, step = 900 (2.014 sec)

INFO:tensorflow:Saving checkpoints for 1000 into /tmpfs/tmp/tmpm5f4wofc/model.ckpt.

INFO:tensorflow:Saving checkpoints for 1000 into /tmpfs/tmp/tmpm5f4wofc/model.ckpt.

INFO:tensorflow:Loss for final step: 0.5814983.

INFO:tensorflow:Loss for final step: 0.5814983.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb083c3978>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb083c3978>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb083c3978>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2020-01-16T06:29:10Z

INFO:tensorflow:Starting evaluation at 2020-01-16T06:29:10Z

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmpm5f4wofc/model.ckpt-1000

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmpm5f4wofc/model.ckpt-1000

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Inference Time : 6.11136s

INFO:tensorflow:Inference Time : 6.11136s

INFO:tensorflow:Finished evaluation at 2020-01-16-06:29:16

INFO:tensorflow:Finished evaluation at 2020-01-16-06:29:16

INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.72588, accuracy_baseline = 0.5, auc = 0.8010811, auc_precision_recall = 0.80198634, average_loss = 0.5980666, global_step = 1000, label/mean = 0.5, loss = 0.5979126, precision = 0.73929995, prediction/mean = 0.49692547, recall = 0.69784

INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.72588, accuracy_baseline = 0.5, auc = 0.8010811, auc_precision_recall = 0.80198634, average_loss = 0.5980666, global_step = 1000, label/mean = 0.5, loss = 0.5979126, precision = 0.73929995, prediction/mean = 0.49692547, recall = 0.69784

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmpfs/tmp/tmpm5f4wofc/model.ckpt-1000

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmpfs/tmp/tmpm5f4wofc/model.ckpt-1000

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0849a668>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0849a668>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0849a668>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2020-01-16T06:29:17Z

INFO:tensorflow:Starting evaluation at 2020-01-16T06:29:17Z

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmpm5f4wofc/model.ckpt-1000

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmpm5f4wofc/model.ckpt-1000

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Inference Time : 6.14494s

INFO:tensorflow:Inference Time : 6.14494s

INFO:tensorflow:Finished evaluation at 2020-01-16-06:29:23

INFO:tensorflow:Finished evaluation at 2020-01-16-06:29:23

INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.71976, accuracy_baseline = 0.5, auc = 0.79239714, auc_precision_recall = 0.79267925, average_loss = 0.6020692, global_step = 1000, label/mean = 0.5, loss = 0.60203844, precision = 0.73721933, prediction/mean = 0.49446532, recall = 0.68296

INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.71976, accuracy_baseline = 0.5, auc = 0.79239714, auc_precision_recall = 0.79267925, average_loss = 0.6020692, global_step = 1000, label/mean = 0.5, loss = 0.60203844, precision = 0.73721933, prediction/mean = 0.49446532, recall = 0.68296

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmpfs/tmp/tmpm5f4wofc/model.ckpt-1000

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmpfs/tmp/tmpm5f4wofc/model.ckpt-1000

INFO:tensorflow:Using default config.

INFO:tensorflow:Using default config.

WARNING:tensorflow:Using temporary folder as model directory: /tmpfs/tmp/tmphftle488

WARNING:tensorflow:Using temporary folder as model directory: /tmpfs/tmp/tmphftle488

INFO:tensorflow:Using config: {'_is_chief': True, '_num_worker_replicas': 1, '_experimental_distribute': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_evaluation_master': '', '_global_id_in_cluster': 0, '_master': '', '_experimental_max_worker_delay_secs': None, '_cluster_spec': ClusterSpec({}), '_task_id': 0, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_task_type': 'worker', '_model_dir': '/tmpfs/tmp/tmphftle488', '_service': None, '_session_creation_timeout_secs': 7200, '_eval_distribute': None, '_save_checkpoints_steps': None, '_device_fn': None, '_train_distribute': None, '_keep_checkpoint_max': 5, '_num_ps_replicas': 0, '_protocol': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_tf_random_seed': None}

INFO:tensorflow:Using config: {'_is_chief': True, '_num_worker_replicas': 1, '_experimental_distribute': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_evaluation_master': '', '_global_id_in_cluster': 0, '_master': '', '_experimental_max_worker_delay_secs': None, '_cluster_spec': ClusterSpec({}), '_task_id': 0, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_task_type': 'worker', '_model_dir': '/tmpfs/tmp/tmphftle488', '_service': None, '_session_creation_timeout_secs': 7200, '_eval_distribute': None, '_save_checkpoints_steps': None, '_device_fn': None, '_train_distribute': None, '_keep_checkpoint_max': 5, '_num_ps_replicas': 0, '_protocol': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_tf_random_seed': None}

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0407dd68>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0407dd68>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0407dd68>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Saving checkpoints for 0 into /tmpfs/tmp/tmphftle488/model.ckpt.

INFO:tensorflow:Saving checkpoints for 0 into /tmpfs/tmp/tmphftle488/model.ckpt.

INFO:tensorflow:loss = 0.71236825, step = 0

INFO:tensorflow:loss = 0.71236825, step = 0

INFO:tensorflow:global_step/sec: 46.6894

INFO:tensorflow:global_step/sec: 46.6894

INFO:tensorflow:loss = 0.6850661, step = 100 (2.145 sec)

INFO:tensorflow:loss = 0.6850661, step = 100 (2.145 sec)

INFO:tensorflow:global_step/sec: 50.5346

INFO:tensorflow:global_step/sec: 50.5346

INFO:tensorflow:loss = 0.67576575, step = 200 (1.978 sec)

INFO:tensorflow:loss = 0.67576575, step = 200 (1.978 sec)

INFO:tensorflow:global_step/sec: 49.6333

INFO:tensorflow:global_step/sec: 49.6333

INFO:tensorflow:loss = 0.65938205, step = 300 (2.015 sec)

INFO:tensorflow:loss = 0.65938205, step = 300 (2.015 sec)

INFO:tensorflow:global_step/sec: 50.876

INFO:tensorflow:global_step/sec: 50.876

INFO:tensorflow:loss = 0.66089565, step = 400 (1.965 sec)

INFO:tensorflow:loss = 0.66089565, step = 400 (1.965 sec)

INFO:tensorflow:global_step/sec: 49.9809

INFO:tensorflow:global_step/sec: 49.9809

INFO:tensorflow:loss = 0.644585, step = 500 (2.001 sec)

INFO:tensorflow:loss = 0.644585, step = 500 (2.001 sec)

INFO:tensorflow:global_step/sec: 49.7781

INFO:tensorflow:global_step/sec: 49.7781

INFO:tensorflow:loss = 0.63085157, step = 600 (2.009 sec)

INFO:tensorflow:loss = 0.63085157, step = 600 (2.009 sec)

INFO:tensorflow:global_step/sec: 50.4085

INFO:tensorflow:global_step/sec: 50.4085

INFO:tensorflow:loss = 0.6060908, step = 700 (1.984 sec)

INFO:tensorflow:loss = 0.6060908, step = 700 (1.984 sec)

INFO:tensorflow:global_step/sec: 48.6824

INFO:tensorflow:global_step/sec: 48.6824

INFO:tensorflow:loss = 0.60139287, step = 800 (2.054 sec)

INFO:tensorflow:loss = 0.60139287, step = 800 (2.054 sec)

INFO:tensorflow:global_step/sec: 49.3103

INFO:tensorflow:global_step/sec: 49.3103

INFO:tensorflow:loss = 0.5839482, step = 900 (2.028 sec)

INFO:tensorflow:loss = 0.5839482, step = 900 (2.028 sec)

INFO:tensorflow:Saving checkpoints for 1000 into /tmpfs/tmp/tmphftle488/model.ckpt.

INFO:tensorflow:Saving checkpoints for 1000 into /tmpfs/tmp/tmphftle488/model.ckpt.

INFO:tensorflow:Loss for final step: 0.5763911.

INFO:tensorflow:Loss for final step: 0.5763911.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb08157320>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb08157320>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb08157320>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2020-01-16T06:29:49Z

INFO:tensorflow:Starting evaluation at 2020-01-16T06:29:49Z

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmphftle488/model.ckpt-1000

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmphftle488/model.ckpt-1000

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Inference Time : 6.07036s

INFO:tensorflow:Inference Time : 6.07036s

INFO:tensorflow:Finished evaluation at 2020-01-16-06:29:55

INFO:tensorflow:Finished evaluation at 2020-01-16-06:29:55

INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.72184, accuracy_baseline = 0.5, auc = 0.7982874, auc_precision_recall = 0.79906416, average_loss = 0.5912257, global_step = 1000, label/mean = 0.5, loss = 0.5910517, precision = 0.72909784, prediction/mean = 0.50009686, recall = 0.706

INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.72184, accuracy_baseline = 0.5, auc = 0.7982874, auc_precision_recall = 0.79906416, average_loss = 0.5912257, global_step = 1000, label/mean = 0.5, loss = 0.5910517, precision = 0.72909784, prediction/mean = 0.50009686, recall = 0.706

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmpfs/tmp/tmphftle488/model.ckpt-1000

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmpfs/tmp/tmphftle488/model.ckpt-1000

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb07f96f28>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb07f96f28>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb07f96f28>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2020-01-16T06:29:57Z

INFO:tensorflow:Starting evaluation at 2020-01-16T06:29:57Z

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmphftle488/model.ckpt-1000

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmphftle488/model.ckpt-1000

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Inference Time : 6.06282s

INFO:tensorflow:Inference Time : 6.06282s

INFO:tensorflow:Finished evaluation at 2020-01-16-06:30:03

INFO:tensorflow:Finished evaluation at 2020-01-16-06:30:03

INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.71744, accuracy_baseline = 0.5, auc = 0.7899664, auc_precision_recall = 0.7884995, average_loss = 0.59602255, global_step = 1000, label/mean = 0.5, loss = 0.5959336, precision = 0.72859544, prediction/mean = 0.49723577, recall = 0.69304

INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.71744, accuracy_baseline = 0.5, auc = 0.7899664, auc_precision_recall = 0.7884995, average_loss = 0.59602255, global_step = 1000, label/mean = 0.5, loss = 0.5959336, precision = 0.72859544, prediction/mean = 0.49723577, recall = 0.69304

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmpfs/tmp/tmphftle488/model.ckpt-1000

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmpfs/tmp/tmphftle488/model.ckpt-1000

INFO:tensorflow:Using default config.

INFO:tensorflow:Using default config.

WARNING:tensorflow:Using temporary folder as model directory: /tmpfs/tmp/tmpeccjjfr6

WARNING:tensorflow:Using temporary folder as model directory: /tmpfs/tmp/tmpeccjjfr6

INFO:tensorflow:Using config: {'_is_chief': True, '_num_worker_replicas': 1, '_experimental_distribute': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_evaluation_master': '', '_global_id_in_cluster': 0, '_master': '', '_experimental_max_worker_delay_secs': None, '_cluster_spec': ClusterSpec({}), '_task_id': 0, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_task_type': 'worker', '_model_dir': '/tmpfs/tmp/tmpeccjjfr6', '_service': None, '_session_creation_timeout_secs': 7200, '_eval_distribute': None, '_save_checkpoints_steps': None, '_device_fn': None, '_train_distribute': None, '_keep_checkpoint_max': 5, '_num_ps_replicas': 0, '_protocol': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_tf_random_seed': None}

INFO:tensorflow:Using config: {'_is_chief': True, '_num_worker_replicas': 1, '_experimental_distribute': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_evaluation_master': '', '_global_id_in_cluster': 0, '_master': '', '_experimental_max_worker_delay_secs': None, '_cluster_spec': ClusterSpec({}), '_task_id': 0, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_task_type': 'worker', '_model_dir': '/tmpfs/tmp/tmpeccjjfr6', '_service': None, '_session_creation_timeout_secs': 7200, '_eval_distribute': None, '_save_checkpoints_steps': None, '_device_fn': None, '_train_distribute': None, '_keep_checkpoint_max': 5, '_num_ps_replicas': 0, '_protocol': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_tf_random_seed': None}

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0824a780>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0824a780>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0824a780>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Saving checkpoints for 0 into /tmpfs/tmp/tmpeccjjfr6/model.ckpt.

INFO:tensorflow:Saving checkpoints for 0 into /tmpfs/tmp/tmpeccjjfr6/model.ckpt.

INFO:tensorflow:loss = 0.82868284, step = 0

INFO:tensorflow:loss = 0.82868284, step = 0

INFO:tensorflow:global_step/sec: 47.3056

INFO:tensorflow:global_step/sec: 47.3056

INFO:tensorflow:loss = 0.6424987, step = 100 (2.116 sec)

INFO:tensorflow:loss = 0.6424987, step = 100 (2.116 sec)

INFO:tensorflow:global_step/sec: 49.0396

INFO:tensorflow:global_step/sec: 49.0396

INFO:tensorflow:loss = 0.6349132, step = 200 (2.039 sec)

INFO:tensorflow:loss = 0.6349132, step = 200 (2.039 sec)

INFO:tensorflow:global_step/sec: 50.0698

INFO:tensorflow:global_step/sec: 50.0698

INFO:tensorflow:loss = 0.6053715, step = 300 (1.997 sec)

INFO:tensorflow:loss = 0.6053715, step = 300 (1.997 sec)

INFO:tensorflow:global_step/sec: 51.3318

INFO:tensorflow:global_step/sec: 51.3318

INFO:tensorflow:loss = 0.6205816, step = 400 (1.949 sec)

INFO:tensorflow:loss = 0.6205816, step = 400 (1.949 sec)

INFO:tensorflow:global_step/sec: 50.5039

INFO:tensorflow:global_step/sec: 50.5039

INFO:tensorflow:loss = 0.63010323, step = 500 (1.980 sec)

INFO:tensorflow:loss = 0.63010323, step = 500 (1.980 sec)

INFO:tensorflow:global_step/sec: 50.7056

INFO:tensorflow:global_step/sec: 50.7056

INFO:tensorflow:loss = 0.6526009, step = 600 (1.972 sec)

INFO:tensorflow:loss = 0.6526009, step = 600 (1.972 sec)

INFO:tensorflow:global_step/sec: 49.1663

INFO:tensorflow:global_step/sec: 49.1663

INFO:tensorflow:loss = 0.55925095, step = 700 (2.034 sec)

INFO:tensorflow:loss = 0.55925095, step = 700 (2.034 sec)

INFO:tensorflow:global_step/sec: 48.7559

INFO:tensorflow:global_step/sec: 48.7559

INFO:tensorflow:loss = 0.57333636, step = 800 (2.051 sec)

INFO:tensorflow:loss = 0.57333636, step = 800 (2.051 sec)

INFO:tensorflow:global_step/sec: 49.4576

INFO:tensorflow:global_step/sec: 49.4576

INFO:tensorflow:loss = 0.5467, step = 900 (2.022 sec)

INFO:tensorflow:loss = 0.5467, step = 900 (2.022 sec)

INFO:tensorflow:Saving checkpoints for 1000 into /tmpfs/tmp/tmpeccjjfr6/model.ckpt.

INFO:tensorflow:Saving checkpoints for 1000 into /tmpfs/tmp/tmpeccjjfr6/model.ckpt.

INFO:tensorflow:Loss for final step: 0.5938266.

INFO:tensorflow:Loss for final step: 0.5938266.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb2c355630>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb2c355630>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb2c355630>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2020-01-16T06:31:10Z

INFO:tensorflow:Starting evaluation at 2020-01-16T06:31:10Z

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmpeccjjfr6/model.ckpt-1000

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmpeccjjfr6/model.ckpt-1000

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Inference Time : 6.17420s

INFO:tensorflow:Inference Time : 6.17420s

INFO:tensorflow:Finished evaluation at 2020-01-16-06:31:16

INFO:tensorflow:Finished evaluation at 2020-01-16-06:31:16

INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.673, accuracy_baseline = 0.5, auc = 0.74213636, auc_precision_recall = 0.7319623, average_loss = 0.60122967, global_step = 1000, label/mean = 0.5, loss = 0.60100406, precision = 0.7030707, prediction/mean = 0.46120778, recall = 0.59896

INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.673, accuracy_baseline = 0.5, auc = 0.74213636, auc_precision_recall = 0.7319623, average_loss = 0.60122967, global_step = 1000, label/mean = 0.5, loss = 0.60100406, precision = 0.7030707, prediction/mean = 0.46120778, recall = 0.59896

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmpfs/tmp/tmpeccjjfr6/model.ckpt-1000

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmpfs/tmp/tmpeccjjfr6/model.ckpt-1000

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0844cb00>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0844cb00>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0844cb00>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2020-01-16T06:31:18Z

INFO:tensorflow:Starting evaluation at 2020-01-16T06:31:18Z

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmpeccjjfr6/model.ckpt-1000

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmpeccjjfr6/model.ckpt-1000

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Inference Time : 6.09360s

INFO:tensorflow:Inference Time : 6.09360s

INFO:tensorflow:Finished evaluation at 2020-01-16-06:31:24

INFO:tensorflow:Finished evaluation at 2020-01-16-06:31:24

INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.66468, accuracy_baseline = 0.5, auc = 0.7266135, auc_precision_recall = 0.718404, average_loss = 0.61351275, global_step = 1000, label/mean = 0.5, loss = 0.61354905, precision = 0.69352263, prediction/mean = 0.46211663, recall = 0.59016

INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.66468, accuracy_baseline = 0.5, auc = 0.7266135, auc_precision_recall = 0.718404, average_loss = 0.61351275, global_step = 1000, label/mean = 0.5, loss = 0.61354905, precision = 0.69352263, prediction/mean = 0.46211663, recall = 0.59016

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmpfs/tmp/tmpeccjjfr6/model.ckpt-1000

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmpfs/tmp/tmpeccjjfr6/model.ckpt-1000

INFO:tensorflow:Using default config.

INFO:tensorflow:Using default config.

WARNING:tensorflow:Using temporary folder as model directory: /tmpfs/tmp/tmp_b1jpp_d

WARNING:tensorflow:Using temporary folder as model directory: /tmpfs/tmp/tmp_b1jpp_d

INFO:tensorflow:Using config: {'_is_chief': True, '_num_worker_replicas': 1, '_experimental_distribute': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_evaluation_master': '', '_global_id_in_cluster': 0, '_master': '', '_experimental_max_worker_delay_secs': None, '_cluster_spec': ClusterSpec({}), '_task_id': 0, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_task_type': 'worker', '_model_dir': '/tmpfs/tmp/tmp_b1jpp_d', '_service': None, '_session_creation_timeout_secs': 7200, '_eval_distribute': None, '_save_checkpoints_steps': None, '_device_fn': None, '_train_distribute': None, '_keep_checkpoint_max': 5, '_num_ps_replicas': 0, '_protocol': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_tf_random_seed': None}

INFO:tensorflow:Using config: {'_is_chief': True, '_num_worker_replicas': 1, '_experimental_distribute': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_evaluation_master': '', '_global_id_in_cluster': 0, '_master': '', '_experimental_max_worker_delay_secs': None, '_cluster_spec': ClusterSpec({}), '_task_id': 0, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_task_type': 'worker', '_model_dir': '/tmpfs/tmp/tmp_b1jpp_d', '_service': None, '_session_creation_timeout_secs': 7200, '_eval_distribute': None, '_save_checkpoints_steps': None, '_device_fn': None, '_train_distribute': None, '_keep_checkpoint_max': 5, '_num_ps_replicas': 0, '_protocol': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_tf_random_seed': None}

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0853df60>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0853df60>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0853df60>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Saving checkpoints for 0 into /tmpfs/tmp/tmp_b1jpp_d/model.ckpt.

INFO:tensorflow:Saving checkpoints for 0 into /tmpfs/tmp/tmp_b1jpp_d/model.ckpt.

INFO:tensorflow:loss = 0.79098344, step = 0

INFO:tensorflow:loss = 0.79098344, step = 0

INFO:tensorflow:global_step/sec: 47.4052

INFO:tensorflow:global_step/sec: 47.4052

INFO:tensorflow:loss = 0.64339375, step = 100 (2.112 sec)

INFO:tensorflow:loss = 0.64339375, step = 100 (2.112 sec)

INFO:tensorflow:global_step/sec: 51.004

INFO:tensorflow:global_step/sec: 51.004

INFO:tensorflow:loss = 0.59382355, step = 200 (1.960 sec)

INFO:tensorflow:loss = 0.59382355, step = 200 (1.960 sec)

INFO:tensorflow:global_step/sec: 50.7916

INFO:tensorflow:global_step/sec: 50.7916

INFO:tensorflow:loss = 0.6179811, step = 300 (1.969 sec)

INFO:tensorflow:loss = 0.6179811, step = 300 (1.969 sec)

INFO:tensorflow:global_step/sec: 50.235

INFO:tensorflow:global_step/sec: 50.235

INFO:tensorflow:loss = 0.6355841, step = 400 (1.991 sec)

INFO:tensorflow:loss = 0.6355841, step = 400 (1.991 sec)

INFO:tensorflow:global_step/sec: 50.3699

INFO:tensorflow:global_step/sec: 50.3699

INFO:tensorflow:loss = 0.6129422, step = 500 (1.986 sec)

INFO:tensorflow:loss = 0.6129422, step = 500 (1.986 sec)

INFO:tensorflow:global_step/sec: 50.3713

INFO:tensorflow:global_step/sec: 50.3713

INFO:tensorflow:loss = 0.6384422, step = 600 (1.984 sec)

INFO:tensorflow:loss = 0.6384422, step = 600 (1.984 sec)

INFO:tensorflow:global_step/sec: 49.9558

INFO:tensorflow:global_step/sec: 49.9558

INFO:tensorflow:loss = 0.62001014, step = 700 (2.002 sec)

INFO:tensorflow:loss = 0.62001014, step = 700 (2.002 sec)

INFO:tensorflow:global_step/sec: 50.1426

INFO:tensorflow:global_step/sec: 50.1426

INFO:tensorflow:loss = 0.61219966, step = 800 (1.994 sec)

INFO:tensorflow:loss = 0.61219966, step = 800 (1.994 sec)

INFO:tensorflow:global_step/sec: 50.5724

INFO:tensorflow:global_step/sec: 50.5724

INFO:tensorflow:loss = 0.54464835, step = 900 (1.977 sec)

INFO:tensorflow:loss = 0.54464835, step = 900 (1.977 sec)

INFO:tensorflow:Saving checkpoints for 1000 into /tmpfs/tmp/tmp_b1jpp_d/model.ckpt.

INFO:tensorflow:Saving checkpoints for 1000 into /tmpfs/tmp/tmp_b1jpp_d/model.ckpt.

INFO:tensorflow:Loss for final step: 0.57599986.

INFO:tensorflow:Loss for final step: 0.57599986.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0858cd30>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0858cd30>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb0858cd30>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2020-01-16T06:31:49Z

INFO:tensorflow:Starting evaluation at 2020-01-16T06:31:49Z

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmp_b1jpp_d/model.ckpt-1000

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmp_b1jpp_d/model.ckpt-1000

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Inference Time : 5.88881s

INFO:tensorflow:Inference Time : 5.88881s

INFO:tensorflow:Finished evaluation at 2020-01-16-06:31:55

INFO:tensorflow:Finished evaluation at 2020-01-16-06:31:55

INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.6796, accuracy_baseline = 0.5, auc = 0.745059, auc_precision_recall = 0.734466, average_loss = 0.5962193, global_step = 1000, label/mean = 0.5, loss = 0.5960689, precision = 0.67351985, prediction/mean = 0.5071719, recall = 0.69712

INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.6796, accuracy_baseline = 0.5, auc = 0.745059, auc_precision_recall = 0.734466, average_loss = 0.5962193, global_step = 1000, label/mean = 0.5, loss = 0.5960689, precision = 0.67351985, prediction/mean = 0.5071719, recall = 0.69712

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmpfs/tmp/tmp_b1jpp_d/model.ckpt-1000

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmpfs/tmp/tmp_b1jpp_d/model.ckpt-1000

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb08202048>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb08202048>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb08202048>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2020-01-16T06:31:56Z

INFO:tensorflow:Starting evaluation at 2020-01-16T06:31:56Z

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmp_b1jpp_d/model.ckpt-1000

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmp_b1jpp_d/model.ckpt-1000

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Inference Time : 5.79275s

INFO:tensorflow:Inference Time : 5.79275s

INFO:tensorflow:Finished evaluation at 2020-01-16-06:32:02

INFO:tensorflow:Finished evaluation at 2020-01-16-06:32:02

INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.66668, accuracy_baseline = 0.5, auc = 0.72791785, auc_precision_recall = 0.7184003, average_loss = 0.609179, global_step = 1000, label/mean = 0.5, loss = 0.60902804, precision = 0.66147405, prediction/mean = 0.506987, recall = 0.6828

INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.66668, accuracy_baseline = 0.5, auc = 0.72791785, auc_precision_recall = 0.7184003, average_loss = 0.609179, global_step = 1000, label/mean = 0.5, loss = 0.60902804, precision = 0.66147405, prediction/mean = 0.506987, recall = 0.6828

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmpfs/tmp/tmp_b1jpp_d/model.ckpt-1000

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmpfs/tmp/tmp_b1jpp_d/model.ckpt-1000

Let's look at the results.

pd.DataFrame.from_dict(results, orient="index")

We can already see some patterns, but first we should establish the baseline accuracy of the test set - the lower bound that can be achieved by outputting only the label of the most represented class:

estimator.evaluate(input_fn=predict_test_input_fn)["accuracy_baseline"]
INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb07f090f0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING:tensorflow:AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb07f090f0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'

WARNING: AutoGraph could not transform <bound method _DNNModelV2.call of <tensorflow_estimator.python.estimator.canned.dnn._DNNModelV2 object at 0x7fbb07f090f0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: attrib() got an unexpected keyword argument 'convert'
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2020-01-16T06:32:03Z

INFO:tensorflow:Starting evaluation at 2020-01-16T06:32:03Z

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmph3gv3f5_/model.ckpt-5000

INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmph3gv3f5_/model.ckpt-5000

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Inference Time : 5.94979s

INFO:tensorflow:Inference Time : 5.94979s

INFO:tensorflow:Finished evaluation at 2020-01-16-06:32:09

INFO:tensorflow:Finished evaluation at 2020-01-16-06:32:09

INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.78836, accuracy_baseline = 0.5, auc = 0.8707019, auc_precision_recall = 0.87288266, average_loss = 0.4507214, global_step = 5000, label/mean = 0.5, loss = 0.45059103, precision = 0.79099053, prediction/mean = 0.50262463, recall = 0.78384

INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.78836, accuracy_baseline = 0.5, auc = 0.8707019, auc_precision_recall = 0.87288266, average_loss = 0.4507214, global_step = 5000, label/mean = 0.5, loss = 0.45059103, precision = 0.79099053, prediction/mean = 0.50262463, recall = 0.78384

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmpfs/tmp/tmph3gv3f5_/model.ckpt-5000

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmpfs/tmp/tmph3gv3f5_/model.ckpt-5000

0.5

Assigning the most represented class will give us accuracy of 50%. There are a couple of things to notice here:

  1. Maybe surprisingly, a model can still be learned on top of fixed, random embeddings. The reason is that even if every word in the dictionary is mapped to a random vector, the estimator can separate the space purely using its fully connected layers.
  2. Allowing training of the module with random embeddings increases both training and test accuracy as oposed to training just the classifier.
  3. Training of the module with pre-trained embeddings also increases both accuracies. Note however the overfitting on the training set. Training a pre-trained module can be dangerous even with regularization in the sense that the embedding weights no longer represent the language model trained on diverse data, instead they converge to the ideal representation of the new dataset.