Yerel TensorFlow Everywhere etkinliğiniz için bugün LCV!
Bu sayfa, Cloud Translation API ile çevrilmiştir.
Switch to English

TensorFlow Transform ile verileri ön işleme

TensorFlow Extended (TFX) Özellik Mühendisliği Bileşeni

Bu örnek colab not defteri, TensorFlow Transform'un ( tf.Transform ) hem bir modeli eğitmek hem de üretimde çıkarımlar sunmak için tam olarak aynı kodu kullanarak verileri ön işlemede nasıl kullanılabileceğine dair biraz daha gelişmiş bir örnek sağlar.

TensorFlow Transform, eğitim veri kümesi üzerinden tam geçiş gerektiren özellikler oluşturmak da dahil olmak üzere TensorFlow için giriş verilerini önceden işlemek için bir kitaplıktır. Örneğin, TensorFlow Transform'u kullanarak şunları yapabilirsiniz:

  • Ortalama ve standart sapmayı kullanarak bir giriş değerini normalleştirin
  • Tüm giriş değerleri üzerinden bir kelime hazinesi oluşturarak dizeleri tam sayılara dönüştürün
  • Gözlemlenen veri dağılımına göre kayan sayıları tamsayılara dönüştürün.

TensorFlow, tek bir örnek veya bir grup örnek üzerinde manipülasyonlar için yerleşik desteğe sahiptir. tf.Transform , tüm eğitim veri kümesinde tam geçişleri desteklemek için bu yetenekleri genişletir.

tf.Transform çıktısı, hem eğitim hem de sunum için kullanabileceğiniz bir TensorFlow grafiği olarak dışa aktarılır. Her iki aşamada da aynı dönüşümler uygulandığından, hem eğitim hem de sunum için aynı grafiğin kullanılması eğriliği önleyebilir.

Bu örnekte ne yapıyoruz

Bu örnekte, sayım verilerini içeren , yaygın olarak kullanılan bir veri kümesini işleyeceğiz ve sınıflandırma yapmak için bir modeli eğiteceğiz. Yol boyunca verileri tf.Transform kullanarak dönüştürüyor tf.Transform .

Pip'i Yükselt

Yerel olarak çalışırken bir sistemde Pip'i yükseltmekten kaçınmak için Colab'de çalıştığımızdan emin olun. Yerel sistemler elbette ayrı olarak yükseltilebilir.

try:
  import colab
  !pip install -q --upgrade pip
except:
  pass

TensorFlow Transform'u yükleyin

pip install -q tensorflow-transform

Python kontrolü, içe aktarmalar ve globaller

Önce Python 3 kullandığımızdan emin olacağız ve sonra devam edip ihtiyacımız olan şeyleri yükleyip içe aktaracağız.

import sys

# Confirm that we're using Python 3
assert sys.version_info.major is 3, 'Oops, not running Python 3. Use Runtime > Change runtime type'
import math
import os
import pprint

import tensorflow as tf
print('TF: {}'.format(tf.__version__))

import apache_beam as beam
print('Beam: {}'.format(beam.__version__))

import tensorflow_transform as tft
import tensorflow_transform.beam as tft_beam
print('Transform: {}'.format(tft.__version__))

from tfx_bsl.public import tfxio

!wget https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/census/adult.data
!wget https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/census/adult.test

train = './adult.data'
test = './adult.test'
TF: 2.4.1
Beam: 2.27.0
Transform: 0.27.0
--2021-01-31 10:17:55--  https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/census/adult.data
Resolving storage.googleapis.com (storage.googleapis.com)... 74.125.204.128, 74.125.203.128, 74.125.23.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|74.125.204.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3974305 (3.8M) [application/octet-stream]
Saving to: ‘adult.data’

adult.data          100%[===================>]   3.79M  --.-KB/s    in 0.03s   

2021-01-31 10:17:55 (140 MB/s) - ‘adult.data’ saved [3974305/3974305]

--2021-01-31 10:17:55--  https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/census/adult.test
Resolving storage.googleapis.com (storage.googleapis.com)... 74.125.204.128, 74.125.203.128, 74.125.23.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|74.125.204.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2003153 (1.9M) [application/octet-stream]
Saving to: ‘adult.test’

adult.test          100%[===================>]   1.91M  --.-KB/s    in 0.02s   

2021-01-31 10:17:56 (87.1 MB/s) - ‘adult.test’ saved [2003153/2003153]


Sütunlarımızı adlandırın

Veri kümemizdeki sütunlara referans vermek için bazı kullanışlı listeler oluşturacağız.

CATEGORICAL_FEATURE_KEYS = [
    'workclass',
    'education',
    'marital-status',
    'occupation',
    'relationship',
    'race',
    'sex',
    'native-country',
]
NUMERIC_FEATURE_KEYS = [
    'age',
    'capital-gain',
    'capital-loss',
    'hours-per-week',
]
OPTIONAL_NUMERIC_FEATURE_KEYS = [
    'education-num',
]
ORDERED_CSV_COLUMNS = [
    'age', 'workclass', 'fnlwgt', 'education', 'education-num',
    'marital-status', 'occupation', 'relationship', 'race', 'sex',
    'capital-gain', 'capital-loss', 'hours-per-week', 'native-country', 'label'
]
LABEL_KEY = 'label'

Özelliklerimizi ve şemamızı tanımlayın

Girdiğimizdeki sütunların ne tür olduğuna bağlı olarak bir şema tanımlayalım. Diğer şeylerin yanı sıra bu, onları doğru şekilde içe aktarmaya yardımcı olacaktır.

RAW_DATA_FEATURE_SPEC = dict(
    [(name, tf.io.FixedLenFeature([], tf.string))
     for name in CATEGORICAL_FEATURE_KEYS] +
    [(name, tf.io.FixedLenFeature([], tf.float32))
     for name in NUMERIC_FEATURE_KEYS] +
    [(name, tf.io.VarLenFeature(tf.float32))
     for name in OPTIONAL_NUMERIC_FEATURE_KEYS] +
    [(LABEL_KEY, tf.io.FixedLenFeature([], tf.string))]
)

SCHEMA = tft.tf_metadata.dataset_metadata.DatasetMetadata(
    tft.tf_metadata.schema_utils.schema_from_feature_spec(RAW_DATA_FEATURE_SPEC)).schema

Hiperparametrelerin ayarlanması ve temel temizlik

Eğitim için kullanılan sabitler ve hiperparametreler. Paket boyutu, veri kümesi açıklamasında listelenen tüm kategorileri ve "?" İçin fazladan bir kategoriyi içerir. bilinmeyeni temsil eder.

testing = os.getenv("WEB_TEST_BROWSER", False)
NUM_OOV_BUCKETS = 1
if testing:
  TRAIN_NUM_EPOCHS = 1
  NUM_TRAIN_INSTANCES = 1
  TRAIN_BATCH_SIZE = 1
  NUM_TEST_INSTANCES = 1
else:
  TRAIN_NUM_EPOCHS = 16
  NUM_TRAIN_INSTANCES = 32561
  TRAIN_BATCH_SIZE = 128
  NUM_TEST_INSTANCES = 16281

# Names of temp files
TRANSFORMED_TRAIN_DATA_FILEBASE = 'train_transformed'
TRANSFORMED_TEST_DATA_FILEBASE = 'test_transformed'
EXPORTED_MODEL_DIR = 'exported_model_dir'

tf.Transform ile tf.Transform işleme

Bir tf.Transform preprocessing_fn oluşturun

Ön işleme fonksiyonu , tf.Transform'un en önemli kavramıdır. Bir ön işleme işlevi, veri kümesinin dönüşümünün gerçekten gerçekleştiği yerdir. Bir tensörün Tensor veya SparseTensor anlamına geldiği bir tensör sözlüğünü kabul eder ve döndürür. Genellikle bir ön işleme işlevinin kalbini oluşturan iki ana API çağrısı grubu vardır:

  1. TensorFlow İşlemleri: Tensörleri kabul eden ve döndüren, genellikle TensorFlow işlemleri anlamına gelen herhangi bir işlev. Bunlar, ham verileri her seferinde bir özellik vektörü olmak üzere dönüştürülmüş verilere dönüştüren grafiğe TensorFlow işlemlerini ekler. Bunlar, hem eğitim hem de hizmet sırasında her örnek için çalışacaktır.
  2. TensorFlow Transform Analizörleri: tf.Transform tarafından sağlanan analizörlerden herhangi biri. Analizciler ayrıca tensörleri de kabul eder ve geri verir, ancak TensorFlow işlemlerinden farklı olarak, eğitim sırasında yalnızca bir kez çalışırlar ve genellikle tüm eğitim veri kümesinde tam bir geçiş yaparlar. Grafiğinize eklenen tensör sabitleri oluştururlar. Örneğin, tft.min , eğitim veri kümesi üzerindeki minimum tensörü hesaplar. tf.Transform, sabit bir çözümleyici seti sağlar, ancak bu gelecek sürümlerde genişletilecektir.
def preprocessing_fn(inputs):
  """Preprocess input columns into transformed columns."""
  # Since we are modifying some features and leaving others unchanged, we
  # start by setting `outputs` to a copy of `inputs.
  outputs = inputs.copy()

  # Scale numeric columns to have range [0, 1].
  for key in NUMERIC_FEATURE_KEYS:
    outputs[key] = tft.scale_to_0_1(inputs[key])

  for key in OPTIONAL_NUMERIC_FEATURE_KEYS:
    # This is a SparseTensor because it is optional. Here we fill in a default
    # value when it is missing.
    sparse = tf.sparse.SparseTensor(inputs[key].indices, inputs[key].values,
                                    [inputs[key].dense_shape[0], 1])
    dense = tf.sparse.to_dense(sp_input=sparse, default_value=0.)
    # Reshaping from a batch of vectors of size 1 to a batch to scalars.
    dense = tf.squeeze(dense, axis=1)
    outputs[key] = tft.scale_to_0_1(dense)

  # For all categorical columns except the label column, we generate a
  # vocabulary but do not modify the feature.  This vocabulary is instead
  # used in the trainer, by means of a feature column, to convert the feature
  # from a string to an integer id.
  for key in CATEGORICAL_FEATURE_KEYS:
    outputs[key] = tft.compute_and_apply_vocabulary(
        tf.strings.strip(inputs[key]),
        num_oov_buckets=NUM_OOV_BUCKETS,
        vocab_filename=key)

  # For the label column we provide the mapping from string to index.
  table_keys = ['>50K', '<=50K']
  initializer = tf.lookup.KeyValueTensorInitializer(
      keys=table_keys,
      values=tf.cast(tf.range(len(table_keys)), tf.int64),
      key_dtype=tf.string,
      value_dtype=tf.int64)
  table = tf.lookup.StaticHashTable(initializer, default_value=-1)
  # Remove trailing periods for test data when the data is read with tf.data.
  label_str = tf.strings.regex_replace(inputs[LABEL_KEY], r'\.', '')
  label_str = tf.strings.strip(label_str)
  data_labels = table.lookup(label_str)
  transformed_label = tf.one_hot(
      indices=data_labels, depth=len(table_keys), on_value=1.0, off_value=0.0)
  outputs[LABEL_KEY] = tf.reshape(transformed_label, [-1, len(table_keys)])

  return outputs

Verileri dönüştürün

Artık verilerimizi Apache Beam ardışık düzeninde dönüştürmeye başlamaya hazırız.

  1. Verileri CSV okuyucuyu kullanarak okuyun
  2. Sayısal verileri ölçeklendiren ve kategorik verileri dizelerden int64 değer indekslerine dönüştüren bir ön işleme hattı kullanarak, her kategori için bir sözlük oluşturarak bunu dönüştürün
  3. Sonucu, daha sonra bir modeli eğitmek için kullanacağımız Example protosların TFRecord olarak yazın.
def transform_data(train_data_file, test_data_file, working_dir):
  """Transform the data and write out as a TFRecord of Example protos.

  Read in the data using the CSV reader, and transform it using a
  preprocessing pipeline that scales numeric data and converts categorical data
  from strings to int64 values indices, by creating a vocabulary for each
  category.

  Args:
    train_data_file: File containing training data
    test_data_file: File containing test data
    working_dir: Directory to write transformed data and metadata to
  """

  # The "with" block will create a pipeline, and run that pipeline at the exit
  # of the block.
  with beam.Pipeline() as pipeline:
    with tft_beam.Context(temp_dir=tempfile.mkdtemp()):
      # Create a TFXIO to read the census data with the schema. To do this we
      # need to list all columns in order since the schema doesn't specify the
      # order of columns in the csv.
      # We first read CSV files and use BeamRecordCsvTFXIO whose .BeamSource()
      # accepts a PCollection[bytes] because we need to patch the records first
      # (see "FixCommasTrainData" below). Otherwise, tfxio.CsvTFXIO can be used
      # to both read the CSV files and parse them to TFT inputs:
      # csv_tfxio = tfxio.CsvTFXIO(...)
      # raw_data = (pipeline | 'ToRecordBatches' >> csv_tfxio.BeamSource())
      csv_tfxio = tfxio.BeamRecordCsvTFXIO(
          physical_format='text',
          column_names=ORDERED_CSV_COLUMNS,
          schema=SCHEMA)

      # Read in raw data and convert using CSV TFXIO.  Note that we apply
      # some Beam transformations here, which will not be encoded in the TF
      # graph since we don't do the from within tf.Transform's methods
      # (AnalyzeDataset, TransformDataset etc.).  These transformations are just
      # to get data into a format that the CSV TFXIO can read, in particular
      # removing spaces after commas.
      raw_data = (
          pipeline
          | 'ReadTrainData' >> beam.io.ReadFromText(
              train_data_file, coder=beam.coders.BytesCoder())
          | 'FixCommasTrainData' >> beam.Map(
              lambda line: line.replace(b', ', b','))
          | 'DecodeTrainData' >> csv_tfxio.BeamSource())

      # Combine data and schema into a dataset tuple.  Note that we already used
      # the schema to read the CSV data, but we also need it to interpret
      # raw_data.
      raw_dataset = (raw_data, csv_tfxio.TensorAdapterConfig())
      transformed_dataset, transform_fn = (
          raw_dataset | tft_beam.AnalyzeAndTransformDataset(preprocessing_fn))
      transformed_data, transformed_metadata = transformed_dataset
      transformed_data_coder = tft.coders.ExampleProtoCoder(
          transformed_metadata.schema)

      _ = (
          transformed_data
          | 'EncodeTrainData' >> beam.Map(transformed_data_coder.encode)
          | 'WriteTrainData' >> beam.io.WriteToTFRecord(
              os.path.join(working_dir, TRANSFORMED_TRAIN_DATA_FILEBASE)))

      # Now apply transform function to test data.  In this case we remove the
      # trailing period at the end of each line, and also ignore the header line
      # that is present in the test data file.
      raw_test_data = (
          pipeline
          | 'ReadTestData' >> beam.io.ReadFromText(
              test_data_file, skip_header_lines=1,
              coder=beam.coders.BytesCoder())
          | 'FixCommasTestData' >> beam.Map(
              lambda line: line.replace(b', ', b','))
          | 'RemoveTrailingPeriodsTestData' >> beam.Map(lambda line: line[:-1])
          | 'DecodeTestData' >> csv_tfxio.BeamSource())

      raw_test_dataset = (raw_test_data, csv_tfxio.TensorAdapterConfig())

      transformed_test_dataset = (
          (raw_test_dataset, transform_fn) | tft_beam.TransformDataset())
      # Don't need transformed data schema, it's the same as before.
      transformed_test_data, _ = transformed_test_dataset

      _ = (
          transformed_test_data
          | 'EncodeTestData' >> beam.Map(transformed_data_coder.encode)
          | 'WriteTestData' >> beam.io.WriteToTFRecord(
              os.path.join(working_dir, TRANSFORMED_TEST_DATA_FILEBASE)))

      # Will write a SavedModel and metadata to working_dir, which can then
      # be read by the tft.TFTransformOutput class.
      _ = (
          transform_fn
          | 'WriteTransformFn' >> tft_beam.WriteTransformFn(working_dir))

Tf.keras kullanarak bir modeli eğitmek için önceden işlenmiş verilerimizi kullanma

tf.Transform hem eğitim hem de hizmet için aynı kodu kullanmamızı ve böylece çarpıklığı tf.Transform nasıl tf.Transform göstermek için bir model eğiteceğiz. Modelimizi eğitmek ve eğitilmiş modelimizi üretime hazırlamak için girdi fonksiyonları oluşturmamız gerekiyor. Eğitim girdi işlevimiz ile hizmet girdi işlevimiz arasındaki temel fark, eğitim verilerinin etiketleri içermesi ve üretim verilerinin içermemesidir. Argümanlar ve dönüşler de biraz farklıdır.

Eğitim için bir girdi işlevi oluşturun

def _make_training_input_fn(tf_transform_output, transformed_examples,
                            batch_size):
  """An input function reading from transformed data, converting to model input.

  Args:
    tf_transform_output: Wrapper around output of tf.Transform.
    transformed_examples: Base filename of examples.
    batch_size: Batch size.

  Returns:
    The input data for training or eval, in the form of k.
  """
  def input_fn():
    return tf.data.experimental.make_batched_features_dataset(
        file_pattern=transformed_examples,
        batch_size=batch_size,
        features=tf_transform_output.transformed_feature_spec(),
        reader=tf.data.TFRecordDataset,
        label_key=LABEL_KEY,
        shuffle=True).prefetch(tf.data.experimental.AUTOTUNE)

  return input_fn

Sunum için bir giriş işlevi oluşturun

Üretimde kullanabileceğimiz bir girdi fonksiyonu oluşturalım ve eğitimli modelimizi hizmete hazırlayalım.

def _make_serving_input_fn(tf_transform_output, raw_examples, batch_size):
  """An input function reading from raw data, converting to model input.

  Args:
    tf_transform_output: Wrapper around output of tf.Transform.
    raw_examples: Base filename of examples.
    batch_size: Batch size.

  Returns:
    The input data for training or eval, in the form of k.
  """

  def get_ordered_raw_data_dtypes():
    result = []
    for col in ORDERED_CSV_COLUMNS:
      if col not in RAW_DATA_FEATURE_SPEC:
        result.append(0.0)
        continue
      spec = RAW_DATA_FEATURE_SPEC[col]
      if isinstance(spec, tf.io.FixedLenFeature):
        result.append(spec.dtype)
      else:
        result.append(0.0)
    return result

  def input_fn():
    dataset = tf.data.experimental.make_csv_dataset(
        file_pattern=raw_examples,
        batch_size=batch_size,
        column_names=ORDERED_CSV_COLUMNS,
        column_defaults=get_ordered_raw_data_dtypes(),
        prefetch_buffer_size=0,
        ignore_errors=True)

    tft_layer = tf_transform_output.transform_features_layer()

    def transform_dataset(data):
      raw_features = {}
      for key, val in data.items():
        if key not in RAW_DATA_FEATURE_SPEC:
          continue
        if isinstance(RAW_DATA_FEATURE_SPEC[key], tf.io.VarLenFeature):
          raw_features[key] = tf.RaggedTensor.from_tensor(
              tf.expand_dims(val, -1)).to_sparse()
          continue
        raw_features[key] = val
      transformed_features = tft_layer(raw_features)
      data_labels = transformed_features.pop(LABEL_KEY)
      return (transformed_features, data_labels)

    return dataset.map(
        transform_dataset,
        num_parallel_calls=tf.data.experimental.AUTOTUNE).prefetch(
            tf.data.experimental.AUTOTUNE)

  return input_fn

Modelimizi Eğitin, Değerlendirin ve Dışa Aktarın

def export_serving_model(tf_transform_output, model, output_dir):
  """Exports a keras model for serving.

  Args:
    tf_transform_output: Wrapper around output of tf.Transform.
    model: A keras model to export for serving.
    output_dir: A directory where the model will be exported to.
  """
  # The layer has to be saved to the model for keras tracking purpases.
  model.tft_layer = tf_transform_output.transform_features_layer()

  @tf.function
  def serve_tf_examples_fn(serialized_tf_examples):
    """Serving tf.function model wrapper."""
    feature_spec = RAW_DATA_FEATURE_SPEC.copy()
    feature_spec.pop(LABEL_KEY)
    parsed_features = tf.io.parse_example(serialized_tf_examples, feature_spec)
    transformed_features = model.tft_layer(parsed_features)
    outputs = model(transformed_features)
    classes_names = tf.constant([['0', '1']])
    classes = tf.tile(classes_names, [tf.shape(outputs)[0], 1])
    return {'classes': classes, 'scores': outputs}

  concrete_serving_fn = serve_tf_examples_fn.get_concrete_function(
      tf.TensorSpec(shape=[None], dtype=tf.string, name='inputs'))
  signatures = {'serving_default': concrete_serving_fn}

  # This is required in order to make this model servable with model_server.
  versioned_output_dir = os.path.join(output_dir, '1')
  model.save(versioned_output_dir, save_format='tf', signatures=signatures)
def train_and_evaluate(working_dir,
                       num_train_instances=NUM_TRAIN_INSTANCES,
                       num_test_instances=NUM_TEST_INSTANCES):
  """Train the model on training data and evaluate on test data.

  Args:
    working_dir: The location of the Transform output.
    num_train_instances: Number of instances in train set
    num_test_instances: Number of instances in test set

  Returns:
    The results from the estimator's 'evaluate' method
  """
  train_data_path_pattern = os.path.join(working_dir,
                                 TRANSFORMED_TRAIN_DATA_FILEBASE + '*')
  eval_data_path_pattern = os.path.join(working_dir,
                            TRANSFORMED_TEST_DATA_FILEBASE + '*')
  tf_transform_output = tft.TFTransformOutput(working_dir)

  train_input_fn = _make_training_input_fn(
      tf_transform_output, train_data_path_pattern, batch_size=TRAIN_BATCH_SIZE)
  train_dataset = train_input_fn()

  # Evaluate model on test dataset.
  eval_input_fn = _make_training_input_fn(
      tf_transform_output, eval_data_path_pattern, batch_size=TRAIN_BATCH_SIZE)
  validation_dataset = eval_input_fn()

  feature_spec = tf_transform_output.transformed_feature_spec().copy()
  feature_spec.pop(LABEL_KEY)

  inputs = {}
  for key, spec in feature_spec.items():
    if isinstance(spec, tf.io.VarLenFeature):
      inputs[key] = tf.keras.layers.Input(
          shape=[None], name=key, dtype=spec.dtype, sparse=True)
    elif isinstance(spec, tf.io.FixedLenFeature):
      inputs[key] = tf.keras.layers.Input(
          shape=spec.shape, name=key, dtype=spec.dtype)
    else:
      raise ValueError('Spec type is not supported: ', key, spec)

  encoded_inputs = {}
  for key in inputs:
    feature = tf.expand_dims(inputs[key], -1)
    if key in CATEGORICAL_FEATURE_KEYS:
      num_buckets = tf_transform_output.num_buckets_for_transformed_feature(key)
      encoding_layer = (
          tf.keras.layers.experimental.preprocessing.CategoryEncoding(
              max_tokens=num_buckets, output_mode='binary', sparse=False))
      encoded_inputs[key] = encoding_layer(feature)
    else:
      encoded_inputs[key] = feature

  stacked_inputs = tf.concat(tf.nest.flatten(encoded_inputs), axis=1)
  output = tf.keras.layers.Dense(100, activation='relu')(stacked_inputs)
  output = tf.keras.layers.Dense(70, activation='relu')(output)
  output = tf.keras.layers.Dense(50, activation='relu')(output)
  output = tf.keras.layers.Dense(20, activation='relu')(output)
  output = tf.keras.layers.Dense(2, activation='sigmoid')(output)
  model = tf.keras.Model(inputs=inputs, outputs=output)

  model.compile(optimizer='adam',
                loss='binary_crossentropy',
                metrics=['accuracy'])
  pprint.pprint(model.summary())

  model.fit(train_dataset, validation_data=validation_dataset,
            epochs=TRAIN_NUM_EPOCHS,
            steps_per_epoch=math.ceil(num_train_instances / TRAIN_BATCH_SIZE),
            validation_steps=math.ceil(num_test_instances / TRAIN_BATCH_SIZE))

  # Export the model.
  exported_model_dir = os.path.join(working_dir, EXPORTED_MODEL_DIR)
  export_serving_model(tf_transform_output, model, exported_model_dir)

  metrics_values = model.evaluate(validation_dataset, steps=num_test_instances)
  metrics_labels = model.metrics_names
  return {l: v for l, v in zip(metrics_labels, metrics_values)}

Hepsini bir araya getirmek

Sayım verilerimizi önceden işlemek, bir model eğitmek ve onu hizmete hazırlamak için ihtiyacımız olan her şeyi oluşturduk. Şimdiye kadar sadece işleri hazırlıyoruz. Koşmaya başlama zamanı!

import tempfile
temp = os.path.join(tempfile.gettempdir(), 'keras')

transform_data(train, test, temp)
results = train_and_evaluate(temp)
pprint.pprint(results)
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.

WARNING:tensorflow:Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

WARNING:tensorflow:Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

WARNING:tensorflow:Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

WARNING:tensorflow:Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_transform/tf_utils.py:266: Tensor.experimental_ref (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use ref() instead.

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_transform/tf_utils.py:266: Tensor.experimental_ref (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use ref() instead.

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:No assets to write.

INFO:tensorflow:No assets to write.

WARNING:tensorflow:Issue encountered when serializing tft_mapper_use.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'Counter' object has no attribute 'name'

WARNING:tensorflow:Issue encountered when serializing tft_mapper_use.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'Counter' object has no attribute 'name'

INFO:tensorflow:SavedModel written to: /tmp/tmp76doyn8o/tftransform_tmp/030f541180784ecbb607ce85d281032a/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/tmp76doyn8o/tftransform_tmp/030f541180784ecbb607ce85d281032a/saved_model.pb

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:No assets to write.

INFO:tensorflow:No assets to write.

WARNING:tensorflow:Issue encountered when serializing tft_mapper_use.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'Counter' object has no attribute 'name'

WARNING:tensorflow:Issue encountered when serializing tft_mapper_use.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'Counter' object has no attribute 'name'

INFO:tensorflow:SavedModel written to: /tmp/tmp76doyn8o/tftransform_tmp/4fe89334a3e34cb589b903954b7a1051/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/tmp76doyn8o/tftransform_tmp/4fe89334a3e34cb589b903954b7a1051/saved_model.pb

WARNING:tensorflow:Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

WARNING:tensorflow:Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

WARNING:tensorflow:Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

WARNING:tensorflow:Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets written to: /tmp/tmp76doyn8o/tftransform_tmp/8b5631664e2c4dd28f9d908a8de142f3/assets

INFO:tensorflow:Assets written to: /tmp/tmp76doyn8o/tftransform_tmp/8b5631664e2c4dd28f9d908a8de142f3/assets

INFO:tensorflow:SavedModel written to: /tmp/tmp76doyn8o/tftransform_tmp/8b5631664e2c4dd28f9d908a8de142f3/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/tmp76doyn8o/tftransform_tmp/8b5631664e2c4dd28f9d908a8de142f3/saved_model.pb

WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.

WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
education (InputLayer)          [(None,)]            0                                            
__________________________________________________________________________________________________
marital-status (InputLayer)     [(None,)]            0                                            
__________________________________________________________________________________________________
native-country (InputLayer)     [(None,)]            0                                            
__________________________________________________________________________________________________
occupation (InputLayer)         [(None,)]            0                                            
__________________________________________________________________________________________________
race (InputLayer)               [(None,)]            0                                            
__________________________________________________________________________________________________
relationship (InputLayer)       [(None,)]            0                                            
__________________________________________________________________________________________________
sex (InputLayer)                [(None,)]            0                                            
__________________________________________________________________________________________________
workclass (InputLayer)          [(None,)]            0                                            
__________________________________________________________________________________________________
age (InputLayer)                [(None,)]            0                                            
__________________________________________________________________________________________________
capital-gain (InputLayer)       [(None,)]            0                                            
__________________________________________________________________________________________________
capital-loss (InputLayer)       [(None,)]            0                                            
__________________________________________________________________________________________________
tf.expand_dims_3 (TFOpLambda)   (None, 1)            0           education[0][0]                  
__________________________________________________________________________________________________
education-num (InputLayer)      [(None,)]            0                                            
__________________________________________________________________________________________________
hours-per-week (InputLayer)     [(None,)]            0                                            
__________________________________________________________________________________________________
tf.expand_dims_6 (TFOpLambda)   (None, 1)            0           marital-status[0][0]             
__________________________________________________________________________________________________
tf.expand_dims_7 (TFOpLambda)   (None, 1)            0           native-country[0][0]             
__________________________________________________________________________________________________
tf.expand_dims_8 (TFOpLambda)   (None, 1)            0           occupation[0][0]                 
__________________________________________________________________________________________________
tf.expand_dims_9 (TFOpLambda)   (None, 1)            0           race[0][0]                       
__________________________________________________________________________________________________
tf.expand_dims_10 (TFOpLambda)  (None, 1)            0           relationship[0][0]               
__________________________________________________________________________________________________
tf.expand_dims_11 (TFOpLambda)  (None, 1)            0           sex[0][0]                        
__________________________________________________________________________________________________
tf.expand_dims_12 (TFOpLambda)  (None, 1)            0           workclass[0][0]                  
__________________________________________________________________________________________________
tf.expand_dims (TFOpLambda)     (None, 1)            0           age[0][0]                        
__________________________________________________________________________________________________
tf.expand_dims_1 (TFOpLambda)   (None, 1)            0           capital-gain[0][0]               
__________________________________________________________________________________________________
tf.expand_dims_2 (TFOpLambda)   (None, 1)            0           capital-loss[0][0]               
__________________________________________________________________________________________________
category_encoding (CategoryEnco (None, 17)           0           tf.expand_dims_3[0][0]           
__________________________________________________________________________________________________
tf.expand_dims_4 (TFOpLambda)   (None, 1)            0           education-num[0][0]              
__________________________________________________________________________________________________
tf.expand_dims_5 (TFOpLambda)   (None, 1)            0           hours-per-week[0][0]             
__________________________________________________________________________________________________
category_encoding_1 (CategoryEn (None, 8)            0           tf.expand_dims_6[0][0]           
__________________________________________________________________________________________________
category_encoding_2 (CategoryEn (None, 43)           0           tf.expand_dims_7[0][0]           
__________________________________________________________________________________________________
category_encoding_3 (CategoryEn (None, 16)           0           tf.expand_dims_8[0][0]           
__________________________________________________________________________________________________
category_encoding_4 (CategoryEn (None, 6)            0           tf.expand_dims_9[0][0]           
__________________________________________________________________________________________________
category_encoding_5 (CategoryEn (None, 7)            0           tf.expand_dims_10[0][0]          
__________________________________________________________________________________________________
category_encoding_6 (CategoryEn (None, 3)            0           tf.expand_dims_11[0][0]          
__________________________________________________________________________________________________
category_encoding_7 (CategoryEn (None, 10)           0           tf.expand_dims_12[0][0]          
__________________________________________________________________________________________________
tf.concat (TFOpLambda)          (None, 115)          0           tf.expand_dims[0][0]             
                                                                 tf.expand_dims_1[0][0]           
                                                                 tf.expand_dims_2[0][0]           
                                                                 category_encoding[0][0]          
                                                                 tf.expand_dims_4[0][0]           
                                                                 tf.expand_dims_5[0][0]           
                                                                 category_encoding_1[0][0]        
                                                                 category_encoding_2[0][0]        
                                                                 category_encoding_3[0][0]        
                                                                 category_encoding_4[0][0]        
                                                                 category_encoding_5[0][0]        
                                                                 category_encoding_6[0][0]        
                                                                 category_encoding_7[0][0]        
__________________________________________________________________________________________________
dense (Dense)                   (None, 100)          11600       tf.concat[0][0]                  
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 70)           7070        dense[0][0]                      
__________________________________________________________________________________________________
dense_2 (Dense)                 (None, 50)           3550        dense_1[0][0]                    
__________________________________________________________________________________________________
dense_3 (Dense)                 (None, 20)           1020        dense_2[0][0]                    
__________________________________________________________________________________________________
dense_4 (Dense)                 (None, 2)            42          dense_3[0][0]                    
==================================================================================================
Total params: 23,282
Trainable params: 23,282
Non-trainable params: 0
__________________________________________________________________________________________________
None
Epoch 1/16
255/255 [==============================] - 3s 8ms/step - loss: 0.4542 - accuracy: 0.7834 - val_loss: 0.3384 - val_accuracy: 0.8420
Epoch 2/16
255/255 [==============================] - 2s 6ms/step - loss: 0.3400 - accuracy: 0.8369 - val_loss: 0.3308 - val_accuracy: 0.8440
Epoch 3/16
255/255 [==============================] - 2s 6ms/step - loss: 0.3256 - accuracy: 0.8476 - val_loss: 0.3401 - val_accuracy: 0.8387
Epoch 4/16
255/255 [==============================] - 2s 6ms/step - loss: 0.3183 - accuracy: 0.8494 - val_loss: 0.3294 - val_accuracy: 0.8448
Epoch 5/16
255/255 [==============================] - 2s 6ms/step - loss: 0.3141 - accuracy: 0.8521 - val_loss: 0.3244 - val_accuracy: 0.8499
Epoch 6/16
255/255 [==============================] - 2s 6ms/step - loss: 0.3074 - accuracy: 0.8555 - val_loss: 0.3220 - val_accuracy: 0.8489
Epoch 7/16
255/255 [==============================] - 2s 6ms/step - loss: 0.3002 - accuracy: 0.8595 - val_loss: 0.3222 - val_accuracy: 0.8489
Epoch 8/16
255/255 [==============================] - 2s 7ms/step - loss: 0.3027 - accuracy: 0.8585 - val_loss: 0.3222 - val_accuracy: 0.8503
Epoch 9/16
255/255 [==============================] - 2s 6ms/step - loss: 0.2941 - accuracy: 0.8595 - val_loss: 0.3245 - val_accuracy: 0.8508
Epoch 10/16
255/255 [==============================] - 2s 6ms/step - loss: 0.2932 - accuracy: 0.8635 - val_loss: 0.3308 - val_accuracy: 0.8474
Epoch 11/16
255/255 [==============================] - 2s 6ms/step - loss: 0.2912 - accuracy: 0.8631 - val_loss: 0.3273 - val_accuracy: 0.8489
Epoch 12/16
255/255 [==============================] - 2s 6ms/step - loss: 0.2807 - accuracy: 0.8680 - val_loss: 0.3369 - val_accuracy: 0.8479
Epoch 13/16
255/255 [==============================] - 2s 6ms/step - loss: 0.2811 - accuracy: 0.8683 - val_loss: 0.3370 - val_accuracy: 0.8497
Epoch 14/16
255/255 [==============================] - 2s 6ms/step - loss: 0.2836 - accuracy: 0.8666 - val_loss: 0.3365 - val_accuracy: 0.8494
Epoch 15/16
255/255 [==============================] - 2s 6ms/step - loss: 0.2736 - accuracy: 0.8710 - val_loss: 0.3365 - val_accuracy: 0.8506
Epoch 16/16
255/255 [==============================] - 2s 6ms/step - loss: 0.2761 - accuracy: 0.8691 - val_loss: 0.3430 - val_accuracy: 0.8492
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Assets written to: /tmp/keras/exported_model_dir/1/assets

INFO:tensorflow:Assets written to: /tmp/keras/exported_model_dir/1/assets

16281/16281 [==============================] - 58s 4ms/step - loss: 0.3435 - accuracy: 0.8491
{'accuracy': 0.8490878939628601, 'loss': 0.34348100423812866}

(İsteğe bağlı) tf.estimator kullanarak bir model eğitmek için önceden işlenmiş verilerimizi kullanma

Keras modeli yerine bir Tahminci modeli kullanmayı tercih ederseniz, bu bölümdeki kod bunun nasıl yapılacağını gösterir.

Eğitim için bir girdi işlevi oluşturun

def _make_training_input_fn(tf_transform_output, transformed_examples,
                            batch_size):
  """Creates an input function reading from transformed data.

  Args:
    tf_transform_output: Wrapper around output of tf.Transform.
    transformed_examples: Base filename of examples.
    batch_size: Batch size.

  Returns:
    The input function for training or eval.
  """
  def input_fn():
    """Input function for training and eval."""
    dataset = tf.data.experimental.make_batched_features_dataset(
        file_pattern=transformed_examples,
        batch_size=batch_size,
        features=tf_transform_output.transformed_feature_spec(),
        reader=tf.data.TFRecordDataset,
        shuffle=True)

    transformed_features = tf.compat.v1.data.make_one_shot_iterator(
        dataset).get_next()

    # Extract features and label from the transformed tensors.
    transformed_labels = tf.where(
        tf.equal(transformed_features.pop(LABEL_KEY), 1))

    return transformed_features, transformed_labels[:,1]

  return input_fn

Sunum için bir giriş işlevi oluşturun

Üretimde kullanabileceğimiz bir girdi fonksiyonu oluşturalım ve eğitimli modelimizi hizmete hazırlayalım.

def _make_serving_input_fn(tf_transform_output):
  """Creates an input function reading from raw data.

  Args:
    tf_transform_output: Wrapper around output of tf.Transform.

  Returns:
    The serving input function.
  """
  raw_feature_spec = RAW_DATA_FEATURE_SPEC.copy()
  # Remove label since it is not available during serving.
  raw_feature_spec.pop(LABEL_KEY)

  def serving_input_fn():
    """Input function for serving."""
    # Get raw features by generating the basic serving input_fn and calling it.
    # Here we generate an input_fn that expects a parsed Example proto to be fed
    # to the model at serving time.  See also
    # tf.estimator.export.build_raw_serving_input_receiver_fn.
    raw_input_fn = tf.estimator.export.build_parsing_serving_input_receiver_fn(
        raw_feature_spec, default_batch_size=None)
    serving_input_receiver = raw_input_fn()

    # Apply the transform function that was used to generate the materialized
    # data.
    raw_features = serving_input_receiver.features
    transformed_features = tf_transform_output.transform_raw_features(
        raw_features)

    return tf.estimator.export.ServingInputReceiver(
        transformed_features, serving_input_receiver.receiver_tensors)

  return serving_input_fn

Giriş verilerimizi FeatureColumns'a sarın

Modelimiz verilerimizi TensorFlow FeatureColumns'da bekleyecektir.

def get_feature_columns(tf_transform_output):
  """Returns the FeatureColumns for the model.

  Args:
    tf_transform_output: A `TFTransformOutput` object.

  Returns:
    A list of FeatureColumns.
  """
  # Wrap scalars as real valued columns.
  real_valued_columns = [tf.feature_column.numeric_column(key, shape=())
                         for key in NUMERIC_FEATURE_KEYS]

  # Wrap categorical columns.
  one_hot_columns = [
      tf.feature_column.indicator_column(
          tf.feature_column.categorical_column_with_identity(
              key=key,
              num_buckets=(NUM_OOV_BUCKETS +
                  tf_transform_output.vocabulary_size_by_name(
                      vocab_filename=key))))
      for key in CATEGORICAL_FEATURE_KEYS]

  return real_valued_columns + one_hot_columns

Modelimizi Eğitin, Değerlendirin ve Dışa Aktarın

def train_and_evaluate(working_dir, num_train_instances=NUM_TRAIN_INSTANCES,
                       num_test_instances=NUM_TEST_INSTANCES):
  """Train the model on training data and evaluate on test data.

  Args:
    working_dir: Directory to read transformed data and metadata from and to
        write exported model to.
    num_train_instances: Number of instances in train set
    num_test_instances: Number of instances in test set

  Returns:
    The results from the estimator's 'evaluate' method
  """
  tf_transform_output = tft.TFTransformOutput(working_dir)

  run_config = tf.estimator.RunConfig()

  estimator = tf.estimator.LinearClassifier(
      feature_columns=get_feature_columns(tf_transform_output),
      config=run_config,
      loss_reduction=tf.losses.Reduction.SUM)

  # Fit the model using the default optimizer.
  train_input_fn = _make_training_input_fn(
      tf_transform_output,
      os.path.join(working_dir, TRANSFORMED_TRAIN_DATA_FILEBASE + '*'),
      batch_size=TRAIN_BATCH_SIZE)
  estimator.train(
      input_fn=train_input_fn,
      max_steps=TRAIN_NUM_EPOCHS * num_train_instances / TRAIN_BATCH_SIZE)

  # Evaluate model on test dataset.
  eval_input_fn = _make_training_input_fn(
      tf_transform_output,
      os.path.join(working_dir, TRANSFORMED_TEST_DATA_FILEBASE + '*'),
      batch_size=1)

  # Export the model.
  serving_input_fn = _make_serving_input_fn(tf_transform_output)
  exported_model_dir = os.path.join(working_dir, EXPORTED_MODEL_DIR)
  estimator.export_saved_model(exported_model_dir, serving_input_fn)

  return estimator.evaluate(input_fn=eval_input_fn, steps=num_test_instances)

Hepsini bir araya getirmek

Sayım verilerimizi önceden işlemek, bir model eğitmek ve onu hizmete hazırlamak için ihtiyaç duyduğumuz her şeyi oluşturduk. Şimdiye kadar sadece işleri hazırlıyoruz. Koşmaya başlama zamanı!

import tempfile
temp = os.path.join(tempfile.gettempdir(), 'estimator')

transform_data(train, test, temp)
results = train_and_evaluate(temp)
pprint.pprint(results)
WARNING:tensorflow:Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

WARNING:tensorflow:Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

WARNING:tensorflow:Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

WARNING:tensorflow:Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:No assets to write.

INFO:tensorflow:No assets to write.

WARNING:tensorflow:Issue encountered when serializing tft_mapper_use.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'Counter' object has no attribute 'name'

WARNING:tensorflow:Issue encountered when serializing tft_mapper_use.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'Counter' object has no attribute 'name'

INFO:tensorflow:SavedModel written to: /tmp/tmps7ehffp6/tftransform_tmp/77ba0e5cdc9d4f558957dc5cb607d7e4/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/tmps7ehffp6/tftransform_tmp/77ba0e5cdc9d4f558957dc5cb607d7e4/saved_model.pb

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:No assets to write.

INFO:tensorflow:No assets to write.

WARNING:tensorflow:Issue encountered when serializing tft_mapper_use.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'Counter' object has no attribute 'name'

WARNING:tensorflow:Issue encountered when serializing tft_mapper_use.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'Counter' object has no attribute 'name'

INFO:tensorflow:SavedModel written to: /tmp/tmps7ehffp6/tftransform_tmp/75f502ee79364f08950b3888f3f67326/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/tmps7ehffp6/tftransform_tmp/75f502ee79364f08950b3888f3f67326/saved_model.pb

WARNING:tensorflow:Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

WARNING:tensorflow:Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

WARNING:tensorflow:Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

WARNING:tensorflow:Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets written to: /tmp/tmps7ehffp6/tftransform_tmp/f79d7c0e1c894a79b164a2598aa7dea0/assets

INFO:tensorflow:Assets written to: /tmp/tmps7ehffp6/tftransform_tmp/f79d7c0e1c894a79b164a2598aa7dea0/assets

INFO:tensorflow:SavedModel written to: /tmp/tmps7ehffp6/tftransform_tmp/f79d7c0e1c894a79b164a2598aa7dea0/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/tmps7ehffp6/tftransform_tmp/f79d7c0e1c894a79b164a2598aa7dea0/saved_model.pb

WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp2webujvo

WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp2webujvo

INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmp2webujvo', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmp2webujvo', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.
/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:1727: UserWarning: `layer.add_variable` is deprecated and will be removed in a future version. Please use `layer.add_weight` method instead.
  warnings.warn('`layer.add_variable` is deprecated and '

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/keras/optimizer_v2/ftrl.py:134: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/keras/optimizer_v2/ftrl.py:134: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...

INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmp2webujvo/model.ckpt.

INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmp2webujvo/model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...

INFO:tensorflow:loss = 88.72284, step = 0

INFO:tensorflow:loss = 88.72284, step = 0

INFO:tensorflow:global_step/sec: 181.834

INFO:tensorflow:global_step/sec: 181.834

INFO:tensorflow:loss = 40.60157, step = 100 (0.551 sec)

INFO:tensorflow:loss = 40.60157, step = 100 (0.551 sec)

INFO:tensorflow:global_step/sec: 228.068

INFO:tensorflow:global_step/sec: 228.068

INFO:tensorflow:loss = 52.099167, step = 200 (0.439 sec)

INFO:tensorflow:loss = 52.099167, step = 200 (0.439 sec)

INFO:tensorflow:global_step/sec: 239.869

INFO:tensorflow:global_step/sec: 239.869

INFO:tensorflow:loss = 38.544167, step = 300 (0.416 sec)

INFO:tensorflow:loss = 38.544167, step = 300 (0.416 sec)

INFO:tensorflow:global_step/sec: 238.285

INFO:tensorflow:global_step/sec: 238.285

INFO:tensorflow:loss = 50.549423, step = 400 (0.420 sec)

INFO:tensorflow:loss = 50.549423, step = 400 (0.420 sec)

INFO:tensorflow:global_step/sec: 237.656

INFO:tensorflow:global_step/sec: 237.656

INFO:tensorflow:loss = 32.181908, step = 500 (0.421 sec)

INFO:tensorflow:loss = 32.181908, step = 500 (0.421 sec)

INFO:tensorflow:global_step/sec: 229.138

INFO:tensorflow:global_step/sec: 229.138

INFO:tensorflow:loss = 27.645021, step = 600 (0.438 sec)

INFO:tensorflow:loss = 27.645021, step = 600 (0.438 sec)

INFO:tensorflow:global_step/sec: 223.488

INFO:tensorflow:global_step/sec: 223.488

INFO:tensorflow:loss = 35.50036, step = 700 (0.446 sec)

INFO:tensorflow:loss = 35.50036, step = 700 (0.446 sec)

INFO:tensorflow:global_step/sec: 237.038

INFO:tensorflow:global_step/sec: 237.038

INFO:tensorflow:loss = 30.367598, step = 800 (0.422 sec)

INFO:tensorflow:loss = 30.367598, step = 800 (0.422 sec)

INFO:tensorflow:global_step/sec: 236.264

INFO:tensorflow:global_step/sec: 236.264

INFO:tensorflow:loss = 39.15401, step = 900 (0.423 sec)

INFO:tensorflow:loss = 39.15401, step = 900 (0.423 sec)

INFO:tensorflow:global_step/sec: 237.658

INFO:tensorflow:global_step/sec: 237.658

INFO:tensorflow:loss = 36.86463, step = 1000 (0.421 sec)

INFO:tensorflow:loss = 36.86463, step = 1000 (0.421 sec)

INFO:tensorflow:global_step/sec: 233.561

INFO:tensorflow:global_step/sec: 233.561

INFO:tensorflow:loss = 51.083702, step = 1100 (0.428 sec)

INFO:tensorflow:loss = 51.083702, step = 1100 (0.428 sec)

INFO:tensorflow:global_step/sec: 234.061

INFO:tensorflow:global_step/sec: 234.061

INFO:tensorflow:loss = 51.468567, step = 1200 (0.427 sec)

INFO:tensorflow:loss = 51.468567, step = 1200 (0.427 sec)

INFO:tensorflow:global_step/sec: 237.764

INFO:tensorflow:global_step/sec: 237.764

INFO:tensorflow:loss = 43.13748, step = 1300 (0.421 sec)

INFO:tensorflow:loss = 43.13748, step = 1300 (0.421 sec)

INFO:tensorflow:global_step/sec: 236.218

INFO:tensorflow:global_step/sec: 236.218

INFO:tensorflow:loss = 64.392105, step = 1400 (0.424 sec)

INFO:tensorflow:loss = 64.392105, step = 1400 (0.424 sec)

INFO:tensorflow:global_step/sec: 231.043

INFO:tensorflow:global_step/sec: 231.043

INFO:tensorflow:loss = 45.05866, step = 1500 (0.433 sec)

INFO:tensorflow:loss = 45.05866, step = 1500 (0.433 sec)

INFO:tensorflow:global_step/sec: 235.736

INFO:tensorflow:global_step/sec: 235.736

INFO:tensorflow:loss = 41.758694, step = 1600 (0.424 sec)

INFO:tensorflow:loss = 41.758694, step = 1600 (0.424 sec)

INFO:tensorflow:global_step/sec: 236.438

INFO:tensorflow:global_step/sec: 236.438

INFO:tensorflow:loss = 35.686638, step = 1700 (0.423 sec)

INFO:tensorflow:loss = 35.686638, step = 1700 (0.423 sec)

INFO:tensorflow:global_step/sec: 234.906

INFO:tensorflow:global_step/sec: 234.906

INFO:tensorflow:loss = 45.96911, step = 1800 (0.426 sec)

INFO:tensorflow:loss = 45.96911, step = 1800 (0.426 sec)

INFO:tensorflow:global_step/sec: 234.637

INFO:tensorflow:global_step/sec: 234.637

INFO:tensorflow:loss = 43.993755, step = 1900 (0.426 sec)

INFO:tensorflow:loss = 43.993755, step = 1900 (0.426 sec)

INFO:tensorflow:global_step/sec: 232.9

INFO:tensorflow:global_step/sec: 232.9

INFO:tensorflow:loss = 33.01324, step = 2000 (0.429 sec)

INFO:tensorflow:loss = 33.01324, step = 2000 (0.429 sec)

INFO:tensorflow:global_step/sec: 234.965

INFO:tensorflow:global_step/sec: 234.965

INFO:tensorflow:loss = 46.896156, step = 2100 (0.426 sec)

INFO:tensorflow:loss = 46.896156, step = 2100 (0.426 sec)

INFO:tensorflow:global_step/sec: 236.46

INFO:tensorflow:global_step/sec: 236.46

INFO:tensorflow:loss = 38.05753, step = 2200 (0.423 sec)

INFO:tensorflow:loss = 38.05753, step = 2200 (0.423 sec)

INFO:tensorflow:global_step/sec: 232.293

INFO:tensorflow:global_step/sec: 232.293

INFO:tensorflow:loss = 37.02017, step = 2300 (0.430 sec)

INFO:tensorflow:loss = 37.02017, step = 2300 (0.430 sec)

INFO:tensorflow:global_step/sec: 239.733

INFO:tensorflow:global_step/sec: 239.733

INFO:tensorflow:loss = 42.071465, step = 2400 (0.417 sec)

INFO:tensorflow:loss = 42.071465, step = 2400 (0.417 sec)

INFO:tensorflow:global_step/sec: 238.622

INFO:tensorflow:global_step/sec: 238.622

INFO:tensorflow:loss = 41.827965, step = 2500 (0.419 sec)

INFO:tensorflow:loss = 41.827965, step = 2500 (0.419 sec)

INFO:tensorflow:global_step/sec: 236.034

INFO:tensorflow:global_step/sec: 236.034

INFO:tensorflow:loss = 38.161346, step = 2600 (0.423 sec)

INFO:tensorflow:loss = 38.161346, step = 2600 (0.423 sec)

INFO:tensorflow:global_step/sec: 233.657

INFO:tensorflow:global_step/sec: 233.657

INFO:tensorflow:loss = 34.463432, step = 2700 (0.428 sec)

INFO:tensorflow:loss = 34.463432, step = 2700 (0.428 sec)

INFO:tensorflow:global_step/sec: 234.617

INFO:tensorflow:global_step/sec: 234.617

INFO:tensorflow:loss = 43.919376, step = 2800 (0.426 sec)

INFO:tensorflow:loss = 43.919376, step = 2800 (0.426 sec)

INFO:tensorflow:global_step/sec: 229.64

INFO:tensorflow:global_step/sec: 229.64

INFO:tensorflow:loss = 38.843185, step = 2900 (0.435 sec)

INFO:tensorflow:loss = 38.843185, step = 2900 (0.435 sec)

INFO:tensorflow:global_step/sec: 231.495

INFO:tensorflow:global_step/sec: 231.495

INFO:tensorflow:loss = 36.27419, step = 3000 (0.432 sec)

INFO:tensorflow:loss = 36.27419, step = 3000 (0.432 sec)

INFO:tensorflow:global_step/sec: 218.252

INFO:tensorflow:global_step/sec: 218.252

INFO:tensorflow:loss = 47.15911, step = 3100 (0.458 sec)

INFO:tensorflow:loss = 47.15911, step = 3100 (0.458 sec)

INFO:tensorflow:global_step/sec: 226.266

INFO:tensorflow:global_step/sec: 226.266

INFO:tensorflow:loss = 44.70205, step = 3200 (0.442 sec)

INFO:tensorflow:loss = 44.70205, step = 3200 (0.442 sec)

INFO:tensorflow:global_step/sec: 231.214

INFO:tensorflow:global_step/sec: 231.214

INFO:tensorflow:loss = 43.25578, step = 3300 (0.432 sec)

INFO:tensorflow:loss = 43.25578, step = 3300 (0.432 sec)

INFO:tensorflow:global_step/sec: 230.828

INFO:tensorflow:global_step/sec: 230.828

INFO:tensorflow:loss = 45.90856, step = 3400 (0.433 sec)

INFO:tensorflow:loss = 45.90856, step = 3400 (0.433 sec)

INFO:tensorflow:global_step/sec: 231.002

INFO:tensorflow:global_step/sec: 231.002

INFO:tensorflow:loss = 47.135113, step = 3500 (0.433 sec)

INFO:tensorflow:loss = 47.135113, step = 3500 (0.433 sec)

INFO:tensorflow:global_step/sec: 226.885

INFO:tensorflow:global_step/sec: 226.885

INFO:tensorflow:loss = 40.516785, step = 3600 (0.441 sec)

INFO:tensorflow:loss = 40.516785, step = 3600 (0.441 sec)

INFO:tensorflow:global_step/sec: 227.943

INFO:tensorflow:global_step/sec: 227.943

INFO:tensorflow:loss = 35.96137, step = 3700 (0.439 sec)

INFO:tensorflow:loss = 35.96137, step = 3700 (0.439 sec)

INFO:tensorflow:global_step/sec: 226.066

INFO:tensorflow:global_step/sec: 226.066

INFO:tensorflow:loss = 47.7844, step = 3800 (0.442 sec)

INFO:tensorflow:loss = 47.7844, step = 3800 (0.442 sec)

INFO:tensorflow:global_step/sec: 232.071

INFO:tensorflow:global_step/sec: 232.071

INFO:tensorflow:loss = 44.618782, step = 3900 (0.431 sec)

INFO:tensorflow:loss = 44.618782, step = 3900 (0.431 sec)

INFO:tensorflow:global_step/sec: 233.805

INFO:tensorflow:global_step/sec: 233.805

INFO:tensorflow:loss = 42.523373, step = 4000 (0.428 sec)

INFO:tensorflow:loss = 42.523373, step = 4000 (0.428 sec)

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 4071...

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 4071...

INFO:tensorflow:Saving checkpoints for 4071 into /tmp/tmp2webujvo/model.ckpt.

INFO:tensorflow:Saving checkpoints for 4071 into /tmp/tmp2webujvo/model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 4071...

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 4071...

INFO:tensorflow:Loss for final step: 44.458717.

INFO:tensorflow:Loss for final step: 44.458717.

WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


WARNING:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Signatures INCLUDED in export for Classify: ['serving_default', 'classification']

INFO:tensorflow:Signatures INCLUDED in export for Classify: ['serving_default', 'classification']

INFO:tensorflow:Signatures INCLUDED in export for Regress: ['regression']

INFO:tensorflow:Signatures INCLUDED in export for Regress: ['regression']

INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']

INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Eval: None

INFO:tensorflow:Signatures INCLUDED in export for Eval: None

INFO:tensorflow:Restoring parameters from /tmp/tmp2webujvo/model.ckpt-4071

INFO:tensorflow:Restoring parameters from /tmp/tmp2webujvo/model.ckpt-4071

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets written to: /tmp/estimator/exported_model_dir/temp-1612088417/assets

INFO:tensorflow:Assets written to: /tmp/estimator/exported_model_dir/temp-1612088417/assets

INFO:tensorflow:SavedModel written to: /tmp/estimator/exported_model_dir/temp-1612088417/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/estimator/exported_model_dir/temp-1612088417/saved_model.pb

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2021-01-31T10:20:19Z

INFO:tensorflow:Starting evaluation at 2021-01-31T10:20:19Z

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from /tmp/tmp2webujvo/model.ckpt-4071

INFO:tensorflow:Restoring parameters from /tmp/tmp2webujvo/model.ckpt-4071

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Evaluation [1628/16281]

INFO:tensorflow:Evaluation [1628/16281]

INFO:tensorflow:Evaluation [3256/16281]

INFO:tensorflow:Evaluation [3256/16281]

INFO:tensorflow:Evaluation [4884/16281]

INFO:tensorflow:Evaluation [4884/16281]

INFO:tensorflow:Evaluation [6512/16281]

INFO:tensorflow:Evaluation [6512/16281]

INFO:tensorflow:Evaluation [8140/16281]

INFO:tensorflow:Evaluation [8140/16281]

INFO:tensorflow:Evaluation [9768/16281]

INFO:tensorflow:Evaluation [9768/16281]

INFO:tensorflow:Evaluation [11396/16281]

INFO:tensorflow:Evaluation [11396/16281]

INFO:tensorflow:Evaluation [13024/16281]

INFO:tensorflow:Evaluation [13024/16281]

INFO:tensorflow:Evaluation [14652/16281]

INFO:tensorflow:Evaluation [14652/16281]

INFO:tensorflow:Evaluation [16280/16281]

INFO:tensorflow:Evaluation [16280/16281]

INFO:tensorflow:Evaluation [16281/16281]

INFO:tensorflow:Evaluation [16281/16281]

INFO:tensorflow:Inference Time : 70.85436s

INFO:tensorflow:Inference Time : 70.85436s

INFO:tensorflow:Finished evaluation at 2021-01-31-10:21:29

INFO:tensorflow:Finished evaluation at 2021-01-31-10:21:29

INFO:tensorflow:Saving dict for global step 4071: accuracy = 0.849395, accuracy_baseline = 0.76377374, auc = 0.9018186, auc_precision_recall = 0.96715224, average_loss = 0.32409185, global_step = 4071, label/mean = 0.76377374, loss = 0.32409185, precision = 0.8747654, prediction/mean = 0.76911706, recall = 0.9369522

INFO:tensorflow:Saving dict for global step 4071: accuracy = 0.849395, accuracy_baseline = 0.76377374, auc = 0.9018186, auc_precision_recall = 0.96715224, average_loss = 0.32409185, global_step = 4071, label/mean = 0.76377374, loss = 0.32409185, precision = 0.8747654, prediction/mean = 0.76911706, recall = 0.9369522

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4071: /tmp/tmp2webujvo/model.ckpt-4071

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4071: /tmp/tmp2webujvo/model.ckpt-4071

{'accuracy': 0.849395,
 'accuracy_baseline': 0.76377374,
 'auc': 0.9018186,
 'auc_precision_recall': 0.96715224,
 'average_loss': 0.32409185,
 'global_step': 4071,
 'label/mean': 0.76377374,
 'loss': 0.32409185,
 'precision': 0.8747654,
 'prediction/mean': 0.76911706,
 'recall': 0.9369522}

Yaptığımız

Bu örnekte, nüfus sayım verilerinin bir veri kümesini önceden tf.Transform ve temizlenmiş ve dönüştürülmüş verilerle bir model eğitmek için tf.Transform kullandık. Ayrıca, eğitimli modelimizi bir üretim ortamında çıkarım yapmak için devreye aldığımızda kullanabileceğimiz bir girdi işlevi oluşturduk. Hem eğitim hem de çıkarım için aynı kodu kullanarak veri çarpıklığıyla ilgili herhangi bir sorunu önlüyoruz. Verileri temizlemek için ihtiyaç duyduğumuz dönüşümü gerçekleştirmek için bir Apache Beam dönüşümü oluşturmayı öğrendik. Ayrıca, bu dönüştürülmüş verilerin tf.keras veya tf.estimator kullanarak bir modeli eğitmek için nasıl kullanılacağını da tf.estimator . Bu, TensorFlow Transform'un yapabileceklerinin sadece küçük bir parçası! tf.Transform ve sizin için neler yapabileceğini keşfetmenizi öneririz.