Unisciti alla comunità SIG TFX-Addons e contribuisci a rendere TFX ancora migliore!
Questa pagina è stata tradotta dall'API Cloud Translation.
Switch to English

Analisi del modello utilizzando TFX Pipeline e TensorFlow Model Analysis

In questo tutorial basato su notebook, creeremo ed eseguiremo una pipeline TFX che crea un semplice modello di classificazione e ne analizza le prestazioni su più esecuzioni. Questo notebook è basato sulla pipeline TFX che abbiamo creato in Simple TFX Pipeline Tutorial . Se non hai ancora letto quel tutorial, dovresti leggerlo prima di procedere con questo taccuino.

Mentre modifichi il tuo modello o lo addestri con un nuovo set di dati, devi verificare se il tuo modello è migliorato o peggiorato. Il solo controllo di metriche di primo livello come la precisione potrebbe non essere sufficiente. Ogni modello addestrato deve essere valutato prima di essere inviato alla produzione.

Aggiungeremo un componente Evaluator alla pipeline creata nel tutorial precedente. Il componente Evaluator esegue un'analisi approfondita dei modelli e confronta il nuovo modello con una linea di base per determinare che sono "abbastanza buoni". Viene implementato utilizzando la libreria TensorFlow Model Analysis .

Vedere Comprensione delle pipeline TFX per saperne di più sui vari concetti in TFX.

Impostare

Il processo di configurazione è lo stesso del tutorial precedente.

Per prima cosa dobbiamo installare il pacchetto TFX Python e scaricare il set di dati che useremo per il nostro modello.

Aggiorna Pip

Per evitare di aggiornare Pip in un sistema quando è in esecuzione localmente, verificare che sia in esecuzione in Colab. I sistemi locali possono ovviamente essere aggiornati separatamente.

try:
  import colab
  !pip install -q --upgrade pip
except:
  pass

Installa TFX

# TODO(b/178712706): Stop using legacy resolver after PIP issue is resolved.
pip install -q -U --use-deprecated=legacy-resolver tfx=='0.28.0'

Hai riavviato il runtime?

Se stai utilizzando Google Colab, la prima volta che esegui la cella sopra, devi riavviare il runtime facendo clic sopra il pulsante "RESTART RUNTIME" o utilizzando il menu "Runtime> Restart runtime ...". Ciò è dovuto al modo in cui Colab carica i pacchetti.

Controlla le versioni TensorFlow e TFX.

import tensorflow as tf
print('TensorFlow version: {}'.format(tf.__version__))
import tfx
print('TFX version: {}'.format(tfx.__version__))
TensorFlow version: 2.4.1
TFX version: 0.28.0

Imposta le variabili

Esistono alcune variabili utilizzate per definire una pipeline. Puoi personalizzare queste variabili come desideri. Per impostazione predefinita, tutto l'output dalla pipeline verrà generato nella directory corrente.

import os

PIPELINE_NAME = "penguin-tfma"

# Output directory to store artifacts generated from the pipeline.
PIPELINE_ROOT = os.path.join('pipelines', PIPELINE_NAME)
# Path to a SQLite DB file to use as an MLMD storage.
METADATA_PATH = os.path.join('metadata', PIPELINE_NAME, 'metadata.db')
# Output directory where created models from the pipeline will be exported.
SERVING_MODEL_DIR = os.path.join('serving_model', PIPELINE_NAME)

from absl import logging
logging.set_verbosity(logging.INFO)  # Set default logging level.

Prepara i dati di esempio

Useremo lo stesso set di dati di Palmer Penguins .

Ci sono quattro caratteristiche numeriche in questo set di dati che erano già state normalizzate per avere un intervallo [0,1]. Costruiremo un modello di classificazione che predice le species di pinguini.

Poiché TFX ExampleGen legge gli input da una directory, è necessario creare una directory e copiarvi il set di dati.

import urllib.request
import tempfile

DATA_ROOT = tempfile.mkdtemp(prefix='tfx-data')  # Create a temporary directory.
_data_url = 'https://raw.githubusercontent.com/tensorflow/tfx/master/tfx/examples/penguin/data/penguins_processed.csv'
_data_filepath = os.path.join(DATA_ROOT, "data.csv")
urllib.request.urlretrieve(_data_url, _data_filepath)
('/tmp/tfx-data1vohhald/data.csv', <http.client.HTTPMessage at 0x7fd3a9a12c50>)

Crea una pipeline

Aggiungeremo un componente Evaluator alla pipeline che abbiamo creato nel Tutorial sulla pipeline TFX semplice .

Un componente Evaluator richiede dati di input da un componente ExampleGen e un modello da un componente Trainer e un oggetto tfma.EvalConfig . Facoltativamente, possiamo fornire un modello di base che può essere utilizzato per confrontare le metriche con il modello appena addestrato.

Un programma di valutazione crea due tipi di artefatti di output, ModelEvaluation e ModelBlessing . ModelEvaluation contiene il risultato della valutazione dettagliato che può essere esaminato e visualizzato ulteriormente con la libreria TFMA. ModelBlessing contiene un risultato booleano se il modello ha soddisfatto determinati criteri e può essere utilizzato in componenti successivi come un Pusher come segnale.

Scrivi il codice di addestramento del modello

Useremo lo stesso codice modello del Tutorial sulla pipeline TFX semplice .

_trainer_module_file = 'penguin_trainer.py'
%%writefile {_trainer_module_file}

# Copied from https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple

from typing import List
from absl import logging
import tensorflow as tf
from tensorflow import keras
from tensorflow_transform.tf_metadata import schema_utils

from tfx.components.trainer.executor import TrainerFnArgs
from tfx.components.trainer.fn_args_utils import DataAccessor
from tfx_bsl.tfxio import dataset_options
from tensorflow_metadata.proto.v0 import schema_pb2

_FEATURE_KEYS = [
    'culmen_length_mm', 'culmen_depth_mm', 'flipper_length_mm', 'body_mass_g'
]
_LABEL_KEY = 'species'

_TRAIN_BATCH_SIZE = 20
_EVAL_BATCH_SIZE = 10

# Since we're not generating or creating a schema, we will instead create
# a feature spec.  Since there are a fairly small number of features this is
# manageable for this dataset.
_FEATURE_SPEC = {
    **{
        feature: tf.io.FixedLenFeature(shape=[1], dtype=tf.float32)
           for feature in _FEATURE_KEYS
       },
    _LABEL_KEY: tf.io.FixedLenFeature(shape=[1], dtype=tf.int64)
}


def _input_fn(file_pattern: List[str],
              data_accessor: DataAccessor,
              schema: schema_pb2.Schema,
              batch_size: int = 200) -> tf.data.Dataset:
  """Generates features and label for training.

  Args:
    file_pattern: List of paths or patterns of input tfrecord files.
    data_accessor: DataAccessor for converting input to RecordBatch.
    schema: schema of the input data.
    batch_size: representing the number of consecutive elements of returned
      dataset to combine in a single batch

  Returns:
    A dataset that contains (features, indices) tuple where features is a
      dictionary of Tensors, and indices is a single Tensor of label indices.
  """
  return data_accessor.tf_dataset_factory(
      file_pattern,
      dataset_options.TensorFlowDatasetOptions(
          batch_size=batch_size, label_key=_LABEL_KEY),
      schema=schema).repeat()


def _build_keras_model() -> tf.keras.Model:
  """Creates a DNN Keras model for classifying penguin data.

  Returns:
    A Keras Model.
  """
  # The model below is built with Functional API, please refer to
  # https://www.tensorflow.org/guide/keras/overview for all API options.
  inputs = [keras.layers.Input(shape=(1,), name=f) for f in _FEATURE_KEYS]
  d = keras.layers.concatenate(inputs)
  for _ in range(2):
    d = keras.layers.Dense(8, activation='relu')(d)
  outputs = keras.layers.Dense(3)(d)

  model = keras.Model(inputs=inputs, outputs=outputs)
  model.compile(
      optimizer=keras.optimizers.Adam(1e-2),
      loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
      metrics=[keras.metrics.SparseCategoricalAccuracy()])

  model.summary(print_fn=logging.info)
  return model


# TFX Trainer will call this function.
def run_fn(fn_args: TrainerFnArgs):
  """Train the model based on given args.

  Args:
    fn_args: Holds args used to train the model as name/value pairs.
  """

  # This schema is usually either an output of SchemaGen or a manually-curated
  # version provided by pipeline author. A schema can also derived from TFT
  # graph if a Transform component is used. In the case when either is missing,
  # `schema_from_feature_spec` could be used to generate schema from very simple
  # feature_spec, but the schema returned would be very primitive.
  schema = schema_utils.schema_from_feature_spec(_FEATURE_SPEC)

  train_dataset = _input_fn(
      fn_args.train_files,
      fn_args.data_accessor,
      schema,
      batch_size=_TRAIN_BATCH_SIZE)
  eval_dataset = _input_fn(
      fn_args.eval_files,
      fn_args.data_accessor,
      schema,
      batch_size=_EVAL_BATCH_SIZE)

  model = _build_keras_model()
  model.fit(
      train_dataset,
      steps_per_epoch=fn_args.train_steps,
      validation_data=eval_dataset,
      validation_steps=fn_args.eval_steps)

  # The result of the training should be saved in `fn_args.serving_model_dir`
  # directory.
  model.save(fn_args.serving_model_dir, save_format='tf')
Writing penguin_trainer.py

Scrivi una definizione della pipeline

Definiremo una funzione per creare una pipeline TFX. Oltre al componente Evaluator di cui abbiamo parlato sopra, aggiungeremo un altro nodo chiamato Resolver . Per verificare che un nuovo modello stia migliorando rispetto al modello precedente, è necessario confrontarlo con un modello pubblicato precedente, chiamato baseline. ML Metadata (MLMD) tiene traccia di tutti gli artefatti precedenti della pipeline e Resolver può trovare quello che era l'ultimo modello benedetto - un modello superato con successo da Evaluator - da MLMD utilizzando una classe di strategia chiamata LatestBlessedModelResolver .

import tensorflow_model_analysis as tfma
from tfx.components import CsvExampleGen
from tfx.components import Evaluator
from tfx.components import Pusher
from tfx.dsl.components.common import resolver
from tfx.components import Trainer
from tfx.components.trainer.executor import GenericExecutor
from tfx.dsl.components.base import executor_spec
from tfx.dsl.experimental import latest_blessed_model_resolver
from tfx.orchestration import metadata
from tfx.orchestration import pipeline
from tfx.proto import pusher_pb2
from tfx.proto import trainer_pb2
from tfx.types import Channel
from tfx.types import standard_artifacts

def _create_pipeline(pipeline_name: str, pipeline_root: str, data_root: str,
                     module_file: str, serving_model_dir: str,
                     metadata_path: str) -> pipeline.Pipeline:
  """Creates a three component penguin pipeline with TFX."""
  # Brings data into the pipeline.
  example_gen = CsvExampleGen(input_base=data_root)

  # Uses user-provided Python function that trains a model.
  trainer = Trainer(
      module_file=module_file,
      custom_executor_spec=executor_spec.ExecutorClassSpec(GenericExecutor),
      examples=example_gen.outputs['examples'],
      train_args=trainer_pb2.TrainArgs(num_steps=100),
      eval_args=trainer_pb2.EvalArgs(num_steps=5))

  # NEW: Get the latest blessed model for Evaluator.
  model_resolver = resolver.Resolver(
      instance_name='latest_blessed_model_resolver',
      strategy_class=latest_blessed_model_resolver.LatestBlessedModelResolver,
      model=Channel(type=standard_artifacts.Model),
      model_blessing=Channel(type=standard_artifacts.ModelBlessing))

  # NEW: Uses TFMA to compute evaluation statistics over features of a model and
  #   perform quality validation of a candidate model (compared to a baseline).

  eval_config = tfma.EvalConfig(
      model_specs=[tfma.ModelSpec(label_key='species')],
      slicing_specs=[
          # An empty slice spec means the overall slice, i.e. the whole dataset.
          tfma.SlicingSpec(),
          # Calculate metrics for each penguin species.
          tfma.SlicingSpec(feature_keys=['species']),
          ],
      metrics_specs=[
          tfma.MetricsSpec(per_slice_thresholds={
              'sparse_categorical_accuracy':
                  tfma.config.PerSliceMetricThresholds(thresholds=[
                      tfma.PerSliceMetricThreshold(
                          slicing_specs=[tfma.SlicingSpec()],
                          threshold=tfma.MetricThreshold(
                              value_threshold=tfma.GenericValueThreshold(
                                   lower_bound={'value': 0.6}),
                              # Change threshold will be ignored if there is no
                              # baseline model resolved from MLMD (first run).
                              change_threshold=tfma.GenericChangeThreshold(
                                  direction=tfma.MetricDirection.HIGHER_IS_BETTER,
                                  absolute={'value': -1e-10}))
                       )]),
          })],
      )
  evaluator = Evaluator(
      examples=example_gen.outputs['examples'],
      model=trainer.outputs['model'],
      baseline_model=model_resolver.outputs['model'],
      eval_config=eval_config)

  # Checks whether the model passed the validation steps and pushes the model
  # to a file destination if check passed.
  pusher = Pusher(
      model=trainer.outputs['model'],
      model_blessing=evaluator.outputs['blessing'], # Pass an evaluation result.
      push_destination=pusher_pb2.PushDestination(
          filesystem=pusher_pb2.PushDestination.Filesystem(
              base_directory=serving_model_dir)))

  components = [
      example_gen,
      trainer,

      # Following two components were added to the pipeline.
      model_resolver,
      evaluator,

      pusher,
  ]

  return pipeline.Pipeline(
      pipeline_name=pipeline_name,
      pipeline_root=pipeline_root,
      metadata_connection_config=metadata.sqlite_metadata_connection_config(
          metadata_path),
      components=components)
INFO:absl:tensorflow_ranking is not available: No module named 'tensorflow_ranking'
INFO:absl:tensorflow_text is not available: No module named 'tensorflow_text'
INFO:absl:tensorflow_text is not available: No module named 'tensorflow_text'
WARNING:absl:RuntimeParameter is only supported on Cloud-based DAG runner currently.

Dobbiamo fornire le seguenti informazioni eval_config tramite eval_config :

  • Metriche aggiuntive da configurare (se si desidera più metriche di quelle definite nel modello).
  • Fette da configurare
  • Modellare le soglie di convalida per verificare se la convalida deve essere inclusa

Poiché SparseCategoricalAccuracy era già incluso nella chiamata model.compile() , verrà incluso automaticamente nell'analisi. Quindi non aggiungiamo ulteriori metriche qui. SparseCategoricalAccuracy verrà utilizzato per decidere se anche il modello è sufficientemente buono.

Calcoliamo le metriche per l'intero set di dati e per ogni specie di pinguino. SlicingSpec specifica come aggreghiamo le metriche dichiarate.

Ci sono due soglie che un nuovo modello dovrebbe superare, una è una soglia assoluta di 0,6 e l'altra è una soglia relativa che dovrebbe essere superiore al modello di base. Quando esegui la pipeline per la prima volta, change_threshold verrà ignorato e verrà controllato solo value_threshold. Se si esegue la pipeline più di una volta, il Resolver troverà un modello dell'esecuzione precedente e verrà utilizzato come modello di base per il confronto.

Per ulteriori informazioni, vedere la guida ai componenti di Evaluator .

Esegui la pipeline

Useremo LocalDagRunner come nel tutorial precedente.

import os
from tfx.orchestration.local import local_dag_runner

local_dag_runner.LocalDagRunner().run(_create_pipeline(
      pipeline_name=PIPELINE_NAME,
      pipeline_root=PIPELINE_ROOT,
      data_root=DATA_ROOT,
      module_file=_trainer_module_file,
      serving_model_dir=SERVING_MODEL_DIR,
      metadata_path=METADATA_PATH))
WARNING:absl:`instance_name` is deprecated, please set the node id directly using `with_id()` or the `.id` setter.
INFO:absl:Running pipeline:
 pipeline_info {
  id: "penguin-tfma"
}
nodes {
  pipeline_node {
    node_info {
      type {
        name: "tfx.components.example_gen.csv_example_gen.component.CsvExampleGen"
      }
      id: "CsvExampleGen"
    }
    contexts {
      contexts {
        type {
          name: "pipeline"
        }
        name {
          field_value {
            string_value: "penguin-tfma"
          }
        }
      }
      contexts {
        type {
          name: "pipeline_run"
        }
        name {
          field_value {
            string_value: "2021-04-30T09:18:51.523923"
          }
        }
      }
      contexts {
        type {
          name: "node"
        }
        name {
          field_value {
            string_value: "penguin-tfma.CsvExampleGen"
          }
        }
      }
    }
    outputs {
      outputs {
        key: "examples"
        value {
          artifact_spec {
            type {
              name: "Examples"
              properties {
                key: "span"
                value: INT
              }
              properties {
                key: "split_names"
                value: STRING
              }
              properties {
                key: "version"
                value: INT
              }
            }
          }
        }
      }
    }
    parameters {
      parameters {
        key: "input_base"
        value {
          field_value {
            string_value: "/tmp/tfx-data1vohhald"
          }
        }
      }
      parameters {
        key: "input_config"
        value {
          field_value {
            string_value: "{\n  \"splits\": [\n    {\n      \"name\": \"single_split\",\n      \"pattern\": \"*\"\n    }\n  ]\n}"
          }
        }
      }
      parameters {
        key: "output_config"
        value {
          field_value {
            string_value: "{\n  \"split_config\": {\n    \"splits\": [\n      {\n        \"hash_buckets\": 2,\n        \"name\": \"train\"\n      },\n      {\n        \"hash_buckets\": 1,\n        \"name\": \"eval\"\n      }\n    ]\n  }\n}"
          }
        }
      }
      parameters {
        key: "output_data_format"
        value {
          field_value {
            int_value: 6
          }
        }
      }
    }
    downstream_nodes: "Evaluator"
    downstream_nodes: "Trainer"
    execution_options {
      caching_options {
      }
    }
  }
}
nodes {
  pipeline_node {
    node_info {
      type {
        name: "tfx.dsl.components.common.resolver.Resolver"
      }
      id: "Resolver.latest_blessed_model_resolver"
    }
    contexts {
      contexts {
        type {
          name: "pipeline"
        }
        name {
          field_value {
            string_value: "penguin-tfma"
          }
        }
      }
      contexts {
        type {
          name: "pipeline_run"
        }
        name {
          field_value {
            string_value: "2021-04-30T09:18:51.523923"
          }
        }
      }
      contexts {
        type {
          name: "node"
        }
        name {
          field_value {
            string_value: "penguin-tfma.Resolver.latest_blessed_model_resolver"
          }
        }
      }
    }
    inputs {
      inputs {
        key: "model"
        value {
          channels {
            context_queries {
              type {
                name: "pipeline"
              }
              name {
                field_value {
                  string_value: "penguin-tfma"
                }
              }
            }
            artifact_query {
              type {
                name: "Model"
              }
            }
          }
        }
      }
      inputs {
        key: "model_blessing"
        value {
          channels {
            context_queries {
              type {
                name: "pipeline"
              }
              name {
                field_value {
                  string_value: "penguin-tfma"
                }
              }
            }
            artifact_query {
              type {
                name: "ModelBlessing"
              }
            }
          }
        }
      }
      resolver_config {
        resolver_steps {
          class_path: "tfx.dsl.experimental.latest_blessed_model_resolver.LatestBlessedModelResolver"
          config_json: "{}"
          input_keys: "model"
          input_keys: "model_blessing"
        }
      }
    }
    downstream_nodes: "Evaluator"
    execution_options {
      caching_options {
      }
    }
  }
}
nodes {
  pipeline_node {
    node_info {
      type {
        name: "tfx.components.trainer.component.Trainer"
      }
      id: "Trainer"
    }
    contexts {
      contexts {
        type {
          name: "pipeline"
        }
        name {
          field_value {
            string_value: "penguin-tfma"
          }
        }
      }
      contexts {
        type {
          name: "pipeline_run"
        }
        name {
          field_value {
            string_value: "2021-04-30T09:18:51.523923"
          }
        }
      }
      contexts {
        type {
          name: "node"
        }
        name {
          field_value {
            string_value: "penguin-tfma.Trainer"
          }
        }
      }
    }
    inputs {
      inputs {
        key: "examples"
        value {
          channels {
            producer_node_query {
              id: "CsvExampleGen"
            }
            context_queries {
              type {
                name: "pipeline"
              }
              name {
                field_value {
                  string_value: "penguin-tfma"
                }
              }
            }
            context_queries {
              type {
                name: "pipeline_run"
              }
              name {
                field_value {
                  string_value: "2021-04-30T09:18:51.523923"
                }
              }
            }
            context_queries {
              type {
                name: "node"
              }
              name {
                field_value {
                  string_value: "penguin-tfma.CsvExampleGen"
                }
              }
            }
            artifact_query {
              type {
                name: "Examples"
              }
            }
            output_key: "examples"
          }
        }
      }
    }
    outputs {
      outputs {
        key: "model"
        value {
          artifact_spec {
            type {
              name: "Model"
            }
          }
        }
      }
      outputs {
        key: "model_run"
        value {
          artifact_spec {
            type {
              name: "ModelRun"
            }
          }
        }
      }
    }
    parameters {
      parameters {
        key: "custom_config"
        value {
          field_value {
            string_value: "null"
          }
        }
      }
      parameters {
        key: "eval_args"
        value {
          field_value {
            string_value: "{\n  \"num_steps\": 5\n}"
          }
        }
      }
      parameters {
        key: "module_file"
        value {
          field_value {
            string_value: "penguin_trainer.py"
          }
        }
      }
      parameters {
        key: "train_args"
        value {
          field_value {
            string_value: "{\n  \"num_steps\": 100\n}"
          }
        }
      }
    }
    upstream_nodes: "CsvExampleGen"
    downstream_nodes: "Evaluator"
    downstream_nodes: "Pusher"
    execution_options {
      caching_options {
      }
    }
  }
}
nodes {
  pipeline_node {
    node_info {
      type {
        name: "tfx.components.evaluator.component.Evaluator"
      }
      id: "Evaluator"
    }
    contexts {
      contexts {
        type {
          name: "pipeline"
        }
        name {
          field_value {
            string_value: "penguin-tfma"
          }
        }
      }
      contexts {
        type {
          name: "pipeline_run"
        }
        name {
          field_value {
            string_value: "2021-04-30T09:18:51.523923"
          }
        }
      }
      contexts {
        type {
          name: "node"
        }
        name {
          field_value {
            string_value: "penguin-tfma.Evaluator"
          }
        }
      }
    }
    inputs {
      inputs {
        key: "baseline_model"
        value {
          channels {
            producer_node_query {
              id: "Resolver.latest_blessed_model_resolver"
            }
            context_queries {
              type {
                name: "pipeline"
              }
              name {
                field_value {
                  string_value: "penguin-tfma"
                }
              }
            }
            context_queries {
              type {
                name: "pipeline_run"
              }
              name {
                field_value {
                  string_value: "2021-04-30T09:18:51.523923"
                }
              }
            }
            context_queries {
              type {
                name: "node"
              }
              name {
                field_value {
                  string_value: "penguin-tfma.Resolver.latest_blessed_model_resolver"
                }
              }
            }
            artifact_query {
              type {
                name: "Model"
              }
            }
            output_key: "model"
          }
        }
      }
      inputs {
        key: "examples"
        value {
          channels {
            producer_node_query {
              id: "CsvExampleGen"
            }
            context_queries {
              type {
                name: "pipeline"
              }
              name {
                field_value {
                  string_value: "penguin-tfma"
                }
              }
            }
            context_queries {
              type {
                name: "pipeline_run"
              }
              name {
                field_value {
                  string_value: "2021-04-30T09:18:51.523923"
                }
              }
            }
            context_queries {
              type {
                name: "node"
              }
              name {
                field_value {
                  string_value: "penguin-tfma.CsvExampleGen"
                }
              }
            }
            artifact_query {
              type {
                name: "Examples"
              }
            }
            output_key: "examples"
          }
        }
      }
      inputs {
        key: "model"
        value {
          channels {
            producer_node_query {
              id: "Trainer"
            }
            context_queries {
              type {
                name: "pipeline"
              }
              name {
                field_value {
                  string_value: "penguin-tfma"
                }
              }
            }
            context_queries {
              type {
                name: "pipeline_run"
              }
              name {
                field_value {
                  string_value: "2021-04-30T09:18:51.523923"
                }
              }
            }
            context_queries {
              type {
                name: "node"
              }
              name {
                field_value {
                  string_value: "penguin-tfma.Trainer"
                }
              }
            }
            artifact_query {
              type {
                name: "Model"
              }
            }
            output_key: "model"
          }
        }
      }
    }
    outputs {
      outputs {
        key: "blessing"
        value {
          artifact_spec {
            type {
              name: "ModelBlessing"
            }
          }
        }
      }
      outputs {
        key: "evaluation"
        value {
          artifact_spec {
            type {
              name: "ModelEvaluation"
            }
          }
        }
      }
    }
    parameters {
      parameters {
        key: "eval_config"
        value {
          field_value {
            string_value: "{\n  \"metrics_specs\": [\n    {\n      \"per_slice_thresholds\": {\n        \"sparse_categorical_accuracy\": {\n          \"thresholds\": [\n            {\n              \"slicing_specs\": [\n                {}\n              ],\n              \"threshold\": {\n                \"change_threshold\": {\n                  \"absolute\": -1e-10,\n                  \"direction\": \"HIGHER_IS_BETTER\"\n                },\n                \"value_threshold\": {\n                  \"lower_bound\": 0.6\n                }\n              }\n            }\n          ]\n        }\n      }\n    }\n  ],\n  \"model_specs\": [\n    {\n      \"label_key\": \"species\"\n    }\n  ],\n  \"slicing_specs\": [\n    {},\n    {\n      \"feature_keys\": [\n        \"species\"\n      ]\n    }\n  ]\n}"
          }
        }
      }
      parameters {
        key: "example_splits"
        value {
          field_value {
            string_value: "null"
          }
        }
      }
    }
    upstream_nodes: "CsvExampleGen"
    upstream_nodes: "Resolver.latest_blessed_model_resolver"
    upstream_nodes: "Trainer"
    downstream_nodes: "Pusher"
    execution_options {
      caching_options {
      }
    }
  }
}
nodes {
  pipeline_node {
    node_info {
      type {
        name: "tfx.components.pusher.component.Pusher"
      }
      id: "Pusher"
    }
    contexts {
      contexts {
        type {
          name: "pipeline"
        }
        name {
          field_value {
            string_value: "penguin-tfma"
          }
        }
      }
      contexts {
        type {
          name: "pipeline_run"
        }
        name {
          field_value {
            string_value: "2021-04-30T09:18:51.523923"
          }
        }
      }
      contexts {
        type {
          name: "node"
        }
        name {
          field_value {
            string_value: "penguin-tfma.Pusher"
          }
        }
      }
    }
    inputs {
      inputs {
        key: "model"
        value {
          channels {
            producer_node_query {
              id: "Trainer"
            }
            context_queries {
              type {
                name: "pipeline"
              }
              name {
                field_value {
                  string_value: "penguin-tfma"
                }
              }
            }
            context_queries {
              type {
                name: "pipeline_run"
              }
              name {
                field_value {
                  string_value: "2021-04-30T09:18:51.523923"
                }
              }
            }
            context_queries {
              type {
                name: "node"
              }
              name {
                field_value {
                  string_value: "penguin-tfma.Trainer"
                }
              }
            }
            artifact_query {
              type {
                name: "Model"
              }
            }
            output_key: "model"
          }
        }
      }
      inputs {
        key: "model_blessing"
        value {
          channels {
            producer_node_query {
              id: "Evaluator"
            }
            context_queries {
              type {
                name: "pipeline"
              }
              name {
                field_value {
                  string_value: "penguin-tfma"
                }
              }
            }
            context_queries {
              type {
                name: "pipeline_run"
              }
              name {
                field_value {
                  string_value: "2021-04-30T09:18:51.523923"
                }
              }
            }
            context_queries {
              type {
                name: "node"
              }
              name {
                field_value {
                  string_value: "penguin-tfma.Evaluator"
                }
              }
            }
            artifact_query {
              type {
                name: "ModelBlessing"
              }
            }
            output_key: "blessing"
          }
        }
      }
    }
    outputs {
      outputs {
        key: "pushed_model"
        value {
          artifact_spec {
            type {
              name: "PushedModel"
            }
          }
        }
      }
    }
    parameters {
      parameters {
        key: "custom_config"
        value {
          field_value {
            string_value: "null"
          }
        }
      }
      parameters {
        key: "push_destination"
        value {
          field_value {
            string_value: "{\n  \"filesystem\": {\n    \"base_directory\": \"serving_model/penguin-tfma\"\n  }\n}"
          }
        }
      }
    }
    upstream_nodes: "Evaluator"
    upstream_nodes: "Trainer"
    execution_options {
      caching_options {
      }
    }
  }
}
runtime_spec {
  pipeline_root {
    field_value {
      string_value: "pipelines/penguin-tfma"
    }
  }
  pipeline_run_id {
    field_value {
      string_value: "2021-04-30T09:18:51.523923"
    }
  }
}
execution_mode: SYNC
deployment_config {
  type_url: "type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig"
  value: "\n\220\001\n\007Trainer\022\204\001\nOtype.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\0221\n/tfx.components.trainer.executor.GenericExecutor\n\243\001\n\rCsvExampleGen\022\221\001\nOtype.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\022>\n<tfx.components.example_gen.csv_example_gen.executor.Executor\n\206\001\n\006Pusher\022|\nOtype.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\022)\n\'tfx.components.pusher.executor.Executor\n\214\001\n\tEvaluator\022\177\nOtype.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\022,\n*tfx.components.evaluator.executor.Executor\022\216\001\n\rCsvExampleGen\022}\nOtype.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\022*\n(tfx.components.example_gen.driver.Driver*[\n0type.googleapis.com/ml_metadata.ConnectionConfig\022\'\032%\n!metadata/penguin-tfma/metadata.db\020\003"
}

INFO:absl:Using deployment config:
 executor_specs {
  key: "CsvExampleGen"
  value {
    python_class_executable_spec {
      class_path: "tfx.components.example_gen.csv_example_gen.executor.Executor"
    }
  }
}
executor_specs {
  key: "Evaluator"
  value {
    python_class_executable_spec {
      class_path: "tfx.components.evaluator.executor.Executor"
    }
  }
}
executor_specs {
  key: "Pusher"
  value {
    python_class_executable_spec {
      class_path: "tfx.components.pusher.executor.Executor"
    }
  }
}
executor_specs {
  key: "Trainer"
  value {
    python_class_executable_spec {
      class_path: "tfx.components.trainer.executor.GenericExecutor"
    }
  }
}
custom_driver_specs {
  key: "CsvExampleGen"
  value {
    python_class_executable_spec {
      class_path: "tfx.components.example_gen.driver.Driver"
    }
  }
}
metadata_connection_config {
  sqlite {
    filename_uri: "metadata/penguin-tfma/metadata.db"
    connection_mode: READWRITE_OPENCREATE
  }
}

INFO:absl:Using connection config:
 sqlite {
  filename_uri: "metadata/penguin-tfma/metadata.db"
  connection_mode: READWRITE_OPENCREATE
}

INFO:absl:Component CsvExampleGen is running.
INFO:absl:Running launcher for node_info {
  type {
    name: "tfx.components.example_gen.csv_example_gen.component.CsvExampleGen"
  }
  id: "CsvExampleGen"
}
contexts {
  contexts {
    type {
      name: "pipeline"
    }
    name {
      field_value {
        string_value: "penguin-tfma"
      }
    }
  }
  contexts {
    type {
      name: "pipeline_run"
    }
    name {
      field_value {
        string_value: "2021-04-30T09:18:51.523923"
      }
    }
  }
  contexts {
    type {
      name: "node"
    }
    name {
      field_value {
        string_value: "penguin-tfma.CsvExampleGen"
      }
    }
  }
}
outputs {
  outputs {
    key: "examples"
    value {
      artifact_spec {
        type {
          name: "Examples"
          properties {
            key: "span"
            value: INT
          }
          properties {
            key: "split_names"
            value: STRING
          }
          properties {
            key: "version"
            value: INT
          }
        }
      }
    }
  }
}
parameters {
  parameters {
    key: "input_base"
    value {
      field_value {
        string_value: "/tmp/tfx-data1vohhald"
      }
    }
  }
  parameters {
    key: "input_config"
    value {
      field_value {
        string_value: "{\n  \"splits\": [\n    {\n      \"name\": \"single_split\",\n      \"pattern\": \"*\"\n    }\n  ]\n}"
      }
    }
  }
  parameters {
    key: "output_config"
    value {
      field_value {
        string_value: "{\n  \"split_config\": {\n    \"splits\": [\n      {\n        \"hash_buckets\": 2,\n        \"name\": \"train\"\n      },\n      {\n        \"hash_buckets\": 1,\n        \"name\": \"eval\"\n      }\n    ]\n  }\n}"
      }
    }
  }
  parameters {
    key: "output_data_format"
    value {
      field_value {
        int_value: 6
      }
    }
  }
}
downstream_nodes: "Evaluator"
downstream_nodes: "Trainer"
execution_options {
  caching_options {
  }
}

INFO:absl:MetadataStore with DB connection initialized
INFO:absl:select span and version = (0, None)
INFO:absl:latest span and version = (0, None)
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:Going to run a new execution 1
INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=1, input_dict={}, output_dict=defaultdict(<class 'list'>, {'examples': [Artifact(artifact: uri: "pipelines/penguin-tfma/CsvExampleGen/examples/1"
custom_properties {
  key: "input_fingerprint"
  value {
    string_value: "split:single_split,num_files:1,total_bytes:25648,xor_checksum:1619774329,sum_checksum:1619774329"
  }
}
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:CsvExampleGen:examples:0"
  }
}
custom_properties {
  key: "span"
  value {
    string_value: "0"
  }
}
, artifact_type: name: "Examples"
properties {
  key: "span"
  value: INT
}
properties {
  key: "split_names"
  value: STRING
}
properties {
  key: "version"
  value: INT
}
)]}), exec_properties={'input_base': '/tmp/tfx-data1vohhald', 'input_config': '{\n  "splits": [\n    {\n      "name": "single_split",\n      "pattern": "*"\n    }\n  ]\n}', 'output_config': '{\n  "split_config": {\n    "splits": [\n      {\n        "hash_buckets": 2,\n        "name": "train"\n      },\n      {\n        "hash_buckets": 1,\n        "name": "eval"\n      }\n    ]\n  }\n}', 'output_data_format': 6, 'span': 0, 'version': None, 'input_fingerprint': 'split:single_split,num_files:1,total_bytes:25648,xor_checksum:1619774329,sum_checksum:1619774329'}, execution_output_uri='pipelines/penguin-tfma/CsvExampleGen/.system/executor_execution/1/executor_output.pb', stateful_working_dir='pipelines/penguin-tfma/CsvExampleGen/.system/stateful_working_dir/2021-04-30T09:18:51.523923', tmp_dir='pipelines/penguin-tfma/CsvExampleGen/.system/executor_execution/1/.temp/', pipeline_node=node_info {
  type {
    name: "tfx.components.example_gen.csv_example_gen.component.CsvExampleGen"
  }
  id: "CsvExampleGen"
}
contexts {
  contexts {
    type {
      name: "pipeline"
    }
    name {
      field_value {
        string_value: "penguin-tfma"
      }
    }
  }
  contexts {
    type {
      name: "pipeline_run"
    }
    name {
      field_value {
        string_value: "2021-04-30T09:18:51.523923"
      }
    }
  }
  contexts {
    type {
      name: "node"
    }
    name {
      field_value {
        string_value: "penguin-tfma.CsvExampleGen"
      }
    }
  }
}
outputs {
  outputs {
    key: "examples"
    value {
      artifact_spec {
        type {
          name: "Examples"
          properties {
            key: "span"
            value: INT
          }
          properties {
            key: "split_names"
            value: STRING
          }
          properties {
            key: "version"
            value: INT
          }
        }
      }
    }
  }
}
parameters {
  parameters {
    key: "input_base"
    value {
      field_value {
        string_value: "/tmp/tfx-data1vohhald"
      }
    }
  }
  parameters {
    key: "input_config"
    value {
      field_value {
        string_value: "{\n  \"splits\": [\n    {\n      \"name\": \"single_split\",\n      \"pattern\": \"*\"\n    }\n  ]\n}"
      }
    }
  }
  parameters {
    key: "output_config"
    value {
      field_value {
        string_value: "{\n  \"split_config\": {\n    \"splits\": [\n      {\n        \"hash_buckets\": 2,\n        \"name\": \"train\"\n      },\n      {\n        \"hash_buckets\": 1,\n        \"name\": \"eval\"\n      }\n    ]\n  }\n}"
      }
    }
  }
  parameters {
    key: "output_data_format"
    value {
      field_value {
        int_value: 6
      }
    }
  }
}
downstream_nodes: "Evaluator"
downstream_nodes: "Trainer"
execution_options {
  caching_options {
  }
}
, pipeline_info=id: "penguin-tfma"
, pipeline_run_id='2021-04-30T09:18:51.523923')
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['-f', '/tmp/tmp2mq186ss.json', '--HistoryManager.hist_file=:memory:']
INFO:absl:Attempting to infer TFX Python dependency for beam
INFO:absl:Copying all content from install dir /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tfx to temp dir /tmp/tmpkouuipsi/build/tfx
INFO:absl:Generating a temp setup file at /tmp/tmpkouuipsi/build/tfx/setup.py
INFO:absl:Creating temporary sdist package, logs available at /tmp/tmpkouuipsi/build/tfx/setup.log
INFO:absl:Added --extra_package=/tmp/tmpkouuipsi/build/tfx/dist/tfx_ephemeral-0.28.0.tar.gz to beam args
INFO:absl:Generating examples.
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.
INFO:absl:Processing input csv data /tmp/tfx-data1vohhald/* to TFExample.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['-f', '/tmp/tmp2mq186ss.json', '--HistoryManager.hist_file=:memory:']
WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
INFO:absl:Examples generated.
INFO:absl:Cleaning up stateless execution info.
INFO:absl:Execution 1 succeeded.
INFO:absl:Cleaning up stateful execution info.
INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'examples': [Artifact(artifact: uri: "pipelines/penguin-tfma/CsvExampleGen/examples/1"
properties {
  key: "split_names"
  value {
    string_value: "[\"train\", \"eval\"]"
  }
}
custom_properties {
  key: "input_fingerprint"
  value {
    string_value: "split:single_split,num_files:1,total_bytes:25648,xor_checksum:1619774329,sum_checksum:1619774329"
  }
}
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:CsvExampleGen:examples:0"
  }
}
custom_properties {
  key: "payload_format"
  value {
    string_value: "FORMAT_TF_EXAMPLE"
  }
}
custom_properties {
  key: "span"
  value {
    string_value: "0"
  }
}
, artifact_type: name: "Examples"
properties {
  key: "span"
  value: INT
}
properties {
  key: "split_names"
  value: STRING
}
properties {
  key: "version"
  value: INT
}
)]}) for exeuction 1
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:Component CsvExampleGen is finished.
INFO:absl:Component Resolver.latest_blessed_model_resolver is running.
INFO:absl:Running launcher for node_info {
  type {
    name: "tfx.dsl.components.common.resolver.Resolver"
  }
  id: "Resolver.latest_blessed_model_resolver"
}
contexts {
  contexts {
    type {
      name: "pipeline"
    }
    name {
      field_value {
        string_value: "penguin-tfma"
      }
    }
  }
  contexts {
    type {
      name: "pipeline_run"
    }
    name {
      field_value {
        string_value: "2021-04-30T09:18:51.523923"
      }
    }
  }
  contexts {
    type {
      name: "node"
    }
    name {
      field_value {
        string_value: "penguin-tfma.Resolver.latest_blessed_model_resolver"
      }
    }
  }
}
inputs {
  inputs {
    key: "model"
    value {
      channels {
        context_queries {
          type {
            name: "pipeline"
          }
          name {
            field_value {
              string_value: "penguin-tfma"
            }
          }
        }
        artifact_query {
          type {
            name: "Model"
          }
        }
      }
    }
  }
  inputs {
    key: "model_blessing"
    value {
      channels {
        context_queries {
          type {
            name: "pipeline"
          }
          name {
            field_value {
              string_value: "penguin-tfma"
            }
          }
        }
        artifact_query {
          type {
            name: "ModelBlessing"
          }
        }
      }
    }
  }
  resolver_config {
    resolver_steps {
      class_path: "tfx.dsl.experimental.latest_blessed_model_resolver.LatestBlessedModelResolver"
      config_json: "{}"
      input_keys: "model"
      input_keys: "model_blessing"
    }
  }
}
downstream_nodes: "Evaluator"
execution_options {
  caching_options {
  }
}

INFO:absl:Running as an resolver node.
INFO:absl:MetadataStore with DB connection initialized
WARNING:absl:Artifact type ModelBlessing is not found in MLMD.
WARNING:absl:Artifact type Model is not found in MLMD.
INFO:absl:Component Resolver.latest_blessed_model_resolver is finished.
INFO:absl:Component Trainer is running.
INFO:absl:Running launcher for node_info {
  type {
    name: "tfx.components.trainer.component.Trainer"
  }
  id: "Trainer"
}
contexts {
  contexts {
    type {
      name: "pipeline"
    }
    name {
      field_value {
        string_value: "penguin-tfma"
      }
    }
  }
  contexts {
    type {
      name: "pipeline_run"
    }
    name {
      field_value {
        string_value: "2021-04-30T09:18:51.523923"
      }
    }
  }
  contexts {
    type {
      name: "node"
    }
    name {
      field_value {
        string_value: "penguin-tfma.Trainer"
      }
    }
  }
}
inputs {
  inputs {
    key: "examples"
    value {
      channels {
        producer_node_query {
          id: "CsvExampleGen"
        }
        context_queries {
          type {
            name: "pipeline"
          }
          name {
            field_value {
              string_value: "penguin-tfma"
            }
          }
        }
        context_queries {
          type {
            name: "pipeline_run"
          }
          name {
            field_value {
              string_value: "2021-04-30T09:18:51.523923"
            }
          }
        }
        context_queries {
          type {
            name: "node"
          }
          name {
            field_value {
              string_value: "penguin-tfma.CsvExampleGen"
            }
          }
        }
        artifact_query {
          type {
            name: "Examples"
          }
        }
        output_key: "examples"
      }
    }
  }
}
outputs {
  outputs {
    key: "model"
    value {
      artifact_spec {
        type {
          name: "Model"
        }
      }
    }
  }
  outputs {
    key: "model_run"
    value {
      artifact_spec {
        type {
          name: "ModelRun"
        }
      }
    }
  }
}
parameters {
  parameters {
    key: "custom_config"
    value {
      field_value {
        string_value: "null"
      }
    }
  }
  parameters {
    key: "eval_args"
    value {
      field_value {
        string_value: "{\n  \"num_steps\": 5\n}"
      }
    }
  }
  parameters {
    key: "module_file"
    value {
      field_value {
        string_value: "penguin_trainer.py"
      }
    }
  }
  parameters {
    key: "train_args"
    value {
      field_value {
        string_value: "{\n  \"num_steps\": 100\n}"
      }
    }
  }
}
upstream_nodes: "CsvExampleGen"
downstream_nodes: "Evaluator"
downstream_nodes: "Pusher"
execution_options {
  caching_options {
  }
}

INFO:absl:MetadataStore with DB connection initialized
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:Going to run a new execution 3
INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=3, input_dict={'examples': [Artifact(artifact: id: 1
type_id: 6
uri: "pipelines/penguin-tfma/CsvExampleGen/examples/1"
properties {
  key: "split_names"
  value {
    string_value: "[\"train\", \"eval\"]"
  }
}
custom_properties {
  key: "input_fingerprint"
  value {
    string_value: "split:single_split,num_files:1,total_bytes:25648,xor_checksum:1619774329,sum_checksum:1619774329"
  }
}
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:CsvExampleGen:examples:0"
  }
}
custom_properties {
  key: "payload_format"
  value {
    string_value: "FORMAT_TF_EXAMPLE"
  }
}
custom_properties {
  key: "span"
  value {
    string_value: "0"
  }
}
state: LIVE
create_time_since_epoch: 1619774333893
last_update_time_since_epoch: 1619774333893
, artifact_type: id: 6
name: "Examples"
properties {
  key: "span"
  value: INT
}
properties {
  key: "split_names"
  value: STRING
}
properties {
  key: "version"
  value: INT
}
)]}, output_dict=defaultdict(<class 'list'>, {'model_run': [Artifact(artifact: uri: "pipelines/penguin-tfma/Trainer/model_run/3"
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:Trainer:model_run:0"
  }
}
, artifact_type: name: "ModelRun"
)], 'model': [Artifact(artifact: uri: "pipelines/penguin-tfma/Trainer/model/3"
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:Trainer:model:0"
  }
}
, artifact_type: name: "Model"
)]}), exec_properties={'train_args': '{\n  "num_steps": 100\n}', 'module_file': 'penguin_trainer.py', 'custom_config': 'null', 'eval_args': '{\n  "num_steps": 5\n}'}, execution_output_uri='pipelines/penguin-tfma/Trainer/.system/executor_execution/3/executor_output.pb', stateful_working_dir='pipelines/penguin-tfma/Trainer/.system/stateful_working_dir/2021-04-30T09:18:51.523923', tmp_dir='pipelines/penguin-tfma/Trainer/.system/executor_execution/3/.temp/', pipeline_node=node_info {
  type {
    name: "tfx.components.trainer.component.Trainer"
  }
  id: "Trainer"
}
contexts {
  contexts {
    type {
      name: "pipeline"
    }
    name {
      field_value {
        string_value: "penguin-tfma"
      }
    }
  }
  contexts {
    type {
      name: "pipeline_run"
    }
    name {
      field_value {
        string_value: "2021-04-30T09:18:51.523923"
      }
    }
  }
  contexts {
    type {
      name: "node"
    }
    name {
      field_value {
        string_value: "penguin-tfma.Trainer"
      }
    }
  }
}
inputs {
  inputs {
    key: "examples"
    value {
      channels {
        producer_node_query {
          id: "CsvExampleGen"
        }
        context_queries {
          type {
            name: "pipeline"
          }
          name {
            field_value {
              string_value: "penguin-tfma"
            }
          }
        }
        context_queries {
          type {
            name: "pipeline_run"
          }
          name {
            field_value {
              string_value: "2021-04-30T09:18:51.523923"
            }
          }
        }
        context_queries {
          type {
            name: "node"
          }
          name {
            field_value {
              string_value: "penguin-tfma.CsvExampleGen"
            }
          }
        }
        artifact_query {
          type {
            name: "Examples"
          }
        }
        output_key: "examples"
      }
    }
  }
}
outputs {
  outputs {
    key: "model"
    value {
      artifact_spec {
        type {
          name: "Model"
        }
      }
    }
  }
  outputs {
    key: "model_run"
    value {
      artifact_spec {
        type {
          name: "ModelRun"
        }
      }
    }
  }
}
parameters {
  parameters {
    key: "custom_config"
    value {
      field_value {
        string_value: "null"
      }
    }
  }
  parameters {
    key: "eval_args"
    value {
      field_value {
        string_value: "{\n  \"num_steps\": 5\n}"
      }
    }
  }
  parameters {
    key: "module_file"
    value {
      field_value {
        string_value: "penguin_trainer.py"
      }
    }
  }
  parameters {
    key: "train_args"
    value {
      field_value {
        string_value: "{\n  \"num_steps\": 100\n}"
      }
    }
  }
}
upstream_nodes: "CsvExampleGen"
downstream_nodes: "Evaluator"
downstream_nodes: "Pusher"
execution_options {
  caching_options {
  }
}
, pipeline_info=id: "penguin-tfma"
, pipeline_run_id='2021-04-30T09:18:51.523923')
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['-f', '/tmp/tmp2mq186ss.json', '--HistoryManager.hist_file=:memory:']
INFO:absl:Attempting to infer TFX Python dependency for beam
INFO:absl:Copying all content from install dir /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tfx to temp dir /tmp/tmp8iv7n11o/build/tfx
INFO:absl:Generating a temp setup file at /tmp/tmp8iv7n11o/build/tfx/setup.py
INFO:absl:Creating temporary sdist package, logs available at /tmp/tmp8iv7n11o/build/tfx/setup.log
INFO:absl:Added --extra_package=/tmp/tmp8iv7n11o/build/tfx/dist/tfx_ephemeral-0.28.0.tar.gz to beam args
INFO:absl:Train on the 'train' split when train_args.splits is not set.
INFO:absl:Evaluate on the 'eval' split when eval_args.splits is not set.
INFO:absl:Loading penguin_trainer.py because it has not been loaded before.
INFO:absl:Training model.
INFO:absl:Feature body_mass_g has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature culmen_depth_mm has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature culmen_length_mm has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature flipper_length_mm has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature species has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature body_mass_g has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature culmen_depth_mm has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature culmen_length_mm has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature flipper_length_mm has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature species has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature body_mass_g has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature culmen_depth_mm has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature culmen_length_mm has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature flipper_length_mm has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature species has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature body_mass_g has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature culmen_depth_mm has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature culmen_length_mm has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature flipper_length_mm has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature species has a shape dim {
  size: 1
}
. Setting to DenseTensor.
INFO:absl:Model: "model"
INFO:absl:__________________________________________________________________________________________________
INFO:absl:Layer (type)                    Output Shape         Param #     Connected to                     
INFO:absl:==================================================================================================
INFO:absl:culmen_length_mm (InputLayer)   [(None, 1)]          0                                            
INFO:absl:__________________________________________________________________________________________________
INFO:absl:culmen_depth_mm (InputLayer)    [(None, 1)]          0                                            
INFO:absl:__________________________________________________________________________________________________
INFO:absl:flipper_length_mm (InputLayer)  [(None, 1)]          0                                            
INFO:absl:__________________________________________________________________________________________________
INFO:absl:body_mass_g (InputLayer)        [(None, 1)]          0                                            
INFO:absl:__________________________________________________________________________________________________
INFO:absl:concatenate (Concatenate)       (None, 4)            0           culmen_length_mm[0][0]           
INFO:absl:                                                                 culmen_depth_mm[0][0]            
INFO:absl:                                                                 flipper_length_mm[0][0]          
INFO:absl:                                                                 body_mass_g[0][0]                
INFO:absl:__________________________________________________________________________________________________
INFO:absl:dense (Dense)                   (None, 8)            40          concatenate[0][0]                
INFO:absl:__________________________________________________________________________________________________
INFO:absl:dense_1 (Dense)                 (None, 8)            72          dense[0][0]                      
INFO:absl:__________________________________________________________________________________________________
INFO:absl:dense_2 (Dense)                 (None, 3)            27          dense_1[0][0]                    
INFO:absl:==================================================================================================
INFO:absl:Total params: 139
INFO:absl:Trainable params: 139
INFO:absl:Non-trainable params: 0
INFO:absl:__________________________________________________________________________________________________
100/100 [==============================] - 1s 6ms/step - loss: 0.9176 - sparse_categorical_accuracy: 0.5134 - val_loss: 0.3488 - val_sparse_categorical_accuracy: 0.8400
INFO:tensorflow:Assets written to: pipelines/penguin-tfma/Trainer/model/3/serving_model_dir/assets
INFO:tensorflow:Assets written to: pipelines/penguin-tfma/Trainer/model/3/serving_model_dir/assets
INFO:absl:Training complete. Model written to pipelines/penguin-tfma/Trainer/model/3/serving_model_dir. ModelRun written to pipelines/penguin-tfma/Trainer/model_run/3
INFO:absl:Cleaning up stateless execution info.
INFO:absl:Execution 3 succeeded.
INFO:absl:Cleaning up stateful execution info.
INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'model_run': [Artifact(artifact: uri: "pipelines/penguin-tfma/Trainer/model_run/3"
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:Trainer:model_run:0"
  }
}
, artifact_type: name: "ModelRun"
)], 'model': [Artifact(artifact: uri: "pipelines/penguin-tfma/Trainer/model/3"
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:Trainer:model:0"
  }
}
, artifact_type: name: "Model"
)]}) for exeuction 3
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:Component Trainer is finished.
INFO:absl:Component Evaluator is running.
INFO:absl:Running launcher for node_info {
  type {
    name: "tfx.components.evaluator.component.Evaluator"
  }
  id: "Evaluator"
}
contexts {
  contexts {
    type {
      name: "pipeline"
    }
    name {
      field_value {
        string_value: "penguin-tfma"
      }
    }
  }
  contexts {
    type {
      name: "pipeline_run"
    }
    name {
      field_value {
        string_value: "2021-04-30T09:18:51.523923"
      }
    }
  }
  contexts {
    type {
      name: "node"
    }
    name {
      field_value {
        string_value: "penguin-tfma.Evaluator"
      }
    }
  }
}
inputs {
  inputs {
    key: "baseline_model"
    value {
      channels {
        producer_node_query {
          id: "Resolver.latest_blessed_model_resolver"
        }
        context_queries {
          type {
            name: "pipeline"
          }
          name {
            field_value {
              string_value: "penguin-tfma"
            }
          }
        }
        context_queries {
          type {
            name: "pipeline_run"
          }
          name {
            field_value {
              string_value: "2021-04-30T09:18:51.523923"
            }
          }
        }
        context_queries {
          type {
            name: "node"
          }
          name {
            field_value {
              string_value: "penguin-tfma.Resolver.latest_blessed_model_resolver"
            }
          }
        }
        artifact_query {
          type {
            name: "Model"
          }
        }
        output_key: "model"
      }
    }
  }
  inputs {
    key: "examples"
    value {
      channels {
        producer_node_query {
          id: "CsvExampleGen"
        }
        context_queries {
          type {
            name: "pipeline"
          }
          name {
            field_value {
              string_value: "penguin-tfma"
            }
          }
        }
        context_queries {
          type {
            name: "pipeline_run"
          }
          name {
            field_value {
              string_value: "2021-04-30T09:18:51.523923"
            }
          }
        }
        context_queries {
          type {
            name: "node"
          }
          name {
            field_value {
              string_value: "penguin-tfma.CsvExampleGen"
            }
          }
        }
        artifact_query {
          type {
            name: "Examples"
          }
        }
        output_key: "examples"
      }
    }
  }
  inputs {
    key: "model"
    value {
      channels {
        producer_node_query {
          id: "Trainer"
        }
        context_queries {
          type {
            name: "pipeline"
          }
          name {
            field_value {
              string_value: "penguin-tfma"
            }
          }
        }
        context_queries {
          type {
            name: "pipeline_run"
          }
          name {
            field_value {
              string_value: "2021-04-30T09:18:51.523923"
            }
          }
        }
        context_queries {
          type {
            name: "node"
          }
          name {
            field_value {
              string_value: "penguin-tfma.Trainer"
            }
          }
        }
        artifact_query {
          type {
            name: "Model"
          }
        }
        output_key: "model"
      }
    }
  }
}
outputs {
  outputs {
    key: "blessing"
    value {
      artifact_spec {
        type {
          name: "ModelBlessing"
        }
      }
    }
  }
  outputs {
    key: "evaluation"
    value {
      artifact_spec {
        type {
          name: "ModelEvaluation"
        }
      }
    }
  }
}
parameters {
  parameters {
    key: "eval_config"
    value {
      field_value {
        string_value: "{\n  \"metrics_specs\": [\n    {\n      \"per_slice_thresholds\": {\n        \"sparse_categorical_accuracy\": {\n          \"thresholds\": [\n            {\n              \"slicing_specs\": [\n                {}\n              ],\n              \"threshold\": {\n                \"change_threshold\": {\n                  \"absolute\": -1e-10,\n                  \"direction\": \"HIGHER_IS_BETTER\"\n                },\n                \"value_threshold\": {\n                  \"lower_bound\": 0.6\n                }\n              }\n            }\n          ]\n        }\n      }\n    }\n  ],\n  \"model_specs\": [\n    {\n      \"label_key\": \"species\"\n    }\n  ],\n  \"slicing_specs\": [\n    {},\n    {\n      \"feature_keys\": [\n        \"species\"\n      ]\n    }\n  ]\n}"
      }
    }
  }
  parameters {
    key: "example_splits"
    value {
      field_value {
        string_value: "null"
      }
    }
  }
}
upstream_nodes: "CsvExampleGen"
upstream_nodes: "Resolver.latest_blessed_model_resolver"
upstream_nodes: "Trainer"
downstream_nodes: "Pusher"
execution_options {
  caching_options {
  }
}

INFO:absl:MetadataStore with DB connection initialized
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:Going to run a new execution 4
INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=4, input_dict={'examples': [Artifact(artifact: id: 1
type_id: 6
uri: "pipelines/penguin-tfma/CsvExampleGen/examples/1"
properties {
  key: "split_names"
  value {
    string_value: "[\"train\", \"eval\"]"
  }
}
custom_properties {
  key: "input_fingerprint"
  value {
    string_value: "split:single_split,num_files:1,total_bytes:25648,xor_checksum:1619774329,sum_checksum:1619774329"
  }
}
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:CsvExampleGen:examples:0"
  }
}
custom_properties {
  key: "payload_format"
  value {
    string_value: "FORMAT_TF_EXAMPLE"
  }
}
custom_properties {
  key: "span"
  value {
    string_value: "0"
  }
}
state: LIVE
create_time_since_epoch: 1619774333893
last_update_time_since_epoch: 1619774333893
, artifact_type: id: 6
name: "Examples"
properties {
  key: "span"
  value: INT
}
properties {
  key: "split_names"
  value: STRING
}
properties {
  key: "version"
  value: INT
}
)], 'model': [Artifact(artifact: id: 3
type_id: 10
uri: "pipelines/penguin-tfma/Trainer/model/3"
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:Trainer:model:0"
  }
}
state: LIVE
create_time_since_epoch: 1619774338864
last_update_time_since_epoch: 1619774338864
, artifact_type: id: 10
name: "Model"
)], 'baseline_model': []}, output_dict=defaultdict(<class 'list'>, {'evaluation': [Artifact(artifact: uri: "pipelines/penguin-tfma/Evaluator/evaluation/4"
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:Evaluator:evaluation:0"
  }
}
, artifact_type: name: "ModelEvaluation"
)], 'blessing': [Artifact(artifact: uri: "pipelines/penguin-tfma/Evaluator/blessing/4"
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:Evaluator:blessing:0"
  }
}
, artifact_type: name: "ModelBlessing"
)]}), exec_properties={'eval_config': '{\n  "metrics_specs": [\n    {\n      "per_slice_thresholds": {\n        "sparse_categorical_accuracy": {\n          "thresholds": [\n            {\n              "slicing_specs": [\n                {}\n              ],\n              "threshold": {\n                "change_threshold": {\n                  "absolute": -1e-10,\n                  "direction": "HIGHER_IS_BETTER"\n                },\n                "value_threshold": {\n                  "lower_bound": 0.6\n                }\n              }\n            }\n          ]\n        }\n      }\n    }\n  ],\n  "model_specs": [\n    {\n      "label_key": "species"\n    }\n  ],\n  "slicing_specs": [\n    {},\n    {\n      "feature_keys": [\n        "species"\n      ]\n    }\n  ]\n}', 'example_splits': 'null'}, execution_output_uri='pipelines/penguin-tfma/Evaluator/.system/executor_execution/4/executor_output.pb', stateful_working_dir='pipelines/penguin-tfma/Evaluator/.system/stateful_working_dir/2021-04-30T09:18:51.523923', tmp_dir='pipelines/penguin-tfma/Evaluator/.system/executor_execution/4/.temp/', pipeline_node=node_info {
  type {
    name: "tfx.components.evaluator.component.Evaluator"
  }
  id: "Evaluator"
}
contexts {
  contexts {
    type {
      name: "pipeline"
    }
    name {
      field_value {
        string_value: "penguin-tfma"
      }
    }
  }
  contexts {
    type {
      name: "pipeline_run"
    }
    name {
      field_value {
        string_value: "2021-04-30T09:18:51.523923"
      }
    }
  }
  contexts {
    type {
      name: "node"
    }
    name {
      field_value {
        string_value: "penguin-tfma.Evaluator"
      }
    }
  }
}
inputs {
  inputs {
    key: "baseline_model"
    value {
      channels {
        producer_node_query {
          id: "Resolver.latest_blessed_model_resolver"
        }
        context_queries {
          type {
            name: "pipeline"
          }
          name {
            field_value {
              string_value: "penguin-tfma"
            }
          }
        }
        context_queries {
          type {
            name: "pipeline_run"
          }
          name {
            field_value {
              string_value: "2021-04-30T09:18:51.523923"
            }
          }
        }
        context_queries {
          type {
            name: "node"
          }
          name {
            field_value {
              string_value: "penguin-tfma.Resolver.latest_blessed_model_resolver"
            }
          }
        }
        artifact_query {
          type {
            name: "Model"
          }
        }
        output_key: "model"
      }
    }
  }
  inputs {
    key: "examples"
    value {
      channels {
        producer_node_query {
          id: "CsvExampleGen"
        }
        context_queries {
          type {
            name: "pipeline"
          }
          name {
            field_value {
              string_value: "penguin-tfma"
            }
          }
        }
        context_queries {
          type {
            name: "pipeline_run"
          }
          name {
            field_value {
              string_value: "2021-04-30T09:18:51.523923"
            }
          }
        }
        context_queries {
          type {
            name: "node"
          }
          name {
            field_value {
              string_value: "penguin-tfma.CsvExampleGen"
            }
          }
        }
        artifact_query {
          type {
            name: "Examples"
          }
        }
        output_key: "examples"
      }
    }
  }
  inputs {
    key: "model"
    value {
      channels {
        producer_node_query {
          id: "Trainer"
        }
        context_queries {
          type {
            name: "pipeline"
          }
          name {
            field_value {
              string_value: "penguin-tfma"
            }
          }
        }
        context_queries {
          type {
            name: "pipeline_run"
          }
          name {
            field_value {
              string_value: "2021-04-30T09:18:51.523923"
            }
          }
        }
        context_queries {
          type {
            name: "node"
          }
          name {
            field_value {
              string_value: "penguin-tfma.Trainer"
            }
          }
        }
        artifact_query {
          type {
            name: "Model"
          }
        }
        output_key: "model"
      }
    }
  }
}
outputs {
  outputs {
    key: "blessing"
    value {
      artifact_spec {
        type {
          name: "ModelBlessing"
        }
      }
    }
  }
  outputs {
    key: "evaluation"
    value {
      artifact_spec {
        type {
          name: "ModelEvaluation"
        }
      }
    }
  }
}
parameters {
  parameters {
    key: "eval_config"
    value {
      field_value {
        string_value: "{\n  \"metrics_specs\": [\n    {\n      \"per_slice_thresholds\": {\n        \"sparse_categorical_accuracy\": {\n          \"thresholds\": [\n            {\n              \"slicing_specs\": [\n                {}\n              ],\n              \"threshold\": {\n                \"change_threshold\": {\n                  \"absolute\": -1e-10,\n                  \"direction\": \"HIGHER_IS_BETTER\"\n                },\n                \"value_threshold\": {\n                  \"lower_bound\": 0.6\n                }\n              }\n            }\n          ]\n        }\n      }\n    }\n  ],\n  \"model_specs\": [\n    {\n      \"label_key\": \"species\"\n    }\n  ],\n  \"slicing_specs\": [\n    {},\n    {\n      \"feature_keys\": [\n        \"species\"\n      ]\n    }\n  ]\n}"
      }
    }
  }
  parameters {
    key: "example_splits"
    value {
      field_value {
        string_value: "null"
      }
    }
  }
}
upstream_nodes: "CsvExampleGen"
upstream_nodes: "Resolver.latest_blessed_model_resolver"
upstream_nodes: "Trainer"
downstream_nodes: "Pusher"
execution_options {
  caching_options {
  }
}
, pipeline_info=id: "penguin-tfma"
, pipeline_run_id='2021-04-30T09:18:51.523923')
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['-f', '/tmp/tmp2mq186ss.json', '--HistoryManager.hist_file=:memory:']
INFO:absl:Attempting to infer TFX Python dependency for beam
INFO:absl:Copying all content from install dir /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tfx to temp dir /tmp/tmp3fqu50ui/build/tfx
INFO:absl:Generating a temp setup file at /tmp/tmp3fqu50ui/build/tfx/setup.py
INFO:absl:Creating temporary sdist package, logs available at /tmp/tmp3fqu50ui/build/tfx/setup.log
INFO:absl:Added --extra_package=/tmp/tmp3fqu50ui/build/tfx/dist/tfx_ephemeral-0.28.0.tar.gz to beam args
ERROR:absl:There are change thresholds, but the baseline is missing. This is allowed only when rubber stamping (first run).
INFO:absl:Request was made to ignore the baseline ModelSpec and any change thresholds. This is likely because a baseline model was not provided: updated_config=
model_specs {
  label_key: "species"
}
slicing_specs {
}
slicing_specs {
  feature_keys: "species"
}
metrics_specs {
  per_slice_thresholds {
    key: "sparse_categorical_accuracy"
    value {
      thresholds {
        slicing_specs {
        }
        threshold {
          value_threshold {
            lower_bound {
              value: 0.6
            }
          }
        }
      }
    }
  }
}

INFO:absl:Using pipelines/penguin-tfma/Trainer/model/3/serving_model_dir as  model.
INFO:absl:The 'example_splits' parameter is not set, using 'eval' split.
INFO:absl:Evaluating model.
INFO:absl:Request was made to ignore the baseline ModelSpec and any change thresholds. This is likely because a baseline model was not provided: updated_config=
model_specs {
  label_key: "species"
}
slicing_specs {
}
slicing_specs {
  feature_keys: "species"
}
metrics_specs {
  model_names: ""
  per_slice_thresholds {
    key: "sparse_categorical_accuracy"
    value {
      thresholds {
        slicing_specs {
        }
        threshold {
          value_threshold {
            lower_bound {
              value: 0.6
            }
          }
        }
      }
    }
  }
}

INFO:absl:Request was made to ignore the baseline ModelSpec and any change thresholds. This is likely because a baseline model was not provided: updated_config=
model_specs {
  label_key: "species"
}
slicing_specs {
}
slicing_specs {
  feature_keys: "species"
}
metrics_specs {
  model_names: ""
  per_slice_thresholds {
    key: "sparse_categorical_accuracy"
    value {
      thresholds {
        slicing_specs {
        }
        threshold {
          value_threshold {
            lower_bound {
              value: 0.6
            }
          }
        }
      }
    }
  }
}

INFO:absl:Request was made to ignore the baseline ModelSpec and any change thresholds. This is likely because a baseline model was not provided: updated_config=
model_specs {
  label_key: "species"
}
slicing_specs {
}
slicing_specs {
  feature_keys: "species"
}
metrics_specs {
  model_names: ""
  per_slice_thresholds {
    key: "sparse_categorical_accuracy"
    value {
      thresholds {
        slicing_specs {
        }
        threshold {
          value_threshold {
            lower_bound {
              value: 0.6
            }
          }
        }
      }
    }
  }
}

WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['-f', '/tmp/tmp2mq186ss.json', '--HistoryManager.hist_file=:memory:']
INFO:absl:Evaluation complete. Results written to pipelines/penguin-tfma/Evaluator/evaluation/4.
INFO:absl:Checking validation results.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:113: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:113: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
INFO:absl:Blessing result True written to pipelines/penguin-tfma/Evaluator/blessing/4.
INFO:absl:Cleaning up stateless execution info.
INFO:absl:Execution 4 succeeded.
INFO:absl:Cleaning up stateful execution info.
INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'evaluation': [Artifact(artifact: uri: "pipelines/penguin-tfma/Evaluator/evaluation/4"
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:Evaluator:evaluation:0"
  }
}
, artifact_type: name: "ModelEvaluation"
)], 'blessing': [Artifact(artifact: uri: "pipelines/penguin-tfma/Evaluator/blessing/4"
custom_properties {
  key: "blessed"
  value {
    int_value: 1
  }
}
custom_properties {
  key: "current_model"
  value {
    string_value: "pipelines/penguin-tfma/Trainer/model/3"
  }
}
custom_properties {
  key: "current_model_id"
  value {
    int_value: 3
  }
}
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:Evaluator:blessing:0"
  }
}
, artifact_type: name: "ModelBlessing"
)]}) for exeuction 4
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:Component Evaluator is finished.
INFO:absl:Component Pusher is running.
INFO:absl:Running launcher for node_info {
  type {
    name: "tfx.components.pusher.component.Pusher"
  }
  id: "Pusher"
}
contexts {
  contexts {
    type {
      name: "pipeline"
    }
    name {
      field_value {
        string_value: "penguin-tfma"
      }
    }
  }
  contexts {
    type {
      name: "pipeline_run"
    }
    name {
      field_value {
        string_value: "2021-04-30T09:18:51.523923"
      }
    }
  }
  contexts {
    type {
      name: "node"
    }
    name {
      field_value {
        string_value: "penguin-tfma.Pusher"
      }
    }
  }
}
inputs {
  inputs {
    key: "model"
    value {
      channels {
        producer_node_query {
          id: "Trainer"
        }
        context_queries {
          type {
            name: "pipeline"
          }
          name {
            field_value {
              string_value: "penguin-tfma"
            }
          }
        }
        context_queries {
          type {
            name: "pipeline_run"
          }
          name {
            field_value {
              string_value: "2021-04-30T09:18:51.523923"
            }
          }
        }
        context_queries {
          type {
            name: "node"
          }
          name {
            field_value {
              string_value: "penguin-tfma.Trainer"
            }
          }
        }
        artifact_query {
          type {
            name: "Model"
          }
        }
        output_key: "model"
      }
    }
  }
  inputs {
    key: "model_blessing"
    value {
      channels {
        producer_node_query {
          id: "Evaluator"
        }
        context_queries {
          type {
            name: "pipeline"
          }
          name {
            field_value {
              string_value: "penguin-tfma"
            }
          }
        }
        context_queries {
          type {
            name: "pipeline_run"
          }
          name {
            field_value {
              string_value: "2021-04-30T09:18:51.523923"
            }
          }
        }
        context_queries {
          type {
            name: "node"
          }
          name {
            field_value {
              string_value: "penguin-tfma.Evaluator"
            }
          }
        }
        artifact_query {
          type {
            name: "ModelBlessing"
          }
        }
        output_key: "blessing"
      }
    }
  }
}
outputs {
  outputs {
    key: "pushed_model"
    value {
      artifact_spec {
        type {
          name: "PushedModel"
        }
      }
    }
  }
}
parameters {
  parameters {
    key: "custom_config"
    value {
      field_value {
        string_value: "null"
      }
    }
  }
  parameters {
    key: "push_destination"
    value {
      field_value {
        string_value: "{\n  \"filesystem\": {\n    \"base_directory\": \"serving_model/penguin-tfma\"\n  }\n}"
      }
    }
  }
}
upstream_nodes: "Evaluator"
upstream_nodes: "Trainer"
execution_options {
  caching_options {
  }
}

INFO:absl:MetadataStore with DB connection initialized
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:Going to run a new execution 5
INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=5, input_dict={'model': [Artifact(artifact: id: 3
type_id: 10
uri: "pipelines/penguin-tfma/Trainer/model/3"
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:Trainer:model:0"
  }
}
state: LIVE
create_time_since_epoch: 1619774338864
last_update_time_since_epoch: 1619774338864
, artifact_type: id: 10
name: "Model"
)], 'model_blessing': [Artifact(artifact: id: 5
type_id: 13
uri: "pipelines/penguin-tfma/Evaluator/blessing/4"
custom_properties {
  key: "blessed"
  value {
    int_value: 1
  }
}
custom_properties {
  key: "current_model"
  value {
    string_value: "pipelines/penguin-tfma/Trainer/model/3"
  }
}
custom_properties {
  key: "current_model_id"
  value {
    int_value: 3
  }
}
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:Evaluator:blessing:0"
  }
}
state: LIVE
create_time_since_epoch: 1619774343802
last_update_time_since_epoch: 1619774343802
, artifact_type: id: 13
name: "ModelBlessing"
)]}, output_dict=defaultdict(<class 'list'>, {'pushed_model': [Artifact(artifact: uri: "pipelines/penguin-tfma/Pusher/pushed_model/5"
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:Pusher:pushed_model:0"
  }
}
, artifact_type: name: "PushedModel"
)]}), exec_properties={'custom_config': 'null', 'push_destination': '{\n  "filesystem": {\n    "base_directory": "serving_model/penguin-tfma"\n  }\n}'}, execution_output_uri='pipelines/penguin-tfma/Pusher/.system/executor_execution/5/executor_output.pb', stateful_working_dir='pipelines/penguin-tfma/Pusher/.system/stateful_working_dir/2021-04-30T09:18:51.523923', tmp_dir='pipelines/penguin-tfma/Pusher/.system/executor_execution/5/.temp/', pipeline_node=node_info {
  type {
    name: "tfx.components.pusher.component.Pusher"
  }
  id: "Pusher"
}
contexts {
  contexts {
    type {
      name: "pipeline"
    }
    name {
      field_value {
        string_value: "penguin-tfma"
      }
    }
  }
  contexts {
    type {
      name: "pipeline_run"
    }
    name {
      field_value {
        string_value: "2021-04-30T09:18:51.523923"
      }
    }
  }
  contexts {
    type {
      name: "node"
    }
    name {
      field_value {
        string_value: "penguin-tfma.Pusher"
      }
    }
  }
}
inputs {
  inputs {
    key: "model"
    value {
      channels {
        producer_node_query {
          id: "Trainer"
        }
        context_queries {
          type {
            name: "pipeline"
          }
          name {
            field_value {
              string_value: "penguin-tfma"
            }
          }
        }
        context_queries {
          type {
            name: "pipeline_run"
          }
          name {
            field_value {
              string_value: "2021-04-30T09:18:51.523923"
            }
          }
        }
        context_queries {
          type {
            name: "node"
          }
          name {
            field_value {
              string_value: "penguin-tfma.Trainer"
            }
          }
        }
        artifact_query {
          type {
            name: "Model"
          }
        }
        output_key: "model"
      }
    }
  }
  inputs {
    key: "model_blessing"
    value {
      channels {
        producer_node_query {
          id: "Evaluator"
        }
        context_queries {
          type {
            name: "pipeline"
          }
          name {
            field_value {
              string_value: "penguin-tfma"
            }
          }
        }
        context_queries {
          type {
            name: "pipeline_run"
          }
          name {
            field_value {
              string_value: "2021-04-30T09:18:51.523923"
            }
          }
        }
        context_queries {
          type {
            name: "node"
          }
          name {
            field_value {
              string_value: "penguin-tfma.Evaluator"
            }
          }
        }
        artifact_query {
          type {
            name: "ModelBlessing"
          }
        }
        output_key: "blessing"
      }
    }
  }
}
outputs {
  outputs {
    key: "pushed_model"
    value {
      artifact_spec {
        type {
          name: "PushedModel"
        }
      }
    }
  }
}
parameters {
  parameters {
    key: "custom_config"
    value {
      field_value {
        string_value: "null"
      }
    }
  }
  parameters {
    key: "push_destination"
    value {
      field_value {
        string_value: "{\n  \"filesystem\": {\n    \"base_directory\": \"serving_model/penguin-tfma\"\n  }\n}"
      }
    }
  }
}
upstream_nodes: "Evaluator"
upstream_nodes: "Trainer"
execution_options {
  caching_options {
  }
}
, pipeline_info=id: "penguin-tfma"
, pipeline_run_id='2021-04-30T09:18:51.523923')
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['-f', '/tmp/tmp2mq186ss.json', '--HistoryManager.hist_file=:memory:']
INFO:absl:Attempting to infer TFX Python dependency for beam
INFO:absl:Copying all content from install dir /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tfx to temp dir /tmp/tmpfdskrws1/build/tfx
INFO:absl:Generating a temp setup file at /tmp/tmpfdskrws1/build/tfx/setup.py
INFO:absl:Creating temporary sdist package, logs available at /tmp/tmpfdskrws1/build/tfx/setup.log
INFO:absl:Added --extra_package=/tmp/tmpfdskrws1/build/tfx/dist/tfx_ephemeral-0.28.0.tar.gz to beam args
INFO:absl:Model version: 1619774344
INFO:absl:Model written to serving path serving_model/penguin-tfma/1619774344.
INFO:absl:Model pushed to pipelines/penguin-tfma/Pusher/pushed_model/5.
INFO:absl:Cleaning up stateless execution info.
INFO:absl:Execution 5 succeeded.
INFO:absl:Cleaning up stateful execution info.
INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'pushed_model': [Artifact(artifact: uri: "pipelines/penguin-tfma/Pusher/pushed_model/5"
custom_properties {
  key: "name"
  value {
    string_value: "penguin-tfma:2021-04-30T09:18:51.523923:Pusher:pushed_model:0"
  }
}
custom_properties {
  key: "pushed"
  value {
    int_value: 1
  }
}
custom_properties {
  key: "pushed_destination"
  value {
    string_value: "serving_model/penguin-tfma/1619774344"
  }
}
custom_properties {
  key: "pushed_version"
  value {
    string_value: "1619774344"
  }
}
, artifact_type: name: "PushedModel"
)]}) for exeuction 5
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:Component Pusher is finished.

Quando la pipeline è stata completata, dovresti essere in grado di vedere qualcosa di simile a quanto segue:

INFO:absl:Blessing result True written to pipelines/penguin-tfma/Evaluator/blessing/4.

Oppure puoi anche controllare manualmente la directory di output in cui sono archiviati gli artefatti generati. Se si visita pipelines/penguin-tfma/Evaluator/blessing/ con un file broswer, si può vedere un file con un nome BLESSED o NOT_BLESSED a seconda del risultato della valutazione.

Se il risultato della benedizione è False , Pusher rifiuterà di serving_model_dir il modello a serving_model_dir , perché il modello non è abbastanza buono per essere utilizzato in produzione.

È possibile eseguire nuovamente la pipeline possibilmente con diverse configurazioni di valutazione. Anche se esegui la pipeline con la stessa identica configurazione e set di dati, il modello addestrato potrebbe essere leggermente diverso a causa della casualità intrinseca dell'addestramento del modello che può portare a un modello NOT_BLESSED .

Esaminare gli output della pipeline

È possibile utilizzare TFMA per indagare e visualizzare il risultato della valutazione nell'artefatto ModelEvaluation.

Ottieni il risultato dell'analisi dagli artefatti di output

È possibile utilizzare le API MLMD per individuare questi output in modo programmatico. Per prima cosa, definiremo alcune funzioni di utilità per cercare gli artefatti di output che sono stati appena prodotti.

from ml_metadata.proto import metadata_store_pb2
from tfx.orchestration.portable.mlmd import execution_lib
from tfx.types import standard_component_specs

# TODO(b/171447278): Move these functions into the TFX library.

def get_latest_execution(metadata, pipeline_name, component_id):
  """Gets the execution objects for the latest run of the pipeline."""
  node_context = metadata.store.get_context_by_type_and_name(
      'node', f'{pipeline_name}.{component_id}')
  executions = metadata.store.get_executions_by_context(node_context.id)
  # Pick the latest one.
  return max(executions, key=lambda e: e.last_update_time_since_epoch)

def get_artifacts_for_component_id(metadata, execution):
  return execution_lib.get_artifacts_dict(metadata, execution.id,
                                          metadata_store_pb2.Event.OUTPUT)

Possiamo trovare l'ultima esecuzione del componente Evaluator e ottenerne gli artefatti di output.

metadata_connection_config = metadata.sqlite_metadata_connection_config(
    METADATA_PATH)

with metadata.Metadata(metadata_connection_config) as metadata_handler:
  execution = get_latest_execution(metadata_handler, PIPELINE_NAME, 'Evaluator')
  evaluator_output = get_artifacts_for_component_id(metadata_handler, execution)
INFO:absl:MetadataStore with DB connection initialized

Evaluator restituisce sempre un artefatto di valutazione e possiamo visualizzarlo utilizzando la libreria TensorFlow Model Analysis. Ad esempio, il codice seguente renderà le metriche di accuratezza per ogni specie di pinguino.

import tensorflow_model_analysis as tfma
eval_artifact = evaluator_output[standard_component_specs.EVALUATION_KEY][0]
eval_result = tfma.load_eval_result(eval_artifact.uri)
tfma.view.render_slicing_metrics(eval_result, slicing_column='species')
SlicingMetricsViewer(config={'weightedExamplesColumn': 'example_count'}, data=[{'slice': 'species:0', 'metrics…

Se scegli "sparse_categorical_accuracy" nell'elenco a discesa Show , puoi vedere i valori di precisione per specie. Potresti voler aggiungere più slice e controllare se il tuo modello è adatto a tutte le distribuzioni e se c'è qualche possibile bias.

Prossimi passi

Scopri di più sull'analisi del modello nel tutorial della libreria TensorFlow Model Analysis .

Puoi trovare ulteriori risorse su https://www.tensorflow.org/tfx/tutorials

Vedere Comprensione delle pipeline TFX per saperne di più sui vari concetti in TFX.