本頁面由 Cloud Translation API 翻譯而成。
Switch to English

培訓師TFX管道組件

Trainer TFX管道組件訓練TensorFlow模型。

培訓師和TensorFlow

Trainer廣泛使用Python TensorFlow API來訓練模型。

零件

培訓師需要:

  • tf。用於培訓和評估的示例。
  • 用戶提供的模塊文件,用於定義訓練器邏輯。
  • 由SchemaGen管道組件創建並由開發人員可選地更改的數據模式。
  • 程序 arg和eval arg的Protobuf定義。
  • (可選)由上游Transform組件生成的變換圖。
  • (可選)用於熱啟動等場景的預訓練模型。
  • (可選)超參數,這些參數將傳遞給用戶模塊功能。與Tuner集成的詳細信息可以在這裡找到。

Trainer發出:至少一個用於推理/服務的模型(通常在SavedModelFormat中),並可選地另一個用於eval的模型(通常是EvalSavedModel)。

我們通過模型重寫庫提供對其他模型格式的支持,例如TFLite 。有關如何轉換Estimator模型和Keras模型的示例,請參見模型重寫庫的鏈接。

基於估計的培訓師

要了解有關將基於Estimator的模型與TFX和Trainer結合使用的信息,請參閱使用tf.Estimator for TFX設計TensorFlow建模代碼

配置培訓師組件

典型的管道Python DSL代碼如下所示:

from tfx.components import Trainer

...

trainer = Trainer(
      module_file=module_file,
      examples=transform.outputs['transformed_examples'],
      schema=infer_schema.outputs['schema'],
      base_models=latest_model_resolver.outputs['latest_model'],
      transform_graph=transform.outputs['transform_graph'],
      train_args=trainer_pb2.TrainArgs(num_steps=10000),
      eval_args=trainer_pb2.EvalArgs(num_steps=5000))

培訓師調用一個培訓模塊,該模塊在module_file參數中指定。一個典型的培訓模塊如下所示:

# TFX will call this function
def trainer_fn(trainer_fn_args, schema):
  """Build the estimator using the high level API.

  Args:
    trainer_fn_args: Holds args used to train the model as name/value pairs.
    schema: Holds the schema of the training examples.

  Returns:
    A dict of the following:

      - estimator: The estimator that will be used for training and eval.
      - train_spec: Spec for training.
      - eval_spec: Spec for eval.
      - eval_input_receiver_fn: Input function for eval.
  """
  # Number of nodes in the first layer of the DNN
  first_dnn_layer_size = 100
  num_dnn_layers = 4
  dnn_decay_factor = 0.7

  train_batch_size = 40
  eval_batch_size = 40

  tf_transform_output = tft.TFTransformOutput(trainer_fn_args.transform_output)

  train_input_fn = lambda: _input_fn(  # pylint: disable=g-long-lambda
      trainer_fn_args.train_files,
      tf_transform_output,
      batch_size=train_batch_size)

  eval_input_fn = lambda: _input_fn(  # pylint: disable=g-long-lambda
      trainer_fn_args.eval_files,
      tf_transform_output,
      batch_size=eval_batch_size)

  train_spec = tf.estimator.TrainSpec(  # pylint: disable=g-long-lambda
      train_input_fn,
      max_steps=trainer_fn_args.train_steps)

  serving_receiver_fn = lambda: _example_serving_receiver_fn(  # pylint: disable=g-long-lambda
      tf_transform_output, schema)

  exporter = tf.estimator.FinalExporter('chicago-taxi', serving_receiver_fn)
  eval_spec = tf.estimator.EvalSpec(
      eval_input_fn,
      steps=trainer_fn_args.eval_steps,
      exporters=[exporter],
      name='chicago-taxi-eval')

  run_config = tf.estimator.RunConfig(
      save_checkpoints_steps=999, keep_checkpoint_max=1)

  run_config = run_config.replace(model_dir=trainer_fn_args.serving_model_dir)
  warm_start_from = trainer_fn_args.base_models[
      0] if trainer_fn_args.base_models else None

  estimator = _build_estimator(
      # Construct layers sizes with exponetial decay
      hidden_units=[
          max(2, int(first_dnn_layer_size * dnn_decay_factor**i))
          for i in range(num_dnn_layers)
      ],
      config=run_config,
      warm_start_from=warm_start_from)

  # Create an input receiver for TFMA processing
  receiver_fn = lambda: _eval_input_receiver_fn(  # pylint: disable=g-long-lambda
      tf_transform_output, schema)

  return {
      'estimator': estimator,
      'train_spec': train_spec,
      'eval_spec': eval_spec,
      'eval_input_receiver_fn': receiver_fn
  }

通用教練

通用訓練器使開發人員可以將任何TensorFlow模型API與訓練器組件一起使用。除了TensorFlow Estimators,開發人員還可以使用Keras模型或自定義訓練循環。有關詳細信息,請參閱通用培訓師RFC

配置培訓師組件以使用GenericExecutor

通用Trainer的典型管道DSL代碼如下所示:

from tfx.components import Trainer
from tfx.components.base import executor_spec
from tfx.components.trainer.executor import GenericExecutor

...

trainer = Trainer(
    module_file=module_file,
    custom_executor_spec=executor_spec.ExecutorClassSpec(GenericExecutor),
    examples=transform.outputs['transformed_examples'],
    transform_graph=transform.outputs['transform_graph'],
    schema=infer_schema.outputs['schema'],
    train_args=trainer_pb2.TrainArgs(num_steps=10000),
    eval_args=trainer_pb2.EvalArgs(num_steps=5000))

培訓師調用一個培訓模塊,該模塊在module_file參數中指定。取而代之的trainer_fn ,一個run_fn在模塊文件要求,如果GenericExecutor在指定custom_executor_spec

如果管道中未使用Transform組件,那麼Trainer將直接從ExampleGen中獲取示例:

trainer = Trainer(
    module_file=module_file,
    custom_executor_spec=executor_spec.ExecutorClassSpec(GenericExecutor),
    examples=example_gen.outputs['examples'],
    schema=infer_schema.outputs['schema'],
    train_args=trainer_pb2.TrainArgs(num_steps=10000),
    eval_args=trainer_pb2.EvalArgs(num_steps=5000))

這是帶有run_fn示例模塊文件