TF 2.0 is out! Get hands-on practice at TF World, Oct 28-31. Use code TF20 for 20% off select passes. Register now

tfx.orchestration.pipeline.Pipeline

View source on GitHub

Class Pipeline

Logical TFX pipeline object.

Attributes:

  • pipeline_args: kwargs used to create real pipeline implementation. This is forwarded to PipelineRunners instead of consumed in this class. This should include:
    • pipeline_name: Required. The unique name of this pipeline.
    • pipeline_root: Required. The root of the pipeline outputs.
  • components: logical components of this pipeline.
  • pipeline_info: An instance of data_types.PipelineInfo that contains basic properties of the pipeline.
  • enable_cache: whether or not cache is enabled for this run.
  • metadata_connection_config: the config to connect to ML metadata.
  • additional_pipeline_args: other pipeline args. e.g. beam runner args.

__init__

View source

__init__(
    pipeline_name,
    pipeline_root,
    metadata_connection_config=None,
    components=None,
    enable_cache=False,
    metadata_db_root=None,
    **kwargs
)

Initialize pipeline.

Args:

  • pipeline_name: name of the pipeline;
  • pipeline_root: path to root directory of the pipeline;
  • metadata_connection_config: the config to connect to ML metadata.
  • components: a list of components in the pipeline (optional only for backward compatible purpose to be used with deprecated PipelineDecorator).
  • enable_cache: whether or not cache is enabled for this run.
  • metadata_db_root: Deprecated. the uri to the metadata database root. Deprecated and will be removed in future version. Please use metadata_connection_config instead.
  • **kwargs: additional kwargs forwarded as pipeline args.
    • beam_pipeline_args: Beam pipeline args for beam jobs within executor. Executor will use beam DirectRunner as Default.

Properties

components

A list of logical components that are deduped and topological sorted.