tfx.orchestration.pipeline.Pipeline

View source on GitHub

Class Pipeline

Logical TFX pipeline object.

Attributes:

• pipeline_args: kwargs used to create real pipeline implementation. This is forwarded to PipelineRunners instead of consumed in this class. This should include:
  • pipeline_name: Required. The unique name of this pipeline.
  • pipeline_root: Required. The root of the pipeline outputs.
• components: logical components of this pipeline.
• pipeline_info: An instance of data_types.PipelineInfo that contains basic properties of the pipeline.
• enable_cache: whether or not cache is enabled for this run.
• metadata_connection_config: the config to connect to ML metadata.
• additional_pipeline_args: other pipeline args. e.g. beam runner args.

__init__

View source

__init__(
    pipeline_name,
    pipeline_root,
    metadata_connection_config=None,
    components=None,
    enable_cache=False,
    metadata_db_root=None,
    **kwargs
)

Initialize pipeline.

Args:

• pipeline_name: name of the pipeline;
• pipeline_root: path to root directory of the pipeline;
• metadata_connection_config: the config to connect to ML metadata.
• components: a list of components in the pipeline (optional only for backward compatible purpose to be used with deprecated PipelineDecorator).
• enable_cache: whether or not cache is enabled for this run.
• metadata_db_root: Deprecated. the uri to the metadata database root. Deprecated and will be removed in future version. Please use metadata_connection_config instead.
• **kwargs: additional kwargs forwarded as pipeline args.
  • beam_pipeline_args: Beam pipeline args for beam jobs within executor. Executor will use beam DirectRunner as Default.

Properties

components

A list of logical components that are deduped and topological sorted.