tfx.v1.orchestration.experimental.KubeflowDagRunner

Kubeflow Pipelines runner.

tfx.v1.orchestration.experimental.KubeflowDagRunner(
    output_dir: Optional[str] = None,
    output_filename: Optional[str] = None,
    config: Optional[tfx.v1.orchestration.experimental.KubeflowDagRunnerConfig] = None,
    pod_labels_to_attach: Optional[Dict[str, str]] = None
)

Constructs a pipeline definition YAML file based on the TFX logical pipeline.

output_dir An optional output directory into which to output the pipeline definition files. Defaults to the current working directory.
output_filename An optional output file name for the pipeline definition file. Defaults to pipeline_name.tar.gz when compiling a TFX pipeline. Currently supports .tar.gz, .tgz, .zip, .yaml, .yml formats. See https://github.com/kubeflow/pipelines/blob/181de66cf9fa87bcd0fe9291926790c400140783/sdk/python/kfp/compiler/compiler.py#L851 for format restriction.
config An optional KubeflowDagRunnerConfig object to specify runtime configuration when running the pipeline under Kubeflow.
pod_labels_to_attach Optional set of pod labels to attach to GKE pod spinned up for this pipeline. Default to the 3 labels:

  1. add-pod-env: true,
  2. pipeline SDK type,
  3. pipeline unique ID, where 2 and 3 are instrumentation of usage tracking.

Methods

run

run(
    pipeline: tfx.v1.dsl.Pipeline
)

Compiles and outputs a Kubeflow Pipeline YAML definition file.

Args
pipeline The logical TFX pipeline to use when building the Kubeflow pipeline.

set_exit_handler

set_exit_handler(
    exit_handler: tfx.v1.types.BaseNode
)

Set exit handler components for the Kubeflow dag runner.

This feature is currently experimental without backward compatibility gaurantee.

Args
exit_handler exit handler component.