Join the SIG TFX-Addons community and help make TFX even better! Join SIG TFX-Addons

tfx.v1.orchestration.experimental.KubeflowDagRunner

Kubeflow Pipelines runner.

Constructs a pipeline definition YAML file based on the TFX logical pipeline.

output_dir An optional output directory into which to output the pipeline definition files. Defaults to the current working directory.
output_filename An optional output file name for the pipeline definition file. Defaults to pipeline_name.tar.gz when compiling a TFX pipeline. Currently supports .tar.gz, .tgz, .zip, .yaml, .yml formats. See https://github.com/kubeflow/pipelines/blob/181de66cf9fa87bcd0fe9291926790c400140783/sdk/python/kfp/compiler/compiler.py#L851 for format restriction.
config An optional KubeflowDagRunnerConfig object to specify runtime configuration when running the pipeline under Kubeflow.
pod_labels_to_attach Optional set of pod labels to attach to GKE pod spinned up for this pipeline. Default to the 3 labels:

  1. add-pod-env: true,
  2. pipeline SDK type,
  3. pipeline unique ID, where 2 and 3 are instrumentation of usage tracking.

Methods

run

Compiles and outputs a Kubeflow Pipeline YAML definition file.

Args
pipeline The logical TFX pipeline to use when building the Kubeflow pipeline.