tfx.orchestration.beam.beam_dag_runner.BeamDagRunner

View source on GitHub

Tfx runner on Beam.

Inherits From: TfxRunner

tfx.orchestration.beam.beam_dag_runner.BeamDagRunner(
    beam_orchestrator_args: Optional[List[Text]] = None,
    config: Optional[tfx.orchestration.config.pipeline_config.PipelineConfig] = None
)

beam_orchestrator_args beam args for the beam orchestrator. Note that this is different from the beam_pipeline_args within additional_pipeline_args, which is for beam pipelines in components.
config Optional pipeline config for customizing the launching of each component. Defaults to pipeline config that supports InProcessComponentLauncher and DockerComponentLauncher.

Methods

run

View source

run(
    tfx_pipeline: tfx.orchestration.pipeline.Pipeline
) -> None

Deploys given logical pipeline on Beam.

Args
tfx_pipeline Logical pipeline containing pipeline args and components.