tfx.orchestration.experimental.core.sync_pipeline_task_gen.SyncPipelineTaskGenerator

Task generator for executing a sync pipeline.

Inherits From: TaskGenerator

Calling generate is not thread-safe. Concurrent calls to generate should be explicitly serialized. Since MLMD may be updated upon call to generate, it's also not safe to call generate on different instances of this class where the instances refer to the same MLMD db and the same pipeline IR.

mlmd_connection ML metadata db connection.
pipeline A pipeline IR proto.
is_task_id_tracked_fn A callable that returns True if a task_id is tracked by the task queue.

Methods

generate

View source

Generates tasks for executing the next executable nodes in the pipeline.

The returned tasks must have exec_task populated. List may be empty if no nodes are ready for execution.

Returns
A list of tasks to execute.