tfx.orchestration.experimental.core.task_gen.TaskGenerator

TaskGenerator interface.

When their generate method is invoked (typically done periodically within an orchestration loop), concrete classes implementing this interface are expected to generate tasks to execute nodes in a pipeline IR spec or system tasks (eg: for garbage collection) based on the state of pipeline execution and related details stored in an MLMD db.

Note on thread safety: Concrete classes of this interface need not have a thread-safe implementation. Onus is on the caller to serialize concurrent calls to generate. Since MLMD db may be updated upon call to generate, it's also not safe to invoke generate concurrently on different instances of TaskGenerator that refer to the same MLMD db and the same pipeline IR.

Methods

generate

View source

Generates a list of tasks to be performed.

Returns
A list of Tasks specifying nodes in a pipeline to be executed or other system tasks.