tfx.orchestration.experimental.core.task_queue.TaskQueue

A thread-safe task queue with duplicate detection.

The life-cycle of a task starts with producers calling enqueue. Consumers call dequeue to obtain the tasks in FIFO order. When processing is complete, consumers must release the tasks by calling task_done.

Methods

contains_task_id

View source

Returns True if the task queue contains a task with the given task_id.

Args
task_id A task id.

Returns
True if a task with task_id was enqueued but task_done has not been invoked yet.

dequeue

View source

Removes and returns a task from the queue.

Once the processing is complete, queue consumers must call task_done.

Args
max_wait_secs If not None, waits a maximum of max_wait_secs when the queue is empty for a task to be enqueued. If no task is present in the queue after the wait, None is returned. If max_wait_secs is None (default), returns None without waiting when the queue is empty.

Returns
A Task or None if the queue is empty.

enqueue

View source

Enqueues the given task if no prior task with the same id exists.

Args
task A Task object.

Returns
True if the task could be enqueued. False if a task with the same id already exists.

is_empty

View source

Returns True if the task queue is empty.

Queue is considered empty only if any enqueued tasks have been dequeued and task_done invoked on them.

task_done

View source

Marks the processing of a task as done.

Consumers should call this method after the task is processed.

Args
task A Task object.

Raises
RuntimeError If attempt is made to mark a non-existent or non-dequeued task as done.