RSVP for your your local TensorFlow Everywhere event today!

tfx.orchestration.airflow.airflow_component.AirflowComponent

Airflow-specific TFX Component.

This class wrap a component run into its own PythonOperator in Airflow.

parent_dag An AirflowPipeline instance as the pipeline DAG.
component An instance of base_node.BaseNode that holds all properties of a logical component.
component_launcher_class The class of the launcher to launch the component.
pipeline_info An instance of data_types.PipelineInfo that holds pipeline properties.
enable_cache Whether or not cache is enabled for this component run.
metadata_connection_config A config proto for metadata connection.
beam_pipeline_args Pipeline arguments for Beam powered Components.
additional_pipeline_args Additional pipeline args.
component_config Component config to launch the component.

dag Returns the Operator's DAG if set, otherwise raises an error
dag_id Returns dag id if it has one or an adhoc + owner
deps Returns the list of dependencies for the operator. These differ from execution context dependencies in that they are specific to tasks and can be extended/overridden by subclasses.
downstream_list @property: list of tasks directly downstream
downstream_task_ids @property: list of ids of tasks directly downstream
log

logger

priority_weight_total Total priority weight for the task. It might include all upstream or downstream tasks. depending on the weight rule.

  • WeightRule.ABSOLUTE - only own weight
  • WeightRule.DOWNSTREAM - adds priority weight of all downstream tasks
  • WeightRule.UPSTREAM - adds priority weight of all upstream tasks
schedule_interval The schedule interval of the DAG always wins over individual tasks so that tasks within a DAG always line up. The task still needs a schedule_interval as it may not be attached to a DAG.
task_type @property: type of the task
upstream_list @property: list of tasks directly upstream
upstream_task_ids @property: list of ids of tasks directly upstream

Methods

add_only_new

Adds only new items to item set

clear

Clears the state of task instances associated with the task, following the parameters specified.

dry_run

Performs dry run for the operator - just render template fields.

execute

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_callable

get_direct_relative_ids

Get the direct relative ids to the current task, upstream or downstream.

get_direct_relatives

Get the direct relatives to the current task, upstream or downstream.

For an operator, gets the URL that the external links specified in extra_links should point to.

:raise ValueError: The error message of a ValueError will be passed on through to the fronted to show up as a tooltip on the disabled link :param dttm: The datetime parsed execution date for the URL being searched for :param link_name: The name of the link we're looking for the URL for. Should be one of the options specified in extra_links :return: A URL

get_flat_relative_ids

Get a flat list of relatives' ids, either upstream or downstream.

get_flat_relatives

Get a flat list of relatives, either upstream or downstream.

get_serialized_fields

Stringified DAGs and operators contain exactly these fields.

get_task_instances

Get a set of task instance related to this task for a specific date range.

get_template_env

Fetch a Jinja template environment from the DAG or instantiate empty environment if no DAG.

has_dag

Returns True if the Operator has been assigned to a DAG.

on_kill

Override this method to cleanup subprocesses when a task instance gets killed. Any use of the threading, subprocess or multiprocessing module within an operator needs to be cleaned up or it will leave ghost processes behind.

post_execute

This hook is triggered right after self.execute() is called. It is passed the execution context and any results returned by the operator.

pre_execute

This hook is triggered right before self.execute() is called.

prepare_template

Hook that is triggered after the templated fields get replaced by their content. If you need your operator to alter the content of the file before the template is rendered, it should override this method to do so.

render_template

Render a templated string. The content can be a collection holding multiple templated strings and will be templated recursively.

:param content: Content to template. Only strings can be templated (may be inside collection). :type content: Any :param context: Dict with values to apply on templated content :type context: dict :param jinja_env: Jinja environment. Can be provided to avoid re-creating Jinja environments during recursion. :type jinja_env: jinja2.Environment :param seen_oids: template fields already rendered (to avoid RecursionError on circular dependencies) :type seen_oids: set :return: Templated content

render_template_fields

Template all attributes listed in template_fields. Note this operation is irreversible.

:param context: Dict with values to apply on content :type context: dict :param jinja_env: Jinja environment :type jinja_env: jinja2.Environment

resolve_template_files

run

Run a set of task instances for a date range.

set_downstream

Set a task or a task list to be directly downstream from the current task.

set_upstream

Set a task or a task list to be directly upstream from the current task.

xcom_pull

See TaskInstance.xcom_pull()

xcom_push

See TaskInstance.xcom_push()

__eq__

Return self==value.

__ge__

Return a >= b. Computed by @total_ordering from (not a < b).

__gt__

Return a > b. Computed by @total_ordering from (not a < b) and (a != b).

__le__

Return a <= b. Computed by @total_ordering from (a < b) or (a == b).

__lt__

Return self<value.

__ne__

Return self!=value.

extra_links

global_operator_extra_link_dict

operator_extra_link_dict

operator_extra_links ()
pool ''
shallow_copy_attrs

template_ext

template_fields

ui_color '#ffefeb'
ui_fgcolor '#000'