tfx.components.FileBasedExampleGen.DRIVER_CLASS

Custom driver for ExampleGen.

Inherits From: DRIVER_CLASS, BaseDriver

tfx.components.FileBasedExampleGen.DRIVER_CLASS(
    metadata_handler: tfx.orchestration.metadata.Metadata
)

This driver supports file based ExampleGen, e.g., for CsvExampleGen and ImportExampleGen.

mlmd_connection ML metadata connection.

Methods

pre_execution

View source

pre_execution(
    input_dict: Dict[Text, tfx.types.Channel],
    output_dict: Dict[Text, tfx.types.Channel],
    exec_properties: Dict[Text, Any],
    driver_args: tfx.orchestration.data_types.DriverArgs,
    pipeline_info: tfx.orchestration.data_types.PipelineInfo,
    component_info: tfx.orchestration.data_types.ComponentInfo
) -> tfx.orchestration.data_types.ExecutionDecision

Handle pre-execution logic.

There are four steps:

  1. Fetches input artifacts from metadata and checks whether uri exists.
  2. Registers execution.
  3. Decides whether a new execution is needed. 4a. If (3), prepare output artifacts. 4b. If not (3), fetch cached output artifacts.

Args
input_dict key -> Channel for inputs.
output_dict key -> Channel for outputs. Uris of the outputs are not assigned.
exec_properties Dict of other execution properties.
driver_args An instance of data_types.DriverArgs class.
pipeline_info An instance of data_types.PipelineInfo, holding pipeline related properties including pipeline_name, pipeline_root and run_id
component_info An instance of data_types.ComponentInfo, holding component related properties including component_type and component_id.

Returns
data_types.ExecutionDecision object.

Raises
RuntimeError if any input as an empty uri.

resolve_exec_properties

View source

resolve_exec_properties(
    exec_properties: Dict[Text, Any],
    pipeline_info: tfx.orchestration.data_types.PipelineInfo,
    component_info: tfx.orchestration.data_types.ComponentInfo
) -> Dict[Text, Any]

Overrides BaseDriver.resolve_exec_properties().

resolve_input_artifacts

View source

resolve_input_artifacts(
    input_dict: Dict[Text, tfx.types.Channel],
    exec_properties: Dict[Text, Any],
    driver_args: tfx.orchestration.data_types.DriverArgs,
    pipeline_info: tfx.orchestration.data_types.PipelineInfo
) -> Dict[Text, List[types.Artifact]]

Resolve input artifacts from metadata.

Subclasses might override this function for customized artifact properties resolution logic. However please note that this function is supposed to be called in normal cases (except head of the pipeline) since it handles artifact info passing from upstream components.

Args
input_dict key -> Channel mapping for inputs generated in logical pipeline.
exec_properties Dict of other execution properties, e.g., configs.
driver_args An instance of data_types.DriverArgs with driver configuration properties.
pipeline_info An instance of data_types.PipelineInfo, holding pipeline related properties including component_type and component_id.

Returns
Final artifacts that will be used in execution.

Raises
ValueError if in interactive mode, the given input channels have not been resolved.

run

View source

run(
    execution_info: tfx.orchestration.portable.data_types.ExecutionInfo
) -> driver_output_pb2.DriverOutput

Invokes the driver with inputs provided by the Launcher.

Args
execution_info a data_types.ExecutionInfo instance representing the execution info needed for the driver execution.

Returns
An DriverOutput instance.

verify_input_artifacts

View source

verify_input_artifacts(
    artifacts_dict: Dict[Text, List[types.Artifact]]
) -> None

Verify that all artifacts have existing uri.

Args
artifacts_dict key -> types.Artifact for inputs.

Raises
RuntimeError if any input as an empty or non-existing uri.