tfx.components.FileBasedExampleGen.DRIVER_CLASS

View source on GitHub

Custom driver for ExampleGen.

Inherits From: DRIVER_CLASS

View aliases

Main aliases

tfx.components.CsvExampleGen.DRIVER_CLASS, tfx.components.ImportExampleGen.DRIVER_CLASS, tfx.components.example_gen.component.FileBasedExampleGen.DRIVER_CLASS, tfx.components.example_gen.csv_example_gen.component.CsvExampleGen.DRIVER_CLASS, tfx.components.example_gen.driver.Driver, tfx.components.example_gen.import_example_gen.component.ImportExampleGen.DRIVER_CLASS

tfx.components.FileBasedExampleGen.DRIVER_CLASS(
    metadata_handler: tfx.orchestration.metadata.Metadata
)

This driver supports file based ExampleGen, it registers external file path as an artifact, e.g., for CsvExampleGen and ImportExampleGen.

Methods

pre_execution

View source

pre_execution(
    input_dict: Dict[Text, tfx.types.Channel],
    output_dict: Dict[Text, tfx.types.Channel],
    exec_properties: Dict[Text, Any],
    driver_args: tfx.orchestration.data_types.DriverArgs,
    pipeline_info: tfx.orchestration.data_types.PipelineInfo,
    component_info: tfx.orchestration.data_types.ComponentInfo
) -> tfx.orchestration.data_types.ExecutionDecision

Handle pre-execution logic.

There are four steps:

1. Fetches input artifacts from metadata and checks whether uri exists.
2. Registers execution.
3. Decides whether a new execution is needed. 4a. If (3), prepare output artifacts. 4b. If not (3), fetch cached output artifacts.

Args
input_dict	key -> Channel for inputs.
output_dict	key -> Channel for outputs. Uris of the outputs are not assigned.
exec_properties	Dict of other execution properties.
driver_args	An instance of data_types.DriverArgs class.
pipeline_info	An instance of data_types.PipelineInfo, holding pipeline related properties including pipeline_name, pipeline_root and run_id
component_info	An instance of data_types.ComponentInfo, holding component related properties including component_type and component_id.

Returns
data_types.ExecutionDecision object.

Raises
RuntimeError	if any input as an empty uri.

resolve_exec_properties

View source

resolve_exec_properties(
    exec_properties: Dict[Text, Any],
    pipeline_info: tfx.orchestration.data_types.PipelineInfo,
    component_info: tfx.orchestration.data_types.ComponentInfo
) -> Dict[Text, Any]

Resolve execution properties.

Subclasses might override this function for customized execution properties resolution logic.

Args
exec_properties	Original execution properties passed in.
pipeline_info	An instance of data_types.PipelineInfo, holding pipeline related properties including pipeline_name, pipeline_root and run_id
component_info	An instance of data_types.ComponentInfo, holding component related properties including component_type and component_id.

Returns
Final execution properties that will be used in execution.

resolve_input_artifacts

View source

resolve_input_artifacts(
    input_channels: Dict[Text, tfx.types.Channel],
    exec_properties: Dict[Text, Any],
    driver_args: tfx.orchestration.data_types.DriverArgs,
    pipeline_info: tfx.orchestration.data_types.PipelineInfo
) -> Dict[Text, List[tfx.types.Artifact]]

Overrides BaseDriver.resolve_input_artifacts().

verify_input_artifacts

View source

verify_input_artifacts(
    artifacts_dict: Dict[Text, List[tfx.types.Artifact]]
) -> None

Verify that all artifacts have existing uri.

Args
artifacts_dict	key -> types.Artifact for inputs.

Raises
RuntimeError	if any input as an empty or non-existing uri.