tfx.components.example_gen.import_example_gen.executor.Executor

View source on GitHub

Class Executor

Generic TFX import example gen executor.

Inherits From: BaseExampleGenExecutor

__init__

View source

__init__(context=None)

Constructs a beam based executor.

Child Classes

class Context

Methods

Do

View source

Do(
    input_dict,
    output_dict,
    exec_properties
)

Take input data source and generates TF Example splits.

Args:

  • input_dict: Input dict from input key to a list of Artifacts. Depends on detailed example gen implementation.
  • output_dict: Output dict from output key to a list of Artifacts.
    • examples: splits of tf examples.
  • exec_properties: A dict of execution properties. Depends on detailed example gen implementation.
    • input: JSON string of example_gen_pb2.Input instance, providing input configuration.
    • output: JSON string of example_gen_pb2.Output instance, providing output configuration.

Returns:

None

GenerateExamplesByBeam

View source

GenerateExamplesByBeam(
    pipeline,
    input_dict,
    exec_properties
)

Converts input source to TF example splits based on configs.

Custom ExampleGen executor should provide GetInputSourceToExamplePTransform for converting input split to TF Examples. Overriding this 'GenerateExamplesByBeam' method instead if complex logic is need, e.g., custom spliting logic.

Args:

  • pipeline: beam pipeline.
  • input_dict: Input dict from input key to a list of Artifacts. Depends on detailed example gen implementation.
  • exec_properties: A dict of execution properties. Depends on detailed example gen implementation.
    • input: JSON string of example_gen_pb2.Input instance, providing input configuration.
    • output: JSON string of example_gen_pb2.Output instance, providing output configuration.

Returns:

Dict of beam PCollection with split name as key, each PCollection is a single output split that contains serialized TF Examples.

GetInputSourceToExamplePTransform

View source

GetInputSourceToExamplePTransform()

Returns PTransform for importing TF examples.