mlmd.metadata_store.MetadataStore

A store for the artifact metadata.

mlmd.metadata_store.MetadataStore(
    config: Union[metadata_store_pb2.ConnectionConfig, metadata_store_pb2.
        MetadataStoreClientConfig],
    enable_upgrade_migration: bool = False
)

config metadata_store_pb2.ConnectionConfig or metadata_store_pb2.MetadataStoreClientConfig. Configuration to connect to the database or the metadata store server.
enable_upgrade_migration if set to True, the library upgrades the db schema and migrates all data if it connects to an old version backend. It is ignored when using GRPC client connection config.

Methods

create_artifact_with_type

View source

create_artifact_with_type(
    artifact: metadata_store_pb2.Artifact,
    artifact_type: metadata_store_pb2.ArtifactType
) -> int

Creates an artifact with a type.

This first gets the type (or creates it if it does not exist), and then puts the artifact into the database with that type.

The type_id should not be specified in the artifact (it is ignored).

Note that this is not a transaction!

  1. First, the type is created as a transaction.
  2. Then the artifact is created as a transaction.

Args
artifact the artifact to create (no id or type_id)
artifact_type the type of the new artifact (no id)

Returns
the artifact ID of the resulting type.

Raises
InvalidArgument if the type is not the same as one with the same name already in the database.

get_artifact_by_type_and_name

View source

get_artifact_by_type_and_name(
    type_name: Text,
    artifact_name: Text
) -> Optional[metadata_store_pb2.Artifact]

Get the artifact of the given type and name.

The API fails if more than one artifact is found.

Args
type_name The artifact type name to look for.
artifact_name The artifact name to look for.

Returns
The Artifact matching the type and name. None if no matched Artifact was found.

get_artifact_type

View source

get_artifact_type(
    type_name: Text
) -> metadata_store_pb2.ArtifactType

Gets an artifact type by name.

Args
type_name the type with that name.

Returns
The type with name type_name.

Raises

tensorflow.errors.NotFoundError: if no type exists tensorflow.errors.InternalError: if query execution fails

get_artifact_types

View source

get_artifact_types() -> List[metadata_store_pb2.ArtifactType]

Gets all artifact types.

Returns
A list of all known ArtifactTypes.

Raises

tensorflow.errors.InternalError: if query execution fails

get_artifact_types_by_id

View source

get_artifact_types_by_id(
    type_ids: Iterable[int]
) -> List[metadata_store_pb2.ArtifactType]

Gets artifact types by ID.

Args
type_ids a sequence of artifact type IDs.

Returns
A list of artifact types.

Raises
InternalError if query execution fails.

get_artifacts

View source

get_artifacts(
    list_options: Optional[mlmd.metadata_store.ListOptions] = None
) -> List[metadata_store_pb2.Artifact]

Gets artifacts.

Args
list_options A set of options to limit the size and adjust order of the returned artifacts.

Returns
A list of artifacts.

Raises
InternalError if query execution fails.
InvalidArgument if list_options is invalid.

get_artifacts_by_context

View source

get_artifacts_by_context(
    context_id: int
) -> List[metadata_store_pb2.Artifact]

Gets all direct artifacts that a context attributes to.

Args
context_id The id of the querying context

Returns
Artifacts attributing to the context.

get_artifacts_by_id

View source

get_artifacts_by_id(
    artifact_ids: Iterable[int]
) -> List[metadata_store_pb2.Artifact]

Gets all artifacts with matching ids.

The result is not index-aligned: if an id is not found, it is not returned.

Args
artifact_ids A list of artifact ids to retrieve.

Returns
Artifacts with matching ids.

get_artifacts_by_type

View source

get_artifacts_by_type(
    type_name: Text
) -> List[metadata_store_pb2.Artifact]

Gets all the artifacts of a given type.

get_artifacts_by_uri

View source

get_artifacts_by_uri(
    uri: Text
) -> List[metadata_store_pb2.Artifact]

Gets all the artifacts of a given uri.

get_context_by_type_and_name

View source

get_context_by_type_and_name(
    type_name: Text,
    context_name: Text
) -> Optional[metadata_store_pb2.Context]

Get the context of the given type and context name.

The API fails if more than one contexts are found.

Args
type_name The context type name to look for.
context_name The context name to look for.

Returns
The Context matching the type and context name. None if no matched Context found.

get_context_type

View source

get_context_type(
    type_name: Text
) -> metadata_store_pb2.ContextType

Gets a context type by name.

Args
type_name the type with that name.

Returns
The type with name type_name.

Raises

tensorflow.errors.NotFoundError: if no type exists tensorflow.errors.InternalError: if query execution fails

get_context_types

View source

get_context_types() -> List[metadata_store_pb2.ContextType]

Gets all context types.

Returns
A list of all known ContextTypes.

Raises

tensorflow.errors.InternalError: if query execution fails

get_context_types_by_id

View source

get_context_types_by_id(
    type_ids: Iterable[int]
) -> List[metadata_store_pb2.ContextType]

Gets context types by ID.

Args
type_ids a sequence of context type IDs.

Returns
A list of context types.

Args
type_ids ids to look for.

Raises
InternalError if query execution fails.

get_contexts

View source

get_contexts(
    list_options: Optional[mlmd.metadata_store.ListOptions] = None
) -> List[metadata_store_pb2.Context]

Gets contexts.

Args
list_options A set of options to limit the size and adjust order of the returned contexts.

Returns
A list of contexts.

Raises
InternalError if query execution fails.
InvalidArgument if list_options is invalid.

get_contexts_by_artifact

View source

get_contexts_by_artifact(
    artifact_id: int
) -> List[metadata_store_pb2.Context]

Gets all context that an artifact is attributed to.

Args
artifact_id The id of the querying artifact

Returns
Contexts that the artifact is attributed to.

get_contexts_by_execution

View source

get_contexts_by_execution(
    execution_id: int
) -> List[metadata_store_pb2.Context]

Gets all context that an execution is associated with.

Args
execution_id The id of the querying execution

Returns
Contexts that the execution is associated with.

get_contexts_by_id

View source

get_contexts_by_id(
    context_ids: Iterable[int]
) -> List[metadata_store_pb2.Context]

Gets all contexts with matching ids.

The result is not index-aligned: if an id is not found, it is not returned.

Args
context_ids A list of context ids to retrieve.

Returns
Contexts with matching ids.

get_contexts_by_type

View source

get_contexts_by_type(
    type_name: Text
) -> List[metadata_store_pb2.Context]

Gets all the contexts of a given type.

Args
type_name The context type name to look for.

Returns
Contexts that matches the context type name.

get_events_by_artifact_ids

View source

get_events_by_artifact_ids(
    artifact_ids: Iterable[int]
) -> List[metadata_store_pb2.Event]

Gets all events with matching artifact ids.

Args
artifact_ids a list of artifact ids.

Returns
Events with the execution IDs given.

Raises
InternalError if query execution fails.

get_events_by_execution_ids

View source

get_events_by_execution_ids(
    execution_ids: Iterable[int]
) -> List[metadata_store_pb2.Event]

Gets all events with matching execution ids.

Args
execution_ids a list of execution ids.

Returns
Events with the execution IDs given.

Raises
InternalError if query execution fails.

get_execution_by_type_and_name

View source

get_execution_by_type_and_name(
    type_name: Text,
    execution_name: Text
) -> Optional[metadata_store_pb2.Execution]

Get the execution of the given type and name.

The API fails if more than one execution is found.

Args
type_name The execution type name to look for.
execution_name The execution name to look for.

Returns
The Execution matching the type and name. None if no matched Execution found.

get_execution_type

View source

get_execution_type(
    type_name: Text
) -> metadata_store_pb2.ExecutionType

Gets an execution type by name.

Args
type_name the type with that name.

Returns
The type with name type_name.

Raises

tensorflow.errors.NotFoundError: if no type exists tensorflow.errors.InternalError: if query execution fails

get_execution_types

View source

get_execution_types() -> List[metadata_store_pb2.ExecutionType]

Gets all execution types.

Returns
A list of all known ExecutionTypes.

Raises

tensorflow.errors.InternalError: if query execution fails

get_execution_types_by_id

View source

get_execution_types_by_id(
    type_ids: Iterable[int]
) -> List[metadata_store_pb2.ExecutionType]

Gets execution types by ID.

Args
type_ids a sequence of execution type IDs.

Returns
A list of execution types.

Args
type_ids ids to look for.

Raises
InternalError if query execution fails.

get_executions

View source

get_executions(
    list_options: Optional[mlmd.metadata_store.ListOptions] = None
) -> List[metadata_store_pb2.Execution]

Gets executions.

Args
list_options A set of options to limit the size and adjust order of the returned executions.

Returns
A list of executions.

Raises
InternalError if query execution fails.
InvalidArgument if list_options is invalid.

get_executions_by_context

View source

get_executions_by_context(
    context_id: int
) -> List[metadata_store_pb2.Execution]

Gets all direct executions that a context associates with.

Args
context_id The id of the querying context

Returns
Executions associating with the context.

get_executions_by_id

View source

get_executions_by_id(
    execution_ids: Iterable[int]
) -> List[metadata_store_pb2.Execution]

Gets all executions with matching ids.

The result is not index-aligned: if an id is not found, it is not returned.

Args
execution_ids A list of execution ids to retrieve.

Returns
Executions with matching ids.

get_executions_by_type

View source

get_executions_by_type(
    type_name: Text
) -> List[metadata_store_pb2.Execution]

Gets all the executions of a given type.

put_artifact_type

View source

put_artifact_type(
    artifact_type: metadata_store_pb2.ArtifactType,
    can_add_fields: bool = False,
    can_omit_fields: bool = False
) -> int

Inserts or updates an artifact type.

If no type exists in the database with the given name, it creates a new type and returns the type_id.

If the request type with the same name already exists (let's call it stored_type), the method enforces the stored_type can be updated only when the request type is backward compatible for the already stored instances.

Backwards compatibility is violated iff:

a) there is a property where the request type and stored_type have different value type (e.g., int vs. string) b) can_add_fields = false and the request type has a new property that is not stored. c) can_omit_fields = false and stored_type has an existing property that is not provided in the request type.

Args
artifact_type the request type to be inserted or updated.
can_add_fields when true, new properties can be added; when false, returns ALREADY_EXISTS if the request type has properties that are not in stored_type.
can_omit_fields when true, stored properties can be omitted in the request type; when false, returns ALREADY_EXISTS if the stored_type has properties not in the request type.

Returns
the type_id of the response.

Raises
AlreadyExistsError If the type is not backward compatible.
InvalidArgumentError If the request type has no name, or any property value type is unknown.

put_artifacts

View source

put_artifacts(
    artifacts: Sequence[metadata_store_pb2.Artifact]
) -> List[int]

Inserts or updates artifacts in the database.

If an artifact_id is specified for an artifact, it is an update. If an artifact_id is unspecified, it will insert a new artifact. For new artifacts, type must be specified. For old artifacts, type must be unchanged or unspecified. When the name of an artifact is given, it should be unique among artifacts of the same ArtifactType.

Args
artifacts A list of artifacts to insert or update.

Returns
A list of artifact ids index-aligned with the input.

Raises
AlreadyExistsError If artifact's name is specified and it is already used by stored artifacts of that ArtifactType.

put_attributions_and_associations

View source

put_attributions_and_associations(
    attributions: Sequence[metadata_store_pb2.Attribution],
    associations: Sequence[metadata_store_pb2.Association]
) -> None

Inserts attribution and association relationships in the database.

The context_id, artifact_id, and execution_id must already exist. If the relationship exists, this call does nothing. Once added, the relationships cannot be modified.

Args
attributions A list of attributions to insert.
associations A list of associations to insert.

put_context_type

View source

put_context_type(
    context_type: metadata_store_pb2.ContextType,
    can_add_fields: bool = False,
    can_omit_fields: bool = False
) -> int

Inserts or updates a context type.

If no type exists in the database with the given name, it creates a new type and returns the type_id.

If the request type with the same name already exists (let's call it stored_type), the method enforces the stored_type can be updated only when the request type is backward compatible for the already stored instances.

Backwards compatibility is violated iff:

a) there is a property where the request type and stored_type have different value type (e.g., int vs. string) b) can_add_fields = false and the request type has a new property that is not stored. c) can_omit_fields = false and stored_type has an existing property that is not provided in the request type.

Args
context_type the request type to be inserted or updated.
can_add_fields when true, new properties can be added; when false, returns ALREADY_EXISTS if the request type has properties that are not in stored_type.
can_omit_fields when true, stored properties can be omitted in the request type; when false, returns ALREADY_EXISTS if the stored_type has properties not in the request type.

Returns
the type_id of the response.

Raises
AlreadyExistsError If the type is not backward compatible.
InvalidArgumentError If the request type has no name, or any property value type is unknown.

put_contexts

View source

put_contexts(
    contexts: Sequence[metadata_store_pb2.Context]
) -> List[int]

Inserts or updates contexts in the database.

If an context_id is specified for an context, it is an update. If an context_id is unspecified, it will insert a new context. For new contexts, type must be specified. For old contexts, type must be unchanged or unspecified. The name of a context cannot be empty, and it should be unique among contexts of the same ContextType.

Args
contexts A list of contexts to insert or update.

Returns
A list of context ids index-aligned with the input.

Raises
InvalidArgumentError If name of the new contexts are empty.
AlreadyExistsError If name of the new contexts already used by stored contexts of that ContextType.

put_events

View source

put_events(
    events: Sequence[metadata_store_pb2.Event]
) -> None

Inserts events in the database.

The execution_id and artifact_id must already exist. Once created, events cannot be modified.

Args
events A list of events to insert.

put_execution

View source

put_execution(
    execution: metadata_store_pb2.Execution,
    artifact_and_events: Sequence[Tuple[metadata_store_pb2.Artifact, Optional[metadata_store_pb2.Event]]
        ],
    contexts: Optional[Sequence[metadata_store_pb2.Context]],
    reuse_context_if_already_exist: bool = False
) -> Tuple[int, List[int], List[int]]

Inserts or updates an Execution with artifacts, events and contexts.

In contrast with other put methods, the method update an execution atomically with its input/output artifacts and events and adds attributions and associations to related contexts.

If an execution_id, artifact_id or context_id is specified, it is an update, otherwise it does an insertion.

Args
execution The execution to be created or updated.
artifact_and_events a pair of Artifact and Event that the execution uses or generates. The event's execution id or artifact id can be empty, as the artifact or execution may not be stored beforehand. If given, the ids must match with the paired Artifact and the input execution.
contexts The Contexts that the execution should be associated with and the artifacts should be attributed to.
reuse_context_if_already_exist when there's a race to publish executions with a new context (no id) with the same context.name, by default there will be one writer succeeds and the rest of the writers fail with AlreadyExists errors. If set is to True, failed writers will reuse the stored context.

Returns
the execution id, the list of artifact's id, and the list of context's id.

Raises
InvalidArgumentError If the id of the input nodes do not align with the store. Please refer to InvalidArgument errors in other put methods.
AlreadyExistsError If the new nodes to be created is already exists. Please refer to AlreadyExists errors in other put methods.

put_execution_type

View source

put_execution_type(
    execution_type: metadata_store_pb2.ExecutionType,
    can_add_fields: bool = False,
    can_omit_fields: bool = False
) -> int

Inserts or updates an execution type.

If no type exists in the database with the given name, it creates a new type and returns the type_id.

If the request type with the same name already exists (let's call it stored_type), the method enforces the stored_type can be updated only when the request type is backward compatible for the already stored instances.

Backwards compatibility is violated iff:

a) there is a property where the request type and stored_type have different value type (e.g., int vs. string) b) can_add_fields = false and the request type has a new property that is not stored. c) can_omit_fields = false and stored_type has an existing property that is not provided in the request type.

Args
execution_type the request type to be inserted or updated.
can_add_fields when true, new properties can be added; when false, returns ALREADY_EXISTS if the request type has properties that are not in stored_type.
can_omit_fields when true, stored properties can be omitted in the request type; when false, returns ALREADY_EXISTS if the stored_type has properties not in the request type.

Returns
the type_id of the response.

Raises
AlreadyExistsError If the type is not backward compatible.
InvalidArgumentError If the request type has no name, or any property value type is unknown.

put_executions

View source

put_executions(
    executions: Sequence[metadata_store_pb2.Execution]
) -> List[int]

Inserts or updates executions in the database.

If an execution_id is specified for an execution, it is an update. If an execution_id is unspecified, it will insert a new execution. For new executions, type must be specified. For old executions, type must be unchanged or unspecified. When the name of an execution is given, it should be unique among executions of the same ExecutionType.

Args
executions A list of executions to insert or update.

Returns
A list of execution ids index-aligned with the input.

Raises
AlreadyExistsError If execution's name is specified and it is already used by stored executions of that ExecutionType.