Missed TensorFlow Dev Summit? Check out the video playlist. Watch recordings

mlmd.metadata_store.MetadataStore

View source on GitHub

A store for the artifact metadata.

mlmd.metadata_store.MetadataStore(
    config, enable_upgrade_migration=False
)

Args:

  • config: metadata_store_pb2.ConnectionConfig or metadata_store_pb2.MetadataStoreClientConfig. Configuration to connect to the database or the metadata store server.
  • enable_upgrade_migration: if set to True, the library upgrades the db schema and migrates all data if it connects to an old version backend. It is ignored when using GRPC client connection config.

Methods

create_artifact_with_type

View source

create_artifact_with_type(
    artifact, artifact_type
)

Creates an artifact with a type.

This first gets the type (or creates it if it does not exist), and then puts the artifact into the database with that type.

The type_id should not be specified in the artifact (it is ignored).

Note that this is not a transaction!

  1. First, the type is created as a transaction.
  2. Then the artifact is created as a transaction.

Args:

  • artifact: the artifact to create (no id or type_id)
  • artifact_type: the type of the new artifact (no id)

Returns:

the artifact ID of the resulting type.

Raises:

  • InvalidArgument: if the type is not the same as one with the same name already in the database.

get_artifact_type

View source

get_artifact_type(
    type_name
)

Gets an artifact type by name.

Args:

  • type_name: the type with that name.

Returns:

The type with name type_name.

Raises:

tensorflow.errors.NotFoundError: if no type exists tensorflow.errors.InternalError: if query execution fails

get_artifact_types

View source

get_artifact_types()

Gets all artifact types.

Returns:

A list of all known ArtifactTypes.

Raises:

tensorflow.errors.InternalError: if query execution fails

get_artifact_types_by_id

View source

get_artifact_types_by_id(
    type_ids
)

Gets artifact types by ID.

Args:

  • type_ids: a sequence of artifact type IDs.

Returns:

A list of artifact types.

Raises:

  • InternalError: if query execution fails.

get_artifacts

View source

get_artifacts()

Gets all artifacts.

Returns:

A list of all artifacts.

Raises:

  • InternalError: if query execution fails.

get_artifacts_by_context

View source

get_artifacts_by_context(
    context_id
)

Gets all direct artifacts that a context attributes to.

Args:

  • context_id: The id of the querying context

Returns:

Artifacts attributing to the context.

get_artifacts_by_id

View source

get_artifacts_by_id(
    artifact_ids
)

Gets all artifacts with matching ids.

The result is not index-aligned: if an id is not found, it is not returned.

Args:

  • artifact_ids: A list of artifact ids to retrieve.

Returns:

Artifacts with matching ids.

get_artifacts_by_type

View source

get_artifacts_by_type(
    type_name
)

Gets all the artifacts of a given type.

get_artifacts_by_uri

View source

get_artifacts_by_uri(
    uri
)

Gets all the artifacts of a given uri.

get_context_by_type_and_name

View source

get_context_by_type_and_name(
    type_name, context_name
)

Get the context of the given type and context name.

The API fails if more than one contexts are found.

Args:

  • type_name: The context type name to look for.
  • context_name: The context name to look for.

Returns:

The Context matching the type and context name. None if no matched Context found.

get_context_type

View source

get_context_type(
    type_name
)

Gets a context type by name.

Args:

  • type_name: the type with that name.

Returns:

The type with name type_name.

Raises:

tensorflow.errors.NotFoundError: if no type exists tensorflow.errors.InternalError: if query execution fails

get_context_types

View source

get_context_types()

Gets all context types.

Returns:

A list of all known ContextTypes.

Raises:

tensorflow.errors.InternalError: if query execution fails

get_context_types_by_id

View source

get_context_types_by_id(
    type_ids
)

Gets context types by ID.

Args:

  • type_ids: a sequence of context type IDs.

Returns:

A list of context types.

Args:

  • type_ids: ids to look for.

Raises:

  • InternalError: if query execution fails.

get_contexts

View source

get_contexts()

Gets all contexts.

Returns:

A list of all contexts.

Raises:

  • InternalError: if query execution fails.

get_contexts_by_artifact

View source

get_contexts_by_artifact(
    artifact_id
)

Gets all context that an artifact is attributed to.

Args:

  • artifact_id: The id of the querying artifact

Returns:

Contexts that the artifact is attributed to.

get_contexts_by_execution

View source

get_contexts_by_execution(
    execution_id
)

Gets all context that an execution is associated with.

Args:

  • execution_id: The id of the querying execution

Returns:

Contexts that the execution is associated with.

get_contexts_by_id

View source

get_contexts_by_id(
    context_ids
)

Gets all contexts with matching ids.

The result is not index-aligned: if an id is not found, it is not returned.

Args:

  • context_ids: A list of context ids to retrieve.

Returns:

Contexts with matching ids.

get_contexts_by_type

View source

get_contexts_by_type(
    type_name
)

Gets all the contexts of a given type.

Args:

  • type_name: The context type name to look for.

Returns:

Contexts that matches the context type name.

get_events_by_artifact_ids

View source

get_events_by_artifact_ids(
    artifact_ids
)

Gets all events with matching artifact ids.

Args:

  • artifact_ids: a list of artifact ids.

Returns:

Events with the execution IDs given.

Raises:

  • InternalError: if query execution fails.

get_events_by_execution_ids

View source

get_events_by_execution_ids(
    execution_ids
)

Gets all events with matching execution ids.

Args:

  • execution_ids: a list of execution ids.

Returns:

Events with the execution IDs given.

Raises:

  • InternalError: if query execution fails.

get_execution_type

View source

get_execution_type(
    type_name
)

Gets an execution type by name.

Args:

  • type_name: the type with that name.

Returns:

The type with name type_name.

Raises:

tensorflow.errors.NotFoundError: if no type exists tensorflow.errors.InternalError: if query execution fails

get_execution_types

View source

get_execution_types()

Gets all execution types.

Returns:

A list of all known ExecutionTypes.

Raises:

tensorflow.errors.InternalError: if query execution fails

get_execution_types_by_id

View source

get_execution_types_by_id(
    type_ids
)

Gets execution types by ID.

Args:

  • type_ids: a sequence of execution type IDs.

Returns:

A list of execution types.

Args:

  • type_ids: ids to look for.

Raises:

  • InternalError: if query execution fails.

get_executions

View source

get_executions()

Gets all executions.

Returns:

A list of all executions.

Raises:

  • InternalError: if query execution fails.

get_executions_by_context

View source

get_executions_by_context(
    context_id
)

Gets all direct executions that a context associates with.

Args:

  • context_id: The id of the querying context

Returns:

Executions associating with the context.

get_executions_by_id

View source

get_executions_by_id(
    execution_ids
)

Gets all executions with matching ids.

The result is not index-aligned: if an id is not found, it is not returned.

Args:

  • execution_ids: A list of execution ids to retrieve.

Returns:

Executions with matching ids.

get_executions_by_type

View source

get_executions_by_type(
    type_name
)

Gets all the executions of a given type.

put_artifact_type

View source

put_artifact_type(
    artifact_type, can_add_fields=False, can_delete_fields=False,
    all_fields_match=True
)

Inserts or updates an artifact type.

Similar to put execution/context type, if no artifact type exists in the database with the given name, it creates a new artifact type (and a database).

If an artifact type with the same name already exists (let's call it old_artifact_type), then the impact depends upon the other options.

If artifact_type == old_artifact_type, then nothing happens.

Otherwise, if there is a field where artifact_type and old_artifact_type have different types, then it fails.

Otherwise, if can_add_fields is False and artifact_type has a field old_artifact_type is missing, then it fails.

Otherwise, if all_fields_match is True and old_artifact_type has a field artifact_type is missing, then it fails.

Otherwise, if can_delete_fields is True and old_artifact_type has a field artifact_type is missing, then it deletes that field.

Otherwise, it does nothing.

Args:

  • artifact_type: the type to add or update.
  • can_add_fields: if true, you can add fields with this operation. If false, then if there are more fields in artifact_type than in the database, the call fails.
  • can_delete_fields: if true, you can remove fields with this operation. If false, then if there are more fields in the current type, they are not removed.
  • all_fields_match: if true, all fields must match, and the method fails if they are not the same.

Returns:

the type_id of the response.

Raises:

  • InvalidArgumentError: If a constraint is violated.

put_artifacts

View source

put_artifacts(
    artifacts
)

Inserts or updates artifacts in the database.

If an artifact_id is specified for an artifact, it is an update. If an artifact_id is unspecified, it will insert a new artifact. For new artifacts, type must be specified. For old artifacts, type must be unchanged or unspecified.

Args:

  • artifacts: A list of artifacts to insert or update.

Returns:

A list of artifact ids index-aligned with the input.

put_attributions_and_associations

View source

put_attributions_and_associations(
    attributions, associations
)

Inserts attribution and association relationships in the database.

The context_id, artifact_id, and execution_id must already exist. If the relationship exists, this call does nothing. Once added, the relationships cannot be modified.

Args:

  • attributions: A list of attributions to insert.
  • associations: A list of associations to insert.

put_context_type

View source

put_context_type(
    context_type, can_add_fields=False, can_delete_fields=False,
    all_fields_match=True
)

Inserts or updates a context type.

Similar to put artifact/execution type, if no context type exists in the database with the given name, it creates a new context type (and a database).

If a context type with the same name already exists (let's call it old_context_type), then the impact depends upon the other options.

If context_type == old_context_type, then nothing happens.

Otherwise, if there is a field where context_type and old_context_type have different types, then it fails.

Otherwise, if can_add_fields is False and context_type has a field old_context_type is missing, then it fails.

Otherwise, if all_fields_match is True and old_context_type has a field context_type is missing, then it fails.

Otherwise, if can_delete_fields is True and old_context_type has a field context_type is missing, then it deletes that field.

Otherwise, it does nothing.

Args:

  • context_type: the type to add or update.
  • can_add_fields: if true, you can add fields with this operation. If false, then if there are more fields in context_type than in the database, the call fails.
  • can_delete_fields: if true, you can remove fields with this operation. If false, then if there are more fields in the current type, they are not removed.
  • all_fields_match: if true, all fields must match, and the method fails if they are not the same.

Returns:

the type_id of the response.

Raises:

  • InvalidArgumentError: If a constraint is violated.

put_contexts

View source

put_contexts(
    contexts
)

Inserts or updates contexts in the database.

If an context_id is specified for an context, it is an update. If an context_id is unspecified, it will insert a new context. For new contexts, type must be specified. For old contexts, type must be unchanged or unspecified. The name of a context cannot be empty, and it should be unique among contexts of the same ContextType.

Args:

  • contexts: A list of contexts to insert or update.

Returns:

A list of context ids index-aligned with the input.

put_events

View source

put_events(
    events
)

Inserts events in the database.

The execution_id and artifact_id must already exist. Once created, events cannot be modified.

Args:

  • events: A list of events to insert.

put_execution

View source

put_execution(
    execution, artifact_and_events, contexts
)

Inserts or updates an Execution with artifacts, events and contexts.

If an execution_id, artifact_id or context_id is specified, it is an update, otherwise it does an insertion.

Args:

  • execution: The execution to be created or updated.
  • artifact_and_events: a pair of Artifact and Event that the execution uses or generates. The event's execution id or artifact id can be empty, as the artifact or execution may not be stored beforehand. If given, the ids must match with the paired Artifact and the input execution.
  • contexts: The Contexts that the execution should be assotiated with and the artifacts should be attributed to.

Returns:

the execution id, the list of artifact's id, and the list of context's id.

put_execution_type

View source

put_execution_type(
    execution_type, can_add_fields=False, can_delete_fields=False,
    all_fields_match=True
)

Inserts or updates an execution type.

Similar to put artifact/context type, if no execution type exists in the database with the given name, it creates a new execution type (and a database).

If an execution type with the same name already exists (let's call it old_execution_type), then the impact depends upon the other options.

If execution_type == old_execution_type, then nothing happens.

Otherwise, if there is a field where execution_type and old_execution_type have different types, then it fails.

Otherwise, if can_add_fields is False and execution_type has a field old_execution_type is missing, then it fails.

Otherwise, if all_fields_match is True and old_execution_type has a field execution_type is missing, then it fails.

Otherwise, if can_delete_fields is True and old_execution_type has a field execution_type is missing, then it deletes that field.

Otherwise, it does nothing. Args: execution_type: the type to add or update. can_add_fields: if true, you can add fields with this operation. If false, then if there are more fields in execution_type than in the database, the call fails. can_delete_fields: if true, you can remove fields with this operation. If false, then if there are more fields. all_fields_match: if true, all fields must match, and the method fails if they are not the same.

Returns:

the type id of the type.

Raises:

  • ValueError: If a constraint is violated.

put_executions

View source

put_executions(
    executions
)

Inserts or updates executions in the database.

If an execution_id is specified for an execution, it is an update. If an execution_id is unspecified, it will insert a new execution. For new executions, type must be specified. For old executions, type must be unchanged or unspecified.

Args:

  • executions: A list of executions to insert or update.

Returns:

A list of execution ids index-aligned with the input.