tflite::Interpreter

#include <interpreter.h>

An interpreter for a graph of nodes that input and output from tensors.

Summary

Each node of the graph processes a set of input tensors and produces a set of output Tensors. All inputs/output tensors are referenced by index.

Usage:


// Create basic model
Interpreter foo(2, 1);
foo.SetTensorParametersReadWrite(0, ...);
foo.SetTensorParametersReadOnly(1, ...);
foo.SetNodeParameters(0, ...)
// Resize input array to 1 length.
foo.ResizeInputTensor(0, 1);
foo.AllocateTensors();
// Install array data
foo.typed_tensor(0)[0] = 3;
foo.Invoke();
foo.typed_tensor(0)[0] = 4;
foo.Invoke();
// Resize input array and set data.
foo.ResizeInputTensor(0, 2);
foo.AllocateTensors();
foo.typed_tensor(0)[0] = 4;
foo.typed_tensor(0)[1] = 8;
foo.Invoke();

Constructors and Destructors

Interpreter(ErrorReporter *error_reporter)
Instantiate an interpreter.
Interpreter(const Interpreter &)
~Interpreter()

Public types

TfLiteDelegatePtr using
std::unique_ptr< TfLiteDelegate, void(*)(TfLiteDelegate *)>

Public static attributes

kTensorsCapacityHeadroom = 16
constexpr int
The capacity headroom of tensors_ vector before calling ops' prepare and invoke function.
kTensorsReservedCapacity = 128
constexpr int

Public functions

AllocateTensors()
TfLiteStatus
Update allocations for all tensors.
EnsureTensorDataIsReadable(int tensor_index)
TfLiteStatus
Ensure the data in tensor.data is readable.
GetAllowFp16PrecisionForFp32() const
bool
Get the half precision flag.
GetBufferHandle(int tensor_index, TfLiteBufferHandle *buffer_handle, TfLiteDelegate **delegate)
TfLiteStatus
Get the delegate buffer handle, and the delegate which can process the buffer handle.
GetInputName(int index) const
const char *
Return the name of a given input.
GetOutputName(int index) const
const char *
Return the name of a given output.
GetProfiler()
Profiler *
Gets the profiler used for op tracing.
Invoke()
TfLiteStatus
Invoke the interpreter (run the whole graph in dependency order).
ModifyGraphWithDelegate(TfLiteDelegate *delegate)
TfLiteStatus
Allow a delegate to look at the graph and modify the graph to handle parts of the graph themselves.
ModifyGraphWithDelegate(TfLiteDelegatePtr delegate)
TfLiteStatus
Same as ModifyGraphWithDelegate except this interpreter takes ownership of the provided delegate.
OpProfilingString(const TfLiteRegistration & op_reg, const TfLiteNode *node) const
const char *
Retrieve an operator's description of its work, for profiling purposes.
ResetVariableTensors()
TfLiteStatus
Reset all variable tensors to the default value.
ResizeInputTensor(int tensor_index, const std::vector< int > & dims)
TfLiteStatus
Change the dimensionality of a given tensor.
SetAllowBufferHandleOutput(bool allow_buffer_handle_output)
void
Set if buffer handle output is allowed.
SetAllowFp16PrecisionForFp32(bool allow)
void
Allow float16 precision for FP32 calculation when possible.
SetBufferHandle(int tensor_index, TfLiteBufferHandle buffer_handle, TfLiteDelegate *delegate)
TfLiteStatus
Set the delegate buffer handle to a tensor.
SetCancellationFunction(void *data, bool(*)(void *) check_cancelled_func)
void
Sets the cancellation function pointer in order to cancel a request in the middle of a call to Invoke().
SetExecutionPlan(const std::vector< int > & new_plan)
TfLiteStatus
WARNING: Experimental interface, subject to change Overrides execution plan.
SetExternalContext(TfLiteExternalContextType type, TfLiteExternalContext *ctx)
void
SetNumThreads(int num_threads)
void
Set the number of threads available to the interpreter.
SetProfiler(Profiler *profiler)
void
Sets the profiler to tracing execution.
UseNNAPI(bool enable)
void
Enable or disable the NN API (true to enable)
execution_plan() const
const std::vector< int > &
WARNING: Experimental interface, subject to change.
inputs() const
const std::vector< int > &
Read only access to list of inputs.
node_and_registration(int node_index) const
const std::pair< TfLiteNode, TfLiteRegistration > *
Get a pointer to an operation and registration data structure if in bounds.
nodes_size() const
size_t
Return the number of ops in the model.
operator=(const Interpreter &)=delete
outputs() const
const std::vector< int > &
Read only access to list of outputs.
tensor(int tensor_index)
TfLiteTensor *
Get a mutable tensor data structure.
tensor(int tensor_index) const
const TfLiteTensor *
Get an immutable tensor data structure.
tensors_size() const
size_t
Return the number of tensors in the model.
typed_input_tensor(int index)
T *
Return a mutable pointer into the data of a given input tensor.
typed_input_tensor(int index) const
const T *
Return an immutable pointer into the data of a given input tensor.
typed_output_tensor(int index)
T *
Return a mutable pointer into the data of a given output tensor.
typed_output_tensor(int index) const
const T *
Return an immutable pointer into the data of a given output tensor.
typed_tensor(int tensor_index)
T *
Perform a checked cast to the appropriate tensor type (mutable pointer version).
typed_tensor(int tensor_index) const
const T *
Perform a checked cast to the appropriate tensor type (immutable pointer version).
variables() const
const std::vector< int > &
Read only access to list of variable tensors.

Public types

TfLiteDelegatePtr

std::unique_ptr< TfLiteDelegate, void(*)(TfLiteDelegate *)> TfLiteDelegatePtr

Public static attributes

kTensorsCapacityHeadroom

constexpr int kTensorsCapacityHeadroom = 16

The capacity headroom of tensors_ vector before calling ops' prepare and invoke function.

In these functions, it's guaranteed allocating up to kTensorsCapacityHeadroom more tensors won't invalidate pointers to existing tensors.

kTensorsReservedCapacity

constexpr int kTensorsReservedCapacity = 128

Public functions

AllocateTensors

TfLiteStatus AllocateTensors()

Update allocations for all tensors.

This will redim dependent tensors using the input tensor dimensionality as given. This is relatively expensive. If you know that your sizes are not changing, you need not call this. Returns status of success or failure.

EnsureTensorDataIsReadable

TfLiteStatus EnsureTensorDataIsReadable(
  int tensor_index
)

Ensure the data in tensor.data is readable.

In case delegate is used, it might require to copy the data from delegate buffer to raw memory. WARNING: This is an experimental API and subject to change.

GetAllowFp16PrecisionForFp32

bool GetAllowFp16PrecisionForFp32() const 

Get the half precision flag.

WARNING: This is an experimental API and subject to change.

GetBufferHandle

TfLiteStatus GetBufferHandle(
  int tensor_index,
  TfLiteBufferHandle *buffer_handle,
  TfLiteDelegate **delegate
)

Get the delegate buffer handle, and the delegate which can process the buffer handle.

WARNING: This is an experimental API and subject to change.

GetInputName

const char * GetInputName(
  int index
) const 

Return the name of a given input.

The given index must be between 0 and inputs().size().

GetOutputName

const char * GetOutputName(
  int index
) const 

Return the name of a given output.

The given index must be between 0 and outputs().size().

GetProfiler

Profiler * GetProfiler()

Gets the profiler used for op tracing.

WARNING: This is an experimental API and subject to change.

Interpreter

 Interpreter(
  ErrorReporter *error_reporter
)

Instantiate an interpreter.

All errors associated with reading and processing this model will be forwarded to the error_reporter object. Note, if error_reporter is nullptr, then a default StderrReporter is used. Ownership of 'error_reporter' remains with the caller.

Interpreter

 Interpreter(
  const Interpreter &
)=delete

Invoke

TfLiteStatus Invoke()

Invoke the interpreter (run the whole graph in dependency order).

NOTE: It is possible that the interpreter is not in a ready state to evaluate (i.e. if a ResizeTensor() has been performed without an AllocateTensors(). Returns status of success or failure.

ModifyGraphWithDelegate

TfLiteStatus ModifyGraphWithDelegate(
  TfLiteDelegate *delegate
)

Allow a delegate to look at the graph and modify the graph to handle parts of the graph themselves.

After this is called, the graph may contain new nodes that replace 1 more nodes. 'delegate' must outlive the interpreter. WARNING: This is an experimental API and subject to change.

ModifyGraphWithDelegate

TfLiteStatus ModifyGraphWithDelegate(
  TfLiteDelegatePtr delegate
)

Same as ModifyGraphWithDelegate except this interpreter takes ownership of the provided delegate.

Be sure to construct the unique_ptr with a suitable destruction function. WARNING: This is an experimental API and subject to change.

OpProfilingString

const char * OpProfilingString(
  const TfLiteRegistration & op_reg,
  const TfLiteNode *node
) const 

Retrieve an operator's description of its work, for profiling purposes.

ResetVariableTensors

TfLiteStatus ResetVariableTensors()

Reset all variable tensors to the default value.

If a variable tensor doesn't have a buffer, reset it to zero. TODO(b/115961645): Implement - If a variable tensor has a buffer, reset it to the value of the buffer. WARNING: This is an experimental API and subject to change.

ResizeInputTensor

TfLiteStatus ResizeInputTensor(
  int tensor_index,
  const std::vector< int > & dims
)

Change the dimensionality of a given tensor.

Note, this is only acceptable for tensor indices that are inputs or variables. Returns status of failure or success. TODO(aselle): Consider implementing ArraySlice equivalent to make this more adept at accepting data without an extra copy. Use absl::ArraySlice if our partners determine that dependency is acceptable.

SetAllowBufferHandleOutput

void SetAllowBufferHandleOutput(
  bool allow_buffer_handle_output
)

Set if buffer handle output is allowed.

When using hardware delegation, Interpreter will make the data of output tensors available in tensor->data by default. If the application can consume the buffer handle directly (e.g. reading output from OpenGL texture), it can set this flag to false, so Interpreter won't copy the data from buffer handle to CPU memory. WARNING: This is an experimental API and subject to change.

SetAllowFp16PrecisionForFp32

void SetAllowFp16PrecisionForFp32(
  bool allow
)

Allow float16 precision for FP32 calculation when possible.

default: not allow. WARNING: This is an experimental API and subject to change.

SetBufferHandle

TfLiteStatus SetBufferHandle(
  int tensor_index,
  TfLiteBufferHandle buffer_handle,
  TfLiteDelegate *delegate
)

Set the delegate buffer handle to a tensor.

It can be called in the following cases:

  1. Set the buffer handle to a tensor that's not being written by a delegate. For example, feeding an OpenGL texture as the input of the inference graph.
  2. Set the buffer handle to a tensor that uses the same delegate. For example, set an OpenGL texture as the output of inference, while the node which produces output is an OpenGL delegate node. WARNING: This is an experimental API and subject to change.

SetCancellationFunction

void SetCancellationFunction(
  void *data,
  bool(*)(void *) check_cancelled_func
)

Sets the cancellation function pointer in order to cancel a request in the middle of a call to Invoke().

The interpreter queries this function during inference, between op invocations; when it returns true, the interpreter will abort execution and return kTfLiteError. The data parameter contains any data used by the cancellation function, and if non-null, remains owned by the caller. WARNING: This is an experimental API and subject to change.

SetExecutionPlan

TfLiteStatus SetExecutionPlan(
  const std::vector< int > & new_plan
)

WARNING: Experimental interface, subject to change Overrides execution plan.

This bounds checks indices sent in.

SetExternalContext

void SetExternalContext(
  TfLiteExternalContextType type,
  TfLiteExternalContext *ctx
)

SetNumThreads

void SetNumThreads(
  int num_threads
)

Set the number of threads available to the interpreter.

SetProfiler

void SetProfiler(
  Profiler *profiler
)

Sets the profiler to tracing execution.

The caller retains ownership of the profiler and must ensure its validity. WARNING: This is an experimental API and subject to change.

UseNNAPI

void UseNNAPI(
  bool enable
)

Enable or disable the NN API (true to enable)

execution_plan

const std::vector< int > & execution_plan() const 

WARNING: Experimental interface, subject to change.

inputs

const std::vector< int > & inputs() const 

Read only access to list of inputs.

node_and_registration

const std::pair< TfLiteNode, TfLiteRegistration > * node_and_registration(
  int node_index
) const 

Get a pointer to an operation and registration data structure if in bounds.

nodes_size

size_t nodes_size() const 

Return the number of ops in the model.

operator=

Interpreter & operator=(
  const Interpreter &
)=delete

outputs

const std::vector< int > & outputs() const 

Read only access to list of outputs.

tensor

TfLiteTensor * tensor(
  int tensor_index
)

Get a mutable tensor data structure.

tensor

const TfLiteTensor * tensor(
  int tensor_index
) const 

Get an immutable tensor data structure.

tensors_size

size_t tensors_size() const 

Return the number of tensors in the model.

typed_input_tensor

T * typed_input_tensor(
  int index
)

Return a mutable pointer into the data of a given input tensor.

The given index must be between 0 and inputs().size().

typed_input_tensor

const T * typed_input_tensor(
  int index
) const 

Return an immutable pointer into the data of a given input tensor.

The given index must be between 0 and inputs().size().

typed_output_tensor

T * typed_output_tensor(
  int index
)

Return a mutable pointer into the data of a given output tensor.

The given index must be between 0 and outputs().size().

typed_output_tensor

const T * typed_output_tensor(
  int index
) const 

Return an immutable pointer into the data of a given output tensor.

The given index must be between 0 and outputs().size().

typed_tensor

T * typed_tensor(
  int tensor_index
)

Perform a checked cast to the appropriate tensor type (mutable pointer version).

typed_tensor

const T * typed_tensor(
  int tensor_index
) const 

Perform a checked cast to the appropriate tensor type (immutable pointer version).

variables

const std::vector< int > & variables() const 

Read only access to list of variable tensors.

~Interpreter

 ~Interpreter()