# `TFLiteElixir.LiteRT.CompiledModel`
[🔗](https://github.com/cocoa-xu/tflite_elixir/blob/main/lib/tflite_elixir/litert/compiled_model.ex#L1)

A model compiled through LiteRT, which is where the accelerators and the
per-operator profile live.

This is a different path into the same runtime as `TFLiteElixir.Interpreter`,
not a replacement for it. What it adds is a choice of accelerator that is
asked for by name and answered honestly, and a profiler:

    {:ok, env} = TFLiteElixir.LiteRT.CompiledModel.environment()
    {:ok, model} = TFLiteElixir.LiteRT.CompiledModel.new(env, path,
                     accelerators: [:cpu, :gpu], profile: true)
    {:ok, outputs} = TFLiteElixir.LiteRT.CompiledModel.run(model, inputs)
    {:ok, slowest} = TFLiteElixir.LiteRT.CompiledModel.summarise_profile(model)

`fully_accelerated?/1` says whether the accelerator took the whole graph or
only part of it, which is the difference between a speedup and a slowdown and
is not otherwise visible.

## One caller at a time

LiteRT does not promise its compiled model API is safe to use from several
threads, and the profile buffer under it says outright that it is not. So a
second concurrent caller is refused here rather than allowed to corrupt
anything, and `{:error, "compiled model is in use by another caller"}` is a
normal answer rather than a fault. `TFLiteElixir.LiteRT.CompiledModel.Server`
is the way to share one model between processes.

## Availability

The LiteRT API is a build option and is off by default, so every function here
can answer `{:error, "tflite_beam was compiled without the LiteRT API"}` on an
ordinary build. `platform_support/0` says what this build can reach.

# `accelerator`

```elixir
@type accelerator() :: :cpu | :gpu | :npu
```

What to run on, in order of preference.

# `event`

```elixir
@type event() :: %{
  tag: binary(),
  us: non_neg_integer(),
  type: atom() | integer(),
  source: atom() | integer()
}
```

A profiling event. An integer type or source is one this build has no name for.

# `metric_value`

```elixir
@type metric_value() :: integer() | float() | boolean() | binary() | :unsupported
```

# `opts`

```elixir
@type opts() :: [
  accelerators: [accelerator()],
  precision: precision(),
  profile: boolean(),
  signature: non_neg_integer() | String.t(),
  max_model_bytes: non_neg_integer()
]
```

# `precision`

```elixir
@type precision() :: :default | :fp16 | :fp32 | :fp16_with_fp32_accum
```

Compute precision. `:default` leaves it to the accelerator, which for Metal
means fp32; `:fp16` trades accuracy for speed.

# `summary_entry`

```elixir
@type summary_entry() :: %{
  tag: binary(),
  kind: :operator | :delegate_operator | :delegate_profiled,
  count: pos_integer(),
  us: non_neg_integer()
}
```

# `available?`

```elixir
@spec available?() :: boolean()
```

Whether this build has the LiteRT API at all.

It is a build option and it is off by default, so on an ordinary build every
other function here answers `{:error, "the LiteRT API was not compiled into
this build..."}`. Asking this first is cheaper than finding out from a call
that was meant to do something.

# `controlling_process`

```elixir
@spec controlling_process(reference()) ::
  {:ok, pid()} | :undefined | {:error, String.t()}
```

Which process the model belongs to, or `:undefined` if it is unclaimed.

An unclaimed model is open to every process. Claiming one is `controlling_process/2`.

# `controlling_process`

```elixir
@spec controlling_process(reference(), pid()) :: :ok | {:error, String.t()}
```

Hand the model to a process, after which no other process may use it.

The claim is dropped when that process dies, so a crash does not strand the
model.

# `controlling_process!`

Raising version of `controlling_process/1`.

# `controlling_process!`

Raising version of `controlling_process/2`.

# `environment`

```elixir
@spec environment() :: {:ok, reference()} | {:error, String.t()}
```

A LiteRT environment, which the accelerator plugins are loaded into.

One is enough for any number of models and it has to outlive them.

# `environment`

```elixir
@spec environment(String.t()) :: {:ok, reference()} | {:error, String.t()}
```

An environment that looks in `runtime_library_dir` for accelerator plugins.

Without a directory the plugins are searched for relative to nothing, which is
the usual reason a GPU accelerator silently does not load.

# `environment!`

Raising version of `environment/0`.

# `environment!`

Raising version of `environment/1`.

# `fully_accelerated`

```elixir
@spec fully_accelerated(reference()) :: {:ok, boolean()} | {:error, String.t()}
```

As `fully_accelerated?/1`, but reports why it could not be answered.

# `fully_accelerated!`

Raising version of `fully_accelerated/1`.

# `fully_accelerated?`

```elixir
@spec fully_accelerated?(reference()) :: boolean()
```

Whether the accelerator took the whole graph.

A partly accelerated model pays for every crossing between the accelerator and
the CPU, and is often slower than the CPU alone, so a `false` here is worth
acting on rather than ignoring.

# `io_sizes`

```elixir
@spec io_sizes(reference()) ::
  {:ok, {[non_neg_integer()], [non_neg_integer()]}} | {:error, String.t()}
```

The byte size of each input and output tensor, as `{inputs, outputs}`.

# `io_sizes!`

Raising version of `io_sizes/1`.

# `new`

```elixir
@spec new(reference(), String.t()) :: {:ok, reference()} | {:error, String.t()}
```

Compile a model with the default options: CPU, no profiling.

# `new`

```elixir
@spec new(reference(), String.t(), opts()) ::
  {:ok, reference()} | {:error, String.t()}
```

Compile a model.

##### Options
- `:accelerators`. What to run on, in order of preference, e.g. `[:cpu, :gpu]`.
  Naming an accelerator asks for it; whether it was used is
  `fully_accelerated?/1`. Defaults to `[:cpu]`.
- `:precision`. `:default`, `:fp16`, `:fp32` or `:fp16_with_fp32_accum`.
- `:profile`. Record per-operator timings, readable with `profile/1` and
  `summarise_profile/1`. Off by default because it is not free.
- `:signature`. Which signature to compile for, by index or by name.
- `:max_model_bytes`. Refuse a model file larger than this.

# `new!`

Raising version of `new/2`.

# `new!`

Raising version of `new/3`.

# `pending_events`

```elixir
@spec pending_events(reference()) :: {:ok, non_neg_integer()} | {:error, String.t()}
```

How many profiling events are waiting, without reading them.

Zero for a model compiled without `profile: true`. This is what sizes the copy
`profile/2` has to make, so it is the number to look at before calling it on a
memory-constrained target.

# `pending_events!`

Raising version of `pending_events/1`.

# `platform_support`

```elixir
@spec platform_support() ::
  {:ok, %{required(atom()) => boolean()}} | {:error, String.t()}
```

Which buffer kinds this platform can reach, e.g. `%{metal: true, opencl: false}`.

Answers `{:error, reason}` on a build without the LiteRT API; `available?/0`
is the question to ask first.

# `profile`

```elixir
@spec profile(reference()) :: {:ok, [event()]} | {:error, String.t()}
```

Every profiling event recorded so far, oldest first.

# `profile`

```elixir
@spec profile(reference(), non_neg_integer()) ::
  {:ok, [event()]} | {:error, String.t()}
```

The most recent `limit` profiling events, or all of them when `limit` is zero.

`limit` bounds the events returned, not the reading: LiteRT will not hand over
part of a backlog, so every call copies whatever `pending_events/1` reports.
That is about 109 MiB for a full buffer, which is nothing on a workstation and
fatal on a board with 256 MB.

# `profile!`

Raising version of `profile/1`.

# `profile!`

Raising version of `profile/2`.

# `reset_profile`

```elixir
@spec reset_profile(reference()) :: :ok | {:error, String.t()}
```

Forget the events recorded so far and keep recording.

# `reset_profile!`

Raising version of `reset_profile/1`.

# `run`

```elixir
@spec run(reference(), [binary()]) :: {:ok, [binary()]} | {:error, String.t()}
```

Run the model over a list of input binaries, one per input tensor.

The sizes each input has to be are `io_sizes/1`, and a wrong size is refused
rather than read past.

# `run!`

Raising version of `run/2`.

# `run_with_metrics`

```elixir
@spec run_with_metrics(reference(), [binary()]) ::
  {:ok, {[binary()], [{binary(), metric_value()}]}} | {:error, String.t()}
```

Run the model and collect whatever counters the accelerator reports.

# `run_with_metrics`

```elixir
@spec run_with_metrics(reference(), [binary()], non_neg_integer()) ::
  {:ok, {[binary()], [{binary(), metric_value()}]}} | {:error, String.t()}
```

Run with metrics collection bracketing the inference.

Usually the counters come back empty. Filling them in is the accelerator's
job, through two entries of its definition that are allowed to be null, so an
empty list means nobody offered anything rather than that something went
wrong. Use `profile/1` for timings; this is for counters a backend chooses to
expose.

# `run_with_metrics!`

Raising version of `run_with_metrics/2`.

# `run_with_metrics!`

Raising version of `run_with_metrics/3`.

# `signatures`

```elixir
@spec signatures(reference(), String.t()) :: {:ok, [binary()]} | {:error, String.t()}
```

The names of a model file's signatures, without compiling it.

A model with no named signature reports the one default signature LiteRT gives
it, so the list is never empty.

# `signatures!`

Raising version of `signatures/2`.

# `summarise_profile`

```elixir
@spec summarise_profile(reference()) ::
  {:ok, [summary_entry()]} | {:error, String.t()}
```

Per-operator totals over every run since the last reset, slowest first.

Only operator events are folded in. The enclosing `Invoke`, tensor allocation
and LiteRT's own buffer handling are events too, and adding them together
would count the operators twice, so they stay in `profile/1` alone.

# `summarise_profile!`

Raising version of `summarise_profile/1`.

---

*Consult [api-reference.md](api-reference.md) for complete listing*