# `TFLiteElixir.Tokenizer.FullTokenizer`
[🔗](https://github.com/cocoa-xu/tflite_elixir/blob/main/lib/tflite_elixir/Tokenizer/full_tokenizer.ex#L1)

Runs end-to-end tokenization.

Related link: [FullTokenizer.swift](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/ios/BertQACore/Models/Tokenizers/FullTokenizer.swift)

# `convert_to_id`

```elixir
@spec convert_to_id([String.t()], map()) :: {:ok, [integer()]} | {:error, String.t()}
```

Convert to ID in the vocab

# `tokenize`

```elixir
@spec tokenize(String.t(), boolean(), map()) :: [String.t()]
```

End-to-end tokenization.

---

*Consult [api-reference.md](api-reference.md) for complete listing*