# `TFLiteElixir.Tokenizer.WordpieceTokenizer`
[🔗](https://github.com/cocoa-xu/tflite_elixir/blob/main/lib/tflite_elixir/Tokenizer/wordpiece_tokenizer.ex#L1)

Runs WordPiece tokenziation.

# `tokenize`

```elixir
@spec tokenize(String.t(), map()) :: [String.t()]
```

Tokenizes a piece of text into its word pieces.

This uses a greedy longest-match-first algorithm to perform tokenization using the given
vocabulary.

For example:

```
input = "unaffable".
output = ["una", "##ffa", "##ble"].
```

```
input = "unaffableX".
output = ["[UNK]"].
```

Related link: https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/ios/BertQACore/Models/Tokenizers/WordpieceTokenizer.swift

---

*Consult [api-reference.md](api-reference.md) for complete listing*