# `TFLiteElixir.Tokenizer.BasicTokenizer`
[🔗](https://github.com/cocoa-xu/tflite_elixir/blob/main/lib/tflite_elixir/Tokenizer/basic_tokenizer.ex#L1)

Runs basic tokenization such as punctuation spliting, lower casing.

# `split_by_whitespace`

```elixir
@spec split_by_whitespace(binary()) :: [String.t()]
```

Split text on whitespace, dropping the runs between words.

# `tokenize`

```elixir
@spec tokenize(String.t(), boolean()) :: [String.t()]
```

Tokenizes a piece of text.

---

*Consult [api-reference.md](api-reference.md) for complete listing*