TFLiteElixir.Tokenizer.FullTokenizer (tflite_elixir v1.0.1)

Copy Markdown View Source

Runs end-to-end tokenization.

Related link: FullTokenizer.swift

Summary

Functions

convert_to_id(tokens, vocab)

Convert to ID in the vocab

tokenize(text, is_case_insensitive, vocab)

End-to-end tokenization.

Functions

convert_to_id(tokens, vocab)

@spec convert_to_id([String.t()], map()) :: {:ok, [integer()]} | {:error, String.t()}

Convert to ID in the vocab

tokenize(text, is_case_insensitive, vocab)

@spec tokenize(String.t(), boolean(), map()) :: [String.t()]

End-to-end tokenization.