Index

optimus_dl.modules.tokenizer.implementations

Modules and Sub-packages

• char: Configuration for character/byte-level tokenizer.
• huggingface: Configuration for Hugging Face tokenizers.
• inline_tokens: Configuration for explicitly specified tokens tokenizer.
• tiktoken: Configuration for Tiktoken tokenizers.