config

optimus_dl.modules.tokenizer.config

BaseTokenizerConfig dataclass

Bases: RegistryConfig

BaseTokenizerConfig(_name: str | None = None, add_bos: bool = True, add_eos: bool = True)

Parameters:

Name	Type	Description	Default
add_bos	bool		True
add_eos	bool		True

Source code in optimus_dl/modules/tokenizer/config.py

@dataclass
class BaseTokenizerConfig(RegistryConfig):
    add_bos: bool = True
    add_eos: bool = True