Index

optimus_dl.modules.data.datasets

Modules and Sub-packages

• base: Base dataset class for data sources.
• composite: Generate a unique seed for each rank based on the base seed, rank, and epoch.
• huggingface: Configuration for Hugging Face datasets.
• loop_dataset: Dataset that infinitely loops over an inner dataset.
• strategies:
• tokenized_dataset: Configuration for pre-tokenized sharded datasets.
• tokenized_flat_dataset: Configuration for flat tokenized datasets.
• txt_lines: Configuration for line-based text datasets.