Index

optimus_dl.modules.data

DataPipeline

Bases: NamedTuple

DataPipeline(datasets, dataloader)

Parameters:

Name	Type	Description	Default
datasets	BaseNode		None
dataloader	BaseNode | Loader		None

Source code in optimus_dl/modules/data/__init__.py

class DataPipeline(NamedTuple):
    datasets: torchdata.nodes.BaseNode
    dataloader: torchdata.nodes.BaseNode | torchdata.nodes.Loader

EvalDataPipeline

Bases: NamedTuple

EvalDataPipeline(datasets, dataloader, eval_freq, eval_iterations, eval_guaranteed_same_batches, eval_checkpointing)

Parameters:

Name	Type	Description	Default
datasets	BaseNode		None
dataloader	BaseNode | Loader		None
eval_freq	int | None		None
eval_iterations	int | None		None
eval_guaranteed_same_batches	bool | None		None
eval_checkpointing	int | None		None

Source code in optimus_dl/modules/data/__init__.py

class EvalDataPipeline(NamedTuple):
    datasets: torchdata.nodes.BaseNode
    dataloader: torchdata.nodes.BaseNode | torchdata.nodes.Loader
    eval_freq: int | None
    eval_iterations: int | None
    eval_guaranteed_same_batches: bool | None
    eval_checkpointing: int | None

LoggingDataNode

Bases: BaseDataset

A simple node that logs reset calls and delegates to a source node. Useful for debugging data pipelines.

Source code in optimus_dl/modules/data/__init__.py

class LoggingDataNode(BaseDataset):
    """A simple node that logs reset calls and delegates to a source node. Useful for debugging data pipelines."""

    def __init__(self, name: str, source: torchdata.nodes.BaseNode):
        super().__init__(None)
        self.name = name
        self.source = source

    def reset(self, initial_state=None):
        logger.info(f"Resetting data node {self.name}, initial_state={initial_state}")
        super().reset(initial_state)
        self.source.reset(initial_state)

    def get_state(self) -> dict[str, Any]:
        return self.source.get_state()

    def next(self):
        return self.source.next()

    def __repr__(self):
        return f"LoggingDataNode(name={self.name}, source={repr(self.source)})"

Modules and Sub-packages

• config:
• datasets:
• presets:
• profiling: Context manager to set the active profiler for the current context.
• transforms: