Permanences
BaseLoggerManager
¶
Bases: Permanence
Common logger interface for all logging backends.
checkpoint()
¶
Save intermediate state during pipeline execution.
Called at configurable checkpoints during execution. Use for saving progress, creating backups, or logging state.
Raises:
| Type | Description |
|---|---|
Exception
|
If checkpointing fails |
get_state()
¶
Get serializable state for inspection or debugging.
Returns a dictionary representation of the permanence state. Useful for logging, debugging, or state inspection.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: Dictionary containing permanence state. |
initialize()
¶
Initialize logger backend via the permanence lifecycle.
log_figure(name, figure)
abstractmethod
¶
Log a pre-built figure object for the active backend.
log_metrics(metrics)
abstractmethod
¶
Log scalar or structured metrics for the active backend.
log_seaborn_graph(name, data, x=None, y=None, kind='line')
¶
Create a seaborn graph and route it through log_figure.
run_sweep(run_once, sweep_config)
¶
Execute backend-specific sweep flow.
Called only if supports_sweep() returns True.
supports_sweep()
¶
Whether this logger backend supports hyperparameter sweeps.
validate()
¶
Validate the permanence state.
Called to verify permanence is in valid state. Use for health checks, data validation, or consistency checks.
Raises:
| Type | Description |
|---|---|
Exception
|
If validation fails |
BasicLogger
¶
Bases: BaseLoggerManager
Filesystem logger.
- debug/warning/error -> terminal
- metrics -> JSONL file in log_dir
- figures -> image files in log_dir
checkpoint()
¶
Save intermediate state during pipeline execution.
Called at configurable checkpoints during execution. Use for saving progress, creating backups, or logging state.
Raises:
| Type | Description |
|---|---|
Exception
|
If checkpointing fails |
get_state()
¶
Get serializable state for inspection or debugging.
Returns a dictionary representation of the permanence state. Useful for logging, debugging, or state inspection.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: Dictionary containing permanence state. |
initialize()
¶
Initialize logger backend via the permanence lifecycle.
log_seaborn_graph(name, data, x=None, y=None, kind='line')
¶
Create a seaborn graph and route it through log_figure.
run_sweep(run_once, sweep_config)
¶
Execute backend-specific sweep flow.
Called only if supports_sweep() returns True.
supports_sweep()
¶
Whether this logger backend supports hyperparameter sweeps.
validate()
¶
Validate the permanence state.
Called to verify permanence is in valid state. Use for health checks, data validation, or consistency checks.
Raises:
| Type | Description |
|---|---|
Exception
|
If validation fails |
Device
¶
Bases: Permanence
Device class for managing CUDA device selection based on VRAM usage.
This class inherits from the Permanence class and is responsible for calculating and setting the best available CUDA device for computation based on VRAM usage.
Example TOML Config
[permanences.network]
type = "Network"
params = { model = "deeplabv3_resnet50", num_classes = 21, pretrained = true }
Methods:
| Name | Description |
|---|---|
__init__ |
|
_calculate_best_device |
Raises VRAMUsageError if the VRAM usage of the best device exceeds 80%. |
cleanup |
Cleans up the device instance. This method doesn't perform any cleanup operations. |
__init__()
¶
Initializes the instance and calculates the best device for computation.
checkpoint()
¶
Save intermediate state during pipeline execution.
Called at configurable checkpoints during execution. Use for saving progress, creating backups, or logging state.
Raises:
| Type | Description |
|---|---|
Exception
|
If checkpointing fails |
cleanup()
¶
Cleans up the device instance.
This method doesn't perform any cleanup operations for the device instance.
Returns:
| Name | Type | Description |
|---|---|---|
None |
None
|
This method doesn't return any value. |
get_state()
¶
Get serializable state for inspection or debugging.
Returns a dictionary representation of the permanence state. Useful for logging, debugging, or state inspection.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: Dictionary containing permanence state. |
initialize()
¶
Initialize the permanence before pipeline execution.
Called once after all permanences are constructed but before any process runs. Use for validation, resource allocation, or setup that depends on other permanences.
Raises:
| Type | Description |
|---|---|
Exception
|
If initialization fails |
is_initialized()
¶
Check if the device is initialized.
Overrides the base method to provide logic for determining if the device is ready for use.
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True if the device is initialized, False otherwise. |
validate()
¶
Validate the permanence state.
Called to verify permanence is in valid state. Use for health checks, data validation, or consistency checks.
Raises:
| Type | Description |
|---|---|
Exception
|
If validation fails |
NullProgressManager
¶
Bases: Permanence
A null progress manager that does not perform any progress tracking.
This class is used if a manager does not provide a progress manager.
checkpoint()
¶
Save intermediate state during pipeline execution.
Called at configurable checkpoints during execution. Use for saving progress, creating backups, or logging state.
Raises:
| Type | Description |
|---|---|
Exception
|
If checkpointing fails |
cleanup()
abstractmethod
¶
Cleans up data from RAM or VRAM.
Called after all processes complete or on error. Should release any held resources (memory, file handles, connections).
Raises:
| Type | Description |
|---|---|
Exception
|
If cleanup fails |
get_state()
¶
Get serializable state for inspection or debugging.
Returns a dictionary representation of the permanence state. Useful for logging, debugging, or state inspection.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: Dictionary containing permanence state. |
initialize()
¶
Initialize the permanence before pipeline execution.
Called once after all permanences are constructed but before any process runs. Use for validation, resource allocation, or setup that depends on other permanences.
Raises:
| Type | Description |
|---|---|
Exception
|
If initialization fails |
is_initialized()
¶
Check if the permanence has been initialized.
Override to provide custom logic for determining initialization state.
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True if initialized, False otherwise. |
validate()
¶
Validate the permanence state.
Called to verify permanence is in valid state. Use for health checks, data validation, or consistency checks.
Raises:
| Type | Description |
|---|---|
Exception
|
If validation fails |
ProgressManager
¶
Bases: Permanence
Manages progress tracking for tasks using the Rich library.
Example TOML Config
[permanences.progress_manager]
type = "ProgressManager"
param
Attributes:
| Name | Type | Description |
|---|---|---|
progress_dict |
dict
|
A dictionary to store progress objects. |
live |
Live
|
A Live object to manage live updates. |
Methods:
| Name | Description |
|---|---|
__init__ |
Initializes the ProgressManager with an optional direct live update. |
_create_progress |
Creates and returns a Progress object with specified color. |
_init_live |
Initializes the live update group with the progress objects. |
progress_task |
A decorator to wrap functions for progress tracking. |
cleanup |
not used |
__init__(console=None, direct=False)
¶
Initializes the instance of the class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
direct
|
bool
|
If True, initializes live progress. Defaults to False. |
False
|
add_progress(name, with_status=False)
¶
Add a progress object to the progress_dict.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
Name key under which the progress object is stored. |
required |
with_status
|
bool
|
Whether to include a status column. Defaults to False. Bar color is assigned automatically via the centralized palette logic. |
False
|
Returns:
| Name | Type | Description |
|---|---|---|
Progress |
None
|
A Progress object configured with the specified color. |
add_progresses(progresses)
¶
Add multiple progress objects to the progress_dict. Also re-initializes the live attribute.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
progresses
|
list[dict[str, Any]]
|
Progress definitions to add. |
required |
add_task_to_progress(task_description, total, visible=False, progress_name=None)
¶
Add a task to the specified progress object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task_description
|
str
|
Description of the task to be added.
When |
required |
total
|
int
|
The total number of steps for the task. |
required |
visible
|
bool
|
Define if the task is visible at creation. DEFAULT: False |
False
|
progress_name
|
str | None
|
Explicit bar name. When provided the bar is looked up directly instead of using fuzzy matching. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
The task_id of the added task. |
checkpoint()
¶
Save intermediate state during pipeline execution.
Called at configurable checkpoints during execution. Use for saving progress, creating backups, or logging state.
Raises:
| Type | Description |
|---|---|
Exception
|
If checkpointing fails |
get_state()
¶
Get serializable state for inspection or debugging.
Returns a dictionary representation of the permanence state. Useful for logging, debugging, or state inspection.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: Dictionary containing permanence state. |
init_live()
¶
Initializes the live attribute with a Live object. This method creates a Group object using the values from the progress_dict attribute and then initializes the live attribute with a Live object that takes the Group object as an argument.
initialize()
¶
Initialize the permanence before pipeline execution.
Called once after all permanences are constructed but before any process runs. Use for validation, resource allocation, or setup that depends on other permanences.
Raises:
| Type | Description |
|---|---|
Exception
|
If initialization fails |
is_initialized()
¶
Check if the permanence has been initialized.
Override to provide custom logic for determining initialization state.
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True if initialized, False otherwise. |
progress_task(task_name, visible=True)
¶
Decorator for wrapping functions with progress tracking.
The decorated function can receive any of these parameters: - total: The value passed when calling the decorated function - task_id: Injected by decorator, use with progress.advance(task_id) - progress: Injected by decorator, the Progress object for manual control
The decorator inspects the function signature and only passes parameters it accepts.
Usage examples
Function that uses total¶
@progress_manager.progress_task("processing") def process_items(total, task_id, progress): for i in range(total): progress.advance(task_id)
Function that doesn't need total¶
@progress_manager.progress_task("cleanup") def cleanup_all(task_id, progress): for item in items: # Has its own iteration progress.advance(task_id)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task_name
|
str
|
Name of the task/progress bar to use |
required |
visible
|
bool
|
Whether task should remain visible after completion |
True
|
Returns:
| Type | Description |
|---|---|
Callable[..., Callable[..., Any]]
|
Decorator function that wraps the target function with progress tracking |
update(progress_name, task_id, status='', advance=False)
¶
Update a task's status text, optionally advancing the bar by one step.
Unlike advance(), this method defaults to a pure status update (no movement).
Pass advance=True to move the bar forward one step at the same time.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
progress_name
|
str
|
Key of the progress bar in |
required |
task_id
|
int
|
Task identifier returned by |
required |
status
|
str
|
Text to display in the status column. |
''
|
advance
|
bool
|
When |
False
|
validate()
¶
Validate the permanence state.
Called to verify permanence is in valid state. Use for health checks, data validation, or consistency checks.
Raises:
| Type | Description |
|---|---|
Exception
|
If validation fails |
TensorBoardLogger
¶
Bases: BaseLoggerManager
TensorBoard-backed logger with the same interface as BasicLogger.
checkpoint()
¶
Save intermediate state during pipeline execution.
Called at configurable checkpoints during execution. Use for saving progress, creating backups, or logging state.
Raises:
| Type | Description |
|---|---|
Exception
|
If checkpointing fails |
get_state()
¶
Get serializable state for inspection or debugging.
Returns a dictionary representation of the permanence state. Useful for logging, debugging, or state inspection.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: Dictionary containing permanence state. |
log_seaborn_graph(name, data, x=None, y=None, kind='line')
¶
Create a seaborn graph and route it through log_figure.
run_sweep(run_once, sweep_config)
¶
Execute backend-specific sweep flow.
Called only if supports_sweep() returns True.
supports_sweep()
¶
Whether this logger backend supports hyperparameter sweeps.
validate()
¶
Validate the permanence state.
Called to verify permanence is in valid state. Use for health checks, data validation, or consistency checks.
Raises:
| Type | Description |
|---|---|
Exception
|
If validation fails |
WandBLogger
¶
Bases: BaseLoggerManager
Weights & Biases logger backend.
checkpoint()
¶
Save intermediate state during pipeline execution.
Called at configurable checkpoints during execution. Use for saving progress, creating backups, or logging state.
Raises:
| Type | Description |
|---|---|
Exception
|
If checkpointing fails |
get_state()
¶
Get serializable state for inspection or debugging.
Returns a dictionary representation of the permanence state. Useful for logging, debugging, or state inspection.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: Dictionary containing permanence state. |
log_seaborn_graph(name, data, x=None, y=None, kind='line')
¶
Create a seaborn graph and route it through log_figure.
validate()
¶
Validate the permanence state.
Called to verify permanence is in valid state. Use for health checks, data validation, or consistency checks.
Raises:
| Type | Description |
|---|---|
Exception
|
If validation fails |