yamle.utils.optimization_utils module

yamle.utils.optimization_utils.get_optimizer(name, parameters, optimizer_config)

Return type:

Optimizer

yamle.utils.optimization_utils.get_scheduler(name, optimizer, scheduler_config)

Return type:

_LRScheduler

class yamle.utils.optimization_utils.ScalarScheduler

Bases: ABC

This is a general class for scalar schedulers.

abstract step()

This method is used to update the scheduler.

Return type:

None

abstract get_value()

This method is used to get the current value of the scheduler.

Return type:

float

abstract reset()

This method is used to reset the scheduler.

Return type:

None

state_dict()

This method is used to get the state of the scheduler.

Return type:

Dict[str, Any]

load_state_dict(state_dict)

This method is used to load the state of the scheduler.

Return type:

None

class yamle.utils.optimization_utils.LinearScalarScheduler(start_value, start_epoch, end_value, end_epoch)

Bases: ScalarScheduler

This class defines a linear scheduler for a scalar value.

Until the start epoch the return value will be start_value. After the start epoch the return value will be linearly increased until the end epoch. After the end epoch the return value will be the end_value.

Parameters:

• start_value (float) – The initial value of the scheduler.

• start_epoch (int) – The epoch to start the scheduler.

• end_value (float) – The final value of the scheduler.

• end_epoch (int) – The epoch to end the scheduler.

step()

This method is used to update the scheduler.

Return type:

None

set_hard_value(value)

This method is used to set a hard value for the scheduler, ignoring the schedule.

Return type:

None

get_value()

This method is used to get the current value of the scheduler.

Return type:

float

reset()

This method is used to reset the scheduler.

Return type:

None

state_dict()

This method is used to get the state of the scheduler.

Return type:

Dict[str, Any]

load_state_dict(state_dict)

This method is used to load the state of the scheduler.

Return type:

None

class yamle.utils.optimization_utils.PowerGrowthScalarScheduler(start_value, start_epoch, end_value, end_epoch, power=1.0)

Bases: ScalarScheduler

This class defines an exponential scheduler for a scalar value.

Until the start epoch the return value will be start_value. Given a power value,

Parameters:

• start_value (float) – The initial value of the scheduler.

• start_epoch (int) – The epoch to start the scheduler.

• end_value (float) – The final value of the scheduler.

• end_epoch (int) – The epoch to end the scheduler.

• gamma (float) – The exponential growth factor.

step()

This method is used to update the scheduler.

Return type:

None

set_hard_value(value)

This method is used to set a hard value for the scheduler, ignoring the schedule.

Return type:

None

get_value()

This method is used to get the current value of the scheduler.

Return type:

float

reset()

This method is used to reset the scheduler.

Return type:

None

state_dict()

This method is used to get the state of the scheduler.

Return type:

Dict[str, Any]

load_state_dict(state_dict)

This method is used to load the state of the scheduler.

Return type:

None

class yamle.utils.optimization_utils.SineScalarScheduler(start_value, start_epoch, end_value, end_epoch)

Bases: ScalarScheduler

This class defines a sine scheduler for a scalar value.

Until the start epoch the return value will be start_value. After the start epoch the return value will be increased until the end epoch. After the end epoch the return value will be the end_value.

Parameters:

• start_value (float) – The initial value of the scheduler.

• start_epoch (int) – The epoch to start the scheduler.

• end_value (float) – The final value of the scheduler.

• end_epoch (int) – The epoch to end the scheduler.

step()

This method is used to update the scheduler.

Return type:

None

set_hard_value(value)

This method is used to set a hard value for the scheduler, ignoring the schedule.

Return type:

None

get_value()

This method is used to get the current value of the scheduler.

Return type:

float

reset()

This method is used to reset the scheduler.

Return type:

None

state_dict()

This method is used to get the state of the scheduler.

Return type:

Dict[str, Any]

load_state_dict(state_dict)

This method is used to load the state of the scheduler.

Return type:

None

class yamle.utils.optimization_utils.CosineScalarScheduler(start_value, start_epoch, end_value, end_epoch)

Bases: ScalarScheduler

This class defines a cosine scheduler for a scalar value.

Until the start epoch the return value will be start_value. After the start epoch the return value will be increased until the end epoch. After the end epoch the return value will be the end_value.

Parameters:

• start_value (float) – The initial value of the scheduler.

• start_epoch (int) – The epoch to start the scheduler.

• end_value (float) – The final value of the scheduler.

• end_epoch (int) – The epoch to end the scheduler.

step()

This method is used to update the scheduler.

Return type:

None

set_hard_value(value)

This method is used to set a hard value for the scheduler, ignoring the schedule.

Return type:

None

get_value()

This method is used to get the current value of the scheduler.

Return type:

float

reset()

This method is used to reset the scheduler.

Return type:

None

state_dict()

This method is used to get the state of the scheduler.

Return type:

Dict[str, Any]

load_state_dict(state_dict)

This method is used to load the state of the scheduler.

Return type:

None

yamle.utils.optimization_utils.recover_frozen_weights(model)

This function is used to recover frozen weights after an optimization step.

The parameters that do have a FROZEN_MASK_KEY and FROZEN_DATA_KEY attribute will be recovered. The FROZEN_MASK_KEY is assumed to be a 1D tensor with the same number of elements as the parameter. The FROZEN_DATA_KEY has the same shape as the parameter. The FROZEN_MASK_KEY is used to select the elements that will be recovered. It should only contain True or False values. The True values are the elements that will be recovered from the FROZEN_DATA_KEY.

Parameters:

model (nn.Module) – The model to recover the frozen weights.

Return type:

None

yamle.utils.optimization_utils.freeze_weights(parameters, masks=None)

This function is used to freeze weights of a model.

The masks are used to select the weights that will be frozen. The masks should have the same number of elements as the parameters. The masks should only contain True or False values. The True values are the elements that will be frozen. If no mask is provided, all the weights will be frozen.

Parameters:

• parameters (Union[nn.Parameter, List[nn.Parameter]]) – The parameters to freeze.

• masks (Optional[Union[torch.Tensor, List[torch.Tensor]]], optional) – The masks to select the weights to freeze. Defaults to None.

Return type:

None

yamle.utils.optimization_utils.split_optimizer_parameters(parameters)

This function is used to split the parameters of a model into multiple dictionaries depending on an id.

Given all model parameters, this function looks at if OPTIMIZER_ID_KEY is set as an attribute of the parameter. If it is set it will add the parameter to the dictionary with the corresponding id. If the OPTIMIZER_ID_KEY is not assigned it is assumed that the parameter is used by the first optimizer.

Return type:

List[Dict[str, Union[List[Parameter], List[Tuple[str, Parameter]]]]]

Returns:

A list of dictionaries with the parameters split.

yamle.utils.optimization_utils.set_optimizer_id(parameters, optimizer_id)

This function is used to set the OPTIMIZER_ID_KEY attribute to the parameters.

Parameters:

• parameters (List[nn.Parameter]) – The parameters to set the optimizer id.

• optimizer_id (int) – The optimizer id to set.

Return type:

None

yamle.utils.optimization_utils.disable_optimization(parameters)

This function is used to disable the optimization of the parameters.

Parameters:

parameters (List[nn.Parameter]) – The parameters to disable the optimization.

Return type:

None

yamle.utils.optimization_utils.enable_optimization(parameters)

This function is used to enable the optimization of the parameters.

Parameters:

parameters (List[nn.Parameter]) – The parameters to enable the optimization.

Return type:

None