yamle.regularizers.weight module

class yamle.regularizers.weight.L1Regularizer

Bases: BaseRegularizer

This is a class for L1 regularization.

class yamle.regularizers.weight.L2Regularizer

Bases: BaseRegularizer

This is a class for L2 regularization.

class yamle.regularizers.weight.L1L2Regularizer

Bases: BaseRegularizer

This is a class for combined L1 and L2 regularization.

class yamle.regularizers.weight.WeightDecayRegularizer

Bases: BaseRegularizer

This is a class for weight decay regularization.

It is implemented in an inefficient manner to be compatible with any optimizer.

During the __call__ method, the weights at time t are cached. Then, during the update_on_step method, the weights, which were already updated by the optimizer, are further updated by weight decay.

The weight decay is applied as follows:

w_{t+1} = (1 - weight) * w_{t} - eta * ∇L(w_{t+1})

Hence, after the optimization step, assuming that only w_{t+1} = w_{t} - \eta * ∇L(w_{t+1}) was applied, we need to apply the -weight * w_{t} term. The weight is scaled by the learning rate.

on_after_training_step(model, weight, lr, *args, **kwargs)

This method is used to update the model on a given step.

Return type:

None