yamle.methods.mimo module#

class yamle.methods.mimo.MIMOMethod(initialise_encoder_members_same=False, num_batch_repetitions=1, input_repetition_probability=None, repeat_evaluation=True, *args, **kwargs)[source]#

Bases: MemberMethod

This class is the extension of the base method for MIMO methods.

The difference is in having to change the prediction to concatenate the num_members dimension. into the first feature dimension.

Parameters:

  • initialise_encoder_members_same (bool) – Whether to initialise the members in the encoder with the same weights.

  • num_batch_repetitions (int) – The number of times some samples are repeated in the batch.

  • input_repetition_probability (Optional[float]) – The probability that the inputs are identical for the ensemble members.

  • repeat_evaluation (bool) – Whether to repeat samples in the evaluation.

on_train_epoch_end()[source]#

This method is called at the end of the training epoch.

Return type:

None

state_dict()[source]#

This method returns the state dict of the MIMO method.

Return type:

Dict[str, Any]

load_state_dict(state_dict)[source]#

This method loads the state dict of the MIMO method.

Return type:

None

static add_specific_args(parent_parser)[source]#

This method adds the specific arguments for the MIMO method.

Return type:

ArgumentParser

test_name: Optional[str]#
prepare_data_per_node: bool#
allow_zero_length_dataloader_with_multiple_devices: bool#
training: bool#
class yamle.methods.mimo.MIMMOMethod(alpha=1.0, prior='uniform', do_not_optimize_depth_weights=False, additional_heads=False, available_heads=None, warm_starting_epochs=0, *args, **kwargs)[source]#

Bases: MIMOMethod

This class is the extension of the MIMO method in which we will try to find the depth for each ensemble member.

Parameters:

  • alpha (float) – The alpha value to regularize the depth loss term.

  • prior (str) – The prior to use for the depth weights.

  • do_not_optimize_depth_weights (bool) – Whether to optimize the depth weights or not.

  • additional_heads (bool) – Whether to enable specific heads for each layer.

  • available_heads (List[bool]) – Toggles to hard enable or disable heads.

  • warm_starting_epochs (int) – The number of epochs to train the model without changing the member or depth weights.

on_train_epoch_end()[source]#

A helper method to log the weights of the ensemble.

Return type:

None

state_dict()[source]#

A helper method to save the weights of the ensemble.

Return type:

Dict[str, Any]

load_state_dict(state_dict)[source]#

A helper method to load the weights of the ensemble.

Return type:

None

static add_specific_args(parent_parser)[source]#

The method to add additional arguments.

Return type:

ArgumentParser

test_name: Optional[str]#
prepare_data_per_node: bool#
allow_zero_length_dataloader_with_multiple_devices: bool#
training: bool#
class yamle.methods.mimo.MixMoMethod(alpha=0.5, r=0.5, initial_p=0.5, *args, **kwargs)[source]#

Bases: MIMOMethod

This is a module which applies the MixMo regularization to a model.

As proposed in: “MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks”

Show that binary mixing in features - particularly with rectangular patches from CutMix - enhances results by making subnetworks stronger and more diverse.

Parameters:

  • alpha (float) – The alpha parameter for the Dirichlet distribution.

  • r (float) – The r parameter applied to the weighting factor.

  • initial_p (float) – Initial probability of applying linear or cutmix augmentation.

on_train_epoch_start()[source]#

A method which is called at the start of the training epoch.

Return type:

None

static add_specific_args(parent_parser)[source]#

This method adds the specific arguments for the MixMo method.

Return type:

ArgumentParser

test_name: Optional[str]#
prepare_data_per_node: bool#
allow_zero_length_dataloader_with_multiple_devices: bool#
training: bool#
class yamle.methods.mimo.MixVitMethod(depth, *args, **kwargs)[source]#

Bases: MIMOMethod

This is a module which applies MixToken augmentation to a vision transformer.

Parameters:

depth (int) – The depth at which to add the source attribution.

static add_specific_args(parent_parser)[source]#

This method adds the specific arguments for the MixMo method.

Return type:

ArgumentParser

test_name: Optional[str]#
prepare_data_per_node: bool#
allow_zero_length_dataloader_with_multiple_devices: bool#
training: bool#
class yamle.methods.mimo.UnMixMoMethod(m_start_value=0, m_end_value=1, m_start_epoch=0, m_end_epoch=100, *args, **kwargs)[source]#

Bases: MixMoMethod

This is a module which applies the unmixing regularization to the model in addition to MixMo.

The key concept is that instead of processing the inputs just through a convolutional or a linear layer, each input is mixed and then unmixed selectively depending on the mixing mask.

Precisely it focuses on fadeout unmixing which has demonstrated to be more effective than full unmixing. The fadeout unmixing is a gradual unmixing of the inputs.

Parameters:

  • m_start_value (Optional[float]) – The initial value for the m parameter. Defaults to 0.

  • m_end_value (Optional[float]) – The final value for the m parameter. Defaults to 1.

  • m_start_epoch (Optional[int]) – The epoch at which the m parameter starts to increase. Defaults to 0.

  • m_end_epoch (Optional[int]) – The epoch at which the m parameter reaches its final value. Defaults to 100.

test_name: Optional[str]#
prepare_data_per_node: bool#
allow_zero_length_dataloader_with_multiple_devices: bool#
training: bool#
on_train_epoch_start()[source]#

A method which is called at the start of the training epoch.

Return type:

None

on_train_epoch_end()[source]#

A method which is called at the end of the training epoch.

Return type:

None

state_dict()[source]#

This method returns the state dict of the MIMO method.

Return type:

Dict[str, Any]

load_state_dict(state_dict)[source]#

This method loads the state dict of the MIMO method.

Return type:

None

static add_specific_args(parent_parser)[source]#

This method adds the specific arguments for the UnMixMo method.

Return type:

ArgumentParser

class yamle.methods.mimo.DataMUXMethod(coder_expansion_factor=1, coder_depth=1, *args, **kwargs)[source]#

Bases: MIMOMethod

This is a module which applies the DataMUX multiplexing and demultiplexing to the input and output of the model.

As proposed in: DataMUX: Data Multiplexing for Neural Networks.

The key concept is that instead of processing the inputs just through a convolutional or a linear layer, each input has a dedicated encoder and the output is processed through a separate decoder to give predictions.

Parameters:

  • coder_expansion_factor (int) – The expansion factor for the coder. Defaults to 1.

  • coder_depth (int) – The depth of the coder. Defaults to 1.

test_name: Optional[str]#
prepare_data_per_node: bool#
allow_zero_length_dataloader_with_multiple_devices: bool#
training: bool#
static add_specific_args(parent_parser)[source]#

Adds the specific arguments for the DataMUXMethod.

Return type:

ArgumentParser