tfaip.trainer.scheduler

Weight decay

Definition of a WeightDecaySchedule

class tfaip.trainer.scheduler.schedule_weightdecay.WeightDecaySchedule(weight_decay: float, learning_rate_schedule: tensorflow.python.keras.optimizer_v2.learning_rate_schedule.LearningRateSchedule, name=None, **kwargs)

Bases: tensorflow.python.keras.optimizer_v2.learning_rate_schedule.LearningRateSchedule

The WeightDecaySchedule computes the weight decay multiplied by a rate

Wrap the LR schedule and multiply by weight decay. Take care of base LR and modify weight decay accordingly

__init__(weight_decay: float, learning_rate_schedule: tensorflow.python.keras.optimizer_v2.learning_rate_schedule.LearningRateSchedule, name=None, **kwargs)

Initialize self. See help(type(self)) for accurate signature.

get_config()

__call__(step)

Call self as a function.

Learning Rate

Definition of the LearningRateParams

class tfaip.trainer.scheduler.learningrate_params.LearningRateParams(lr: float = 0.001, step_function: bool = True, offset_epochs: int = 0, steps_per_epoch: int = - 1, epochs: int = - 1)

Bases: abc.ABC

Base definition of learning rate params

abstract static cls()

create()

lr: float = 0.001

step_function: bool = True

offset_epochs: int = 0

steps_per_epoch: int = -1

epochs: int = -1

__init__(lr: float = 0.001, step_function: bool = True, offset_epochs: int = 0, steps_per_epoch: int = - 1, epochs: int = - 1) → None

Initialize self. See help(type(self)) for accurate signature.

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → A

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → A

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[A]

to_dict(encode_json=False) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str

Definition of the base LearningRateSchedule

class tfaip.trainer.scheduler.learningrate.LearningRateSchedule(params: LearningRateParams, name=None, **kwargs)

Bases: tensorflow.python.keras.optimizer_v2.learning_rate_schedule.LearningRateSchedule, abc.ABC

Base class for learning rate schedules.

A custom implementation must overwrite lr(epoch)

__init__(params: LearningRateParams, name=None, **kwargs)

Initialize self. See help(type(self)) for accurate signature.

get_config()

classmethod from_config(config)

Instantiates a LearningRateSchedule from its config.

Parameters

config – Output of get_config().

Returns

A LearningRateSchedule instance.

__call__(step)

Call self as a function.

abstract lr(epoch) → float

Constant

Definition of the Constant learning rate schedule

class tfaip.trainer.scheduler.constant_params.Constant(lr: float = 0.001, step_function: bool = True, offset_epochs: int = 0, steps_per_epoch: int = - 1, epochs: int = - 1)

Bases: tfaip.trainer.scheduler.learningrate_params.LearningRateParams

static cls()

__init__(lr: float = 0.001, step_function: bool = True, offset_epochs: int = 0, steps_per_epoch: int = - 1, epochs: int = - 1) → None

Initialize self. See help(type(self)) for accurate signature.

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → A

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → A

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[A]

to_dict(encode_json=False) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str

Definition of the ConstantSchedule

class tfaip.trainer.scheduler.constant.ConstantSchedule(params: tfaip.trainer.scheduler.constant_params.Constant)

Bases: tfaip.trainer.scheduler.learningrate.LearningRateSchedule

__init__(params: tfaip.trainer.scheduler.constant_params.Constant)

Initialize self. See help(type(self)) for accurate signature.

lr(epoch)

Cosine Decay

Definition of the CosineDecayParams, WarmupCosineDecayParams, and WarmupConstantCosineDecayParams

class tfaip.trainer.scheduler.cosine_decay_params.CosineDecayParams(lr: float = 0.001, step_function: bool = True, offset_epochs: int = 0, steps_per_epoch: int = - 1, epochs: int = - 1, learning_circle: int = 3, lr_decay_rate: float = 0.99, decay_fraction: float = 0.1, final_epochs: int = 50)

Bases: tfaip.trainer.scheduler.learningrate_params.LearningRateParams

Definition of a cosine decay

static cls()

learning_circle: int = 3

lr_decay_rate: float = 0.99

decay_fraction: float = 0.1

final_epochs: int = 50

__init__(lr: float = 0.001, step_function: bool = True, offset_epochs: int = 0, steps_per_epoch: int = - 1, epochs: int = - 1, learning_circle: int = 3, lr_decay_rate: float = 0.99, decay_fraction: float = 0.1, final_epochs: int = 50) → None

Initialize self. See help(type(self)) for accurate signature.

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → A

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → A

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[A]

to_dict(encode_json=False) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str

class tfaip.trainer.scheduler.cosine_decay_params.WarmupCosineDecayParams(lr: float = 0.001, step_function: bool = True, offset_epochs: int = 0, steps_per_epoch: int = - 1, epochs: int = - 1, learning_circle: int = 3, lr_decay_rate: float = 0.99, decay_fraction: float = 0.1, final_epochs: int = 50, warmup_epochs: int = 10, warmup_factor: int = 10)

Bases: tfaip.trainer.scheduler.cosine_decay_params.CosineDecayParams

Cosine decay with warmup

static cls()

warmup_epochs: int = 10

warmup_factor: int = 10

__init__(lr: float = 0.001, step_function: bool = True, offset_epochs: int = 0, steps_per_epoch: int = - 1, epochs: int = - 1, learning_circle: int = 3, lr_decay_rate: float = 0.99, decay_fraction: float = 0.1, final_epochs: int = 50, warmup_epochs: int = 10, warmup_factor: int = 10) → None

Initialize self. See help(type(self)) for accurate signature.

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → A

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → A

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[A]

to_dict(encode_json=False) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str

class tfaip.trainer.scheduler.cosine_decay_params.WarmupConstantCosineDecayParams(lr: float = 0.001, step_function: bool = True, offset_epochs: int = 0, steps_per_epoch: int = - 1, epochs: int = - 1, learning_circle: int = 3, lr_decay_rate: float = 0.99, decay_fraction: float = 0.1, final_epochs: int = 50, warmup_epochs: int = 10, warmup_factor: int = 10, constant_epochs: int = 10)

Bases: tfaip.trainer.scheduler.cosine_decay_params.WarmupCosineDecayParams

static cls()

constant_epochs: int = 10

__init__(lr: float = 0.001, step_function: bool = True, offset_epochs: int = 0, steps_per_epoch: int = - 1, epochs: int = - 1, learning_circle: int = 3, lr_decay_rate: float = 0.99, decay_fraction: float = 0.1, final_epochs: int = 50, warmup_epochs: int = 10, warmup_factor: int = 10, constant_epochs: int = 10) → None

Initialize self. See help(type(self)) for accurate signature.

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → A

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → A

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[A]

to_dict(encode_json=False) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str

Definition of the CosineDecaySchedule

class tfaip.trainer.scheduler.cosine_decay.CosineDecaySchedule(params: LearningRateParams, name=None, **kwargs)

Bases: tfaip.trainer.scheduler.learningrate.LearningRateSchedule

Cosine decay

lr(epoch)

class tfaip.trainer.scheduler.cosine_decay.WarmupCosineDecaySchedule(params: LearningRateParams, name=None, **kwargs)

Bases: tfaip.trainer.scheduler.learningrate.LearningRateSchedule

Cosine decay with warmup

lr(epoch)

class tfaip.trainer.scheduler.cosine_decay.WarmupConstantCosineDecaySchedule(params: LearningRateParams, name=None, **kwargs)

Bases: tfaip.trainer.scheduler.learningrate.LearningRateSchedule

Warmup then constant then cosine decay

lr(epoch)

tfaip.trainer.scheduler.cosine_decay.cosine_decay(learn_rate, epoch, batch, decay, alpha, epochs, final_epochs, delay=0, name=None)

tfaip.trainer.scheduler.cosine_decay.warmup_cosine_decay(learn_rate, epoch, batch, decay, alpha, epochs, final_epochs, warmup_epochs, warmup_factor, name=None)

piecewise definde function: from 0 to warmup_epoch: linear increas from learningrate to warmup_factor*learningrate from warmup_epoch to epochs - final_epochs: decay using alpha and learning circle from epochs - final_epochs to end: cosine cooldown like in adam final/cosine_decay

tfaip.trainer.scheduler.cosine_decay.warmup_constant_cosine_decay(learn_rate, epoch, batch, decay, alpha, epochs, final_epochs, warmup_epochs, warmup_factor, constant_epochs, name=None)

piecewise defined function: from 0 to warmup_epoch: linear increase from learningrate to warmup_factor*learningrate from warmup_epoch to epochs - final_epochs: decay using alpha and learning circle from epochs - final_epochs to end: cosine cooldown like in adam final/cosine_decay

Exponential Decay

Definition of the ExponentialDecayParams

class tfaip.trainer.scheduler.exponential_decay_params.ExponentialDecayParams(lr: float = 0.001, step_function: bool = True, offset_epochs: int = 0, steps_per_epoch: int = - 1, epochs: int = - 1, learning_circle: int = 3, lr_decay_rate: float = 0.99)

Bases: tfaip.trainer.scheduler.learningrate_params.LearningRateParams

Exponential decay parameters

static cls()

learning_circle: int = 3

lr_decay_rate: float = 0.99

__init__(lr: float = 0.001, step_function: bool = True, offset_epochs: int = 0, steps_per_epoch: int = - 1, epochs: int = - 1, learning_circle: int = 3, lr_decay_rate: float = 0.99) → None

Initialize self. See help(type(self)) for accurate signature.

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → A

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → A

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[A]

to_dict(encode_json=False) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str

Definition of the ExponentialDecaySchedule

class tfaip.trainer.scheduler.exponential_decay.ExponentialDecaySchedule(params: tfaip.trainer.scheduler.exponential_decay_params.ExponentialDecayParams)

Bases: tfaip.trainer.scheduler.learningrate.LearningRateSchedule

Exponential decay

This class simply wraps keras.optimizers.schedules.ExponentialDecay

__init__(params: tfaip.trainer.scheduler.exponential_decay_params.ExponentialDecayParams)

Initialize self. See help(type(self)) for accurate signature.

lr(epoch)