tfaip.trainer.optimizer¶

Optimizers¶

Definition of the various Optimizers and their OptimizerParams

class tfaip.trainer.optimizer.optimizers.OptimizerParams(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None)¶

Bases: abc.ABC

General parameters of a Optimizer

abstract create() → Tuple[Type[tf.keras.optimizers.Optimizer], Dict[str, Any]]¶

clip_norm: Optional[float] = None¶

clip_value: Optional[float] = None¶

global_clip_norm: Optional[float] = None¶

__init__(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None) → None¶: Initialize self. See help(type(self)) for accurate signature.

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → A¶

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → A¶

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[A]¶

to_dict(encode_json=False) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]¶

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str¶

class tfaip.trainer.optimizer.optimizers.SGDOptimizer(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, momentum: float = 0.0, nesterov: bool = False, weight_decay: float = 0.0)¶

Bases: tfaip.trainer.optimizer.optimizers.OptimizerParams

The Stochastic Gradient Optimizer

momentum: float = 0.0¶

nesterov: bool = False¶

weight_decay: float = 0.0¶

create()¶

__init__(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, momentum: float = 0.0, nesterov: bool = False, weight_decay: float = 0.0) → None¶: Initialize self. See help(type(self)) for accurate signature.

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → A¶

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → A¶

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[A]¶

to_dict(encode_json=False) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]¶

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str¶

class tfaip.trainer.optimizer.optimizers.AdamOptimizer(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, beta_1: float = 0.9, beta_2: float = 0.999, epsilon: float = 1e-07, weight_decay: float = 0.0)¶

Bases: tfaip.trainer.optimizer.optimizers.OptimizerParams

The Adam optimizer

beta_1: float = 0.9¶

beta_2: float = 0.999¶

epsilon: float = 1e-07¶

weight_decay: float = 0.0¶

create()¶

__init__(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, beta_1: float = 0.9, beta_2: float = 0.999, epsilon: float = 1e-07, weight_decay: float = 0.0) → None¶: Initialize self. See help(type(self)) for accurate signature.

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → A¶

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → A¶

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[A]¶

to_dict(encode_json=False) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]¶

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str¶

class tfaip.trainer.optimizer.optimizers.AdamaxOptimizer(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, beta_1: float = 0.9, beta_2: float = 0.999, epsilon: float = 1e-07, weight_decay: float = 0.0)¶

Bases: tfaip.trainer.optimizer.optimizers.AdamOptimizer

The Adamax Optimizer

create()¶

__init__(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, beta_1: float = 0.9, beta_2: float = 0.999, epsilon: float = 1e-07, weight_decay: float = 0.0) → None¶: Initialize self. See help(type(self)) for accurate signature.

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → A¶

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → A¶

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[A]¶

to_dict(encode_json=False) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]¶

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str¶

class tfaip.trainer.optimizer.optimizers.RMSpropOptimizer(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, momentum: float = 0.0, epsilon: float = 1e-07, rho: float = 0.0, centered: bool = False)¶

Bases: tfaip.trainer.optimizer.optimizers.OptimizerParams

The RMSprop Optimizer

momentum: float = 0.0¶

epsilon: float = 1e-07¶

rho: float = 0.0¶

centered: bool = False¶

create()¶

__init__(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, momentum: float = 0.0, epsilon: float = 1e-07, rho: float = 0.0, centered: bool = False) → None¶: Initialize self. See help(type(self)) for accurate signature.

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → A¶

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → A¶

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[A]¶

to_dict(encode_json=False) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]¶

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str¶

class tfaip.trainer.optimizer.optimizers.AdaBeliefOptimizer(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, beta_1: float = 0.9, beta_2: float = 0.999, epsilon: float = 1e-14, weight_decay: float = 0.0, rectify: bool = True, amsgrad: bool = False, sma_threshold: float = 5.0, total_steps: int = 0, warmup_proportion: float = 0.1, min_lr: float = 0.0)¶

Bases: tfaip.trainer.optimizer.optimizers.OptimizerParams

The AdaBeliefOptimizer

beta_1: float = 0.9¶

beta_2: float = 0.999¶

epsilon: float = 1e-14¶

weight_decay: float = 0.0¶

rectify: bool = True¶

amsgrad: bool = False¶

sma_threshold: float = 5.0¶

total_steps: int = 0¶

warmup_proportion: float = 0.1¶

min_lr: float = 0.0¶

create()¶

__init__(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, beta_1: float = 0.9, beta_2: float = 0.999, epsilon: float = 1e-14, weight_decay: float = 0.0, rectify: bool = True, amsgrad: bool = False, sma_threshold: float = 5.0, total_steps: int = 0, warmup_proportion: float = 0.1, min_lr: float = 0.0) → None¶: Initialize self. See help(type(self)) for accurate signature.

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → A¶

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → A¶

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[A]¶

to_dict(encode_json=False) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]¶

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str¶

Gradient Accumulation Optimizer¶

Setup for GradientAccumulation

tfaip.trainer.optimizer.gradient_accumulation_optimizer.create_gradient_accumulation_optimizer(accum_steps: int, parent_optimizer: Type[tensorflow.python.keras.optimizer_v2.optimizer_v2.OptimizerV2], optimizer: dict) → tensorflow.python.keras.optimizer_v2.optimizer_v2.OptimizerV2¶

Weights Moving Average¶

Functionality to implement an exponential moving average on validation weights

class tfaip.trainer.optimizer.weights_moving_average.WeightsMovingAverage(average_decay=0.99, **kwargs)¶

Bases: tensorflow_addons.optimizers.moving_average.MovingAverage

Wrapper for an Optimizer to compute the exponential moving average of trained weights

__init__(average_decay=0.99, **kwargs)¶

Construct a new MovingAverage optimizer.

Parameters

optimizer – str or tf.keras.optimizers.Optimizer that will be used to compute and apply gradients.
average_decay – float. Decay to use to maintain the moving averages of trained variables.
num_updates – Optional count of the number of updates applied to variables.
start_step – int. What step to start the moving average.
dynamic_decay – bool. Whether to change the decay based on the number of optimizer updates. Decay will start at 0.1 and gradually increase up to average_decay after each optimizer update.
name – Optional name for the operations created when applying gradients. Defaults to “MovingAverage”.
**kwargs – keyword arguments. Allowed to be {clipnorm, clipvalue, lr, decay}. clipnorm is clip gradients by norm; clipvalue is clip gradients by value, decay is included for backward compatibility to allow time inverse decay of learning rate. lr is included for backward compatibility, recommended to use learning_rate instead.

to_avg(var_list)¶

to_model(var_list)¶