tfaip.trainer.optimizer

Optimizers

Definition of the various Optimizers and their OptimizerParams

class tfaip.trainer.optimizer.optimizers.OptimizerParams(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None)

Bases: abc.ABC

General parameters of a Optimizer

abstract create() → Tuple[Type[tf.keras.optimizers.Optimizer], Dict[str, Any]]

clip_norm: Optional[float] = None

clip_value: Optional[float] = None

global_clip_norm: Optional[float] = None

__init__(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None) → None

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → dataclasses_json.api.A

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → dataclasses_json.api.A

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[dataclasses_json.mm.A]

to_dict(encode_json=False, include_cls=True) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str

class tfaip.trainer.optimizer.optimizers.SGDOptimizer(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, momentum: float = 0.0, nesterov: bool = False, weight_decay: float = 0.0)

Bases: tfaip.trainer.optimizer.optimizers.OptimizerParams

The Stochastic Gradient Optimizer

momentum: float = 0.0

nesterov: bool = False

weight_decay: float = 0.0

create()

__init__(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, momentum: float = 0.0, nesterov: bool = False, weight_decay: float = 0.0) → None

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → dataclasses_json.api.A

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → dataclasses_json.api.A

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[dataclasses_json.mm.A]

to_dict(encode_json=False, include_cls=True) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str

class tfaip.trainer.optimizer.optimizers.AdamOptimizer(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, beta_1: float = 0.9, beta_2: float = 0.999, epsilon: float = 1e-07, weight_decay: float = 0.0)

Bases: tfaip.trainer.optimizer.optimizers.OptimizerParams

The Adam optimizer

beta_1: float = 0.9

beta_2: float = 0.999

epsilon: float = 1e-07

weight_decay: float = 0.0

create()

__init__(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, beta_1: float = 0.9, beta_2: float = 0.999, epsilon: float = 1e-07, weight_decay: float = 0.0) → None

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → dataclasses_json.api.A

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → dataclasses_json.api.A

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[dataclasses_json.mm.A]

to_dict(encode_json=False, include_cls=True) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str

class tfaip.trainer.optimizer.optimizers.AdamaxOptimizer(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, beta_1: float = 0.9, beta_2: float = 0.999, epsilon: float = 1e-07, weight_decay: float = 0.0)

Bases: tfaip.trainer.optimizer.optimizers.AdamOptimizer

The Adamax Optimizer

create()

__init__(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, beta_1: float = 0.9, beta_2: float = 0.999, epsilon: float = 1e-07, weight_decay: float = 0.0) → None

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → dataclasses_json.api.A

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → dataclasses_json.api.A

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[dataclasses_json.mm.A]

to_dict(encode_json=False, include_cls=True) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str

class tfaip.trainer.optimizer.optimizers.RMSpropOptimizer(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, momentum: float = 0.0, epsilon: float = 1e-07, rho: float = 0.0, centered: bool = False)

Bases: tfaip.trainer.optimizer.optimizers.OptimizerParams

The RMSprop Optimizer

momentum: float = 0.0

epsilon: float = 1e-07

rho: float = 0.0

centered: bool = False

create()

__init__(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, momentum: float = 0.0, epsilon: float = 1e-07, rho: float = 0.0, centered: bool = False) → None

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → dataclasses_json.api.A

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → dataclasses_json.api.A

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[dataclasses_json.mm.A]

to_dict(encode_json=False, include_cls=True) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str

class tfaip.trainer.optimizer.optimizers.AdaBeliefOptimizer(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, beta_1: float = 0.9, beta_2: float = 0.999, epsilon: float = 1e-14, weight_decay: float = 0.0, rectify: bool = True, amsgrad: bool = False, sma_threshold: float = 5.0, total_steps: int = 0, warmup_proportion: float = 0.1, min_lr: float = 0.0)

Bases: tfaip.trainer.optimizer.optimizers.OptimizerParams

The AdaBeliefOptimizer

beta_1: float = 0.9

beta_2: float = 0.999

epsilon: float = 1e-14

weight_decay: float = 0.0

rectify: bool = True

amsgrad: bool = False

sma_threshold: float = 5.0

total_steps: int = 0

warmup_proportion: float = 0.1

min_lr: float = 0.0

create()

__init__(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, beta_1: float = 0.9, beta_2: float = 0.999, epsilon: float = 1e-14, weight_decay: float = 0.0, rectify: bool = True, amsgrad: bool = False, sma_threshold: float = 5.0, total_steps: int = 0, warmup_proportion: float = 0.1, min_lr: float = 0.0) → None

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → dataclasses_json.api.A

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → dataclasses_json.api.A

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[dataclasses_json.mm.A]

to_dict(encode_json=False, include_cls=True) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str

class tfaip.trainer.optimizer.optimizers.LAMBOptimizer(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, beta_1: float = 0.9, beta_2: float = 0.999, epsilon: float = 1e-06, weight_decay: float = 0.0, exclude_from_weight_decay: Optional[List[str]] = None, exclude_from_layer_adaptation: Optional[List[str]] = None)

Bases: tfaip.trainer.optimizer.optimizers.OptimizerParams

The LAMBOptimizer

beta_1: float = 0.9

beta_2: float = 0.999

epsilon: float = 1e-06

weight_decay: float = 0.0

exclude_from_weight_decay: Optional[List[str]] = None

exclude_from_layer_adaptation: Optional[List[str]] = None

create()

__init__(clip_norm: Optional[float] = None, clip_value: Optional[float] = None, global_clip_norm: Optional[float] = None, beta_1: float = 0.9, beta_2: float = 0.999, epsilon: float = 1e-06, weight_decay: float = 0.0, exclude_from_weight_decay: Optional[List[str]] = None, exclude_from_layer_adaptation: Optional[List[str]] = None) → None

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → dataclasses_json.api.A

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → dataclasses_json.api.A

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[dataclasses_json.mm.A]

to_dict(encode_json=False, include_cls=True) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str

Gradient Accumulation Optimizer

Setup for GradientAccumulation

tfaip.trainer.optimizer.gradient_accumulation_optimizer.create_gradient_accumulation_optimizer(accum_steps: int, parent_optimizer: Type[keras.optimizer_v2.optimizer_v2.OptimizerV2], optimizer: dict) → keras.optimizer_v2.optimizer_v2.OptimizerV2

Weights Moving Average

Functionality to implement an exponential moving average on validation weights

class tfaip.trainer.optimizer.weights_moving_average.WeightsMovingAverage(average_decay=0.99, **kwargs)

Bases: tensorflow_addons.optimizers.moving_average.MovingAverage

Wrapper for an Optimizer to compute the exponential moving average of trained weights

__init__(average_decay=0.99, **kwargs)

Construct a new MovingAverage optimizer.

Parameters

optimizer – str or tf.keras.optimizers.Optimizer that will be used to compute and apply gradients.
average_decay – float. Decay to use to maintain the moving averages of trained variables.
num_updates – Optional count of the number of updates applied to variables.
start_step – int. What step to start the moving average.
dynamic_decay – bool. Whether to change the decay based on the number of optimizer updates. Decay will start at 0.1 and gradually increase up to average_decay after each optimizer update.
name – Optional name for the operations created when applying gradients. Defaults to “MovingAverage”.
**kwargs – keyword arguments. Allowed to be {clipnorm, clipvalue, lr, decay}. clipnorm is clip gradients by norm; clipvalue is clip gradients by value, decay is included for backward compatibility to allow time inverse decay of learning rate. lr is included for backward compatibility, recommended to use learning_rate instead.

to_avg(var_list)

to_model(var_list)