tfaip.data

DataBase

Module that defines DataBase

tfaip.data.data.dict_to_input_layers(d: Dict[str, tensorflow.python.framework.tensor_spec.TensorSpec]) → Dict[str, keras.engine.input_layer.Input]

tfaip.data.data.validate_specs(func)

class tfaip.data.data.DataBase(params: tfaip.data.data.TDP, **kwargs)

Bases: Generic[tfaip.data.data.TDP], abc.ABC

DataBase class to provide training and validation data.

Override _input_layer_specs, and _output_layer_specs in a custom implementation

classmethod params_cls() → Type[tfaip.data.data.TDP]

classmethod default_params() → tfaip.data.data.TDP

classmethod data_pipeline_cls() → Type[tfaip.data.pipeline.datapipeline.DataPipeline]

DataPipeline for this Data.

Overwrite this and return a custom DataPipeline if you need to modify the tf.data.Dataset (see TFDatasetGenerator).

__init__(params: tfaip.data.data.TDP, **kwargs)

preload(progress_bar=True)

print_params()

property params: tfaip.data.data.TDP

padding_values() → Dict[str, Union[numpy.ndarray, int, numpy.int8, numpy.int16, numpy.int32, numpy.int64, float, numpy.float16, numpy.float32, numpy.float64, bool]]

element_length_fn() → Callable[[Dict[str, Union[tensorflow.python.framework.ops.Tensor, tensorflow.python.keras.engine.keras_tensor.KerasTensor, keras.engine.keras_tensor.KerasTensor]]], Union[tensorflow.python.framework.ops.Tensor, tensorflow.python.keras.engine.keras_tensor.KerasTensor, keras.engine.keras_tensor.KerasTensor]]: Element length for bucked_by_sequence_length

create_pipeline(pipeline_params: tfaip.data.databaseparams.DataPipelineParams, params: tfaip.data.databaseparams.DataGeneratorParams) → tfaip.data.pipeline.datapipeline.DataPipeline

get_or_create_pipeline(pipeline_params: tfaip.data.databaseparams.DataPipelineParams, params: Optional[tfaip.data.databaseparams.DataGeneratorParams]) → tfaip.data.pipeline.datapipeline.DataPipeline

pipeline_by_mode(mode: tfaip.data.pipeline.definitions.PipelineMode) → tfaip.data.pipeline.datapipeline.DataPipeline

create_input_layers() → Dict[str, keras.engine.input_layer.Input]

create_target_as_input_layers() → Dict[str, keras.engine.input_layer.Input]

create_meta_as_input_layers() → Dict[str, keras.engine.input_layer.Input]

input_layer_specs(**kwargs)

target_layer_specs(**kwargs)

meta_layer_specs(**kwargs)

dataset_input_layer_specs() → Dict[str, tensorflow.python.framework.tensor_spec.TensorSpec]

tf.data.Dataset generator: Input specs.

Usually, these should not be overwritten, only if using a custom TFDatasetGenerator (and thus a custom DataPipeline).

dataset_target_layer_specs() → Dict[str, tensorflow.python.framework.tensor_spec.TensorSpec]

tf.data.Dataset generator: Targets specs

Usually, these should not be overwritten, only if using a custom TFDatasetGenerator (and thus a custom DataPipeline).

dataset_meta_layer_specs() → Dict[str, tensorflow.python.framework.tensor_spec.TensorSpec]

tf.data.Dataset generator: Meta specs

Usually, these should not be overwritten, only if using a custom TFDatasetGenerator (and thus a custom DataPipeline).

dump_resources(root_path: str, data_params_dict: dict)

input_tensor_key_for_regularization() → Optional[str]: input gradient regularization helper method !!! the inputs must be stored in a dict under this key !!!

DataBaseParams

Definition of DataBaseParams, DataPipelineParams, and DataGeneratorParams

class tfaip.data.databaseparams.DataGeneratorParams

Bases: abc.ABC

Parameter class that defines how to construct a DataGenerator.

static cls() → Type[DataGenerator]

create(mode: tfaip.data.pipeline.definitions.PipelineMode) → DataGenerator

__init__() → None

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → dataclasses_json.api.A

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → dataclasses_json.api.A

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[dataclasses_json.mm.A]

to_dict(encode_json=False, include_cls=True) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str

class tfaip.data.databaseparams.DataPipelineParams(batch_size: int = 16, limit: int = -1, prefetch: int = -1, num_processes: int = 4, batch_drop_remainder: bool = False, shuffle_buffer_size: int = -1, mode: tfaip.data.pipeline.definitions.PipelineMode = PipelineMode.TRAINING, bucket_boundaries: List[int] = <factory>, bucket_batch_sizes: Optional[List[int]] = None, use_shared_memory: bool = False)

Bases: object

Parameter class that defines the general parameters, e.g. batch size, prefetching, number of processes, … of a certain (e.g. train or val) pipeline.

batch_size: int = 16

limit: int = -1

prefetch: int = -1

num_processes: int = 4

batch_drop_remainder: bool = False

shuffle_buffer_size: int = -1

mode: tfaip.data.pipeline.definitions.PipelineMode = 'training'

bucket_boundaries: List[int]

bucket_batch_sizes: Optional[List[int]] = None

use_shared_memory: bool = False

__init__(batch_size: int = 16, limit: int = -1, prefetch: int = -1, num_processes: int = 4, batch_drop_remainder: bool = False, shuffle_buffer_size: int = -1, mode: tfaip.data.pipeline.definitions.PipelineMode = PipelineMode.TRAINING, bucket_boundaries: List[int] = <factory>, bucket_batch_sizes: Optional[List[int]] = None, use_shared_memory: bool = False) → None

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → dataclasses_json.api.A

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → dataclasses_json.api.A

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[dataclasses_json.mm.A]

to_dict(encode_json=False, include_cls=True) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str

class tfaip.data.databaseparams.DataBaseParams(pre_proc: tfaip.data.pipeline.processor.params.DataProcessorPipelineParams = <factory>, post_proc: tfaip.data.pipeline.processor.params.DataProcessorPipelineParams = <factory>, resource_base_path: str = '/home/docs/checkouts/readthedocs.org/user_builds/tfaip/checkouts/latest/docs/source')

Bases: abc.ABC

Parameters that define the overall setup of the data pipelines (pre_proc and post_proc)

Parameters of this class will be shared among all DataProcessors.

abstract static cls() → Type[DataBase]

create() → DataBase

pre_proc: tfaip.data.pipeline.processor.params.DataProcessorPipelineParams

post_proc: tfaip.data.pipeline.processor.params.DataProcessorPipelineParams

resource_base_path: str = '/home/docs/checkouts/readthedocs.org/user_builds/tfaip/checkouts/latest/docs/source'

__init__(pre_proc: tfaip.data.pipeline.processor.params.DataProcessorPipelineParams = <factory>, post_proc: tfaip.data.pipeline.processor.params.DataProcessorPipelineParams = <factory>, resource_base_path: str = '/home/docs/checkouts/readthedocs.org/user_builds/tfaip/checkouts/latest/docs/source') → None

classmethod from_dict(kvs: Optional[Union[dict, list, str, int, float, bool]], *, infer_missing=False) → dataclasses_json.api.A

classmethod from_json(s: Union[str, bytes, bytearray], *, parse_float=None, parse_int=None, parse_constant=None, infer_missing=False, **kw) → dataclasses_json.api.A

classmethod schema(*, infer_missing: bool = False, only=None, exclude=(), many: bool = False, context=None, load_only=(), dump_only=(), partial: bool = False, unknown=None) → dataclasses_json.mm.SchemaF[dataclasses_json.mm.A]

to_dict(encode_json=False, include_cls=True) → Dict[str, Optional[Union[dict, list, str, int, float, bool]]]

to_json(*, skipkeys: bool = False, ensure_ascii: bool = True, check_circular: bool = True, allow_nan: bool = True, indent: Optional[Union[int, str]] = None, separators: Optional[Tuple[str, str]] = None, default: Optional[Callable] = None, sort_keys: bool = False, **kw) → str