create_dl_layer_activation🔗
Short description🔗
create_dl_layer_activation — Create an activation layer.
Signature🔗
create_dl_layer_activation( dl_layer DLLayerInput, string LayerName, string ActivationType, attribute.name GenParamName, attribute.value GenParamValue, out dl_layer DLLayerActivation )
Description🔗
The operator create_dl_layer_activation creates an activation layer
whose handle is returned in DLLayerActivation.
The parameter DLLayerInput determines the feeding input layer and
expects the layer handle as value.
The parameter LayerName sets an individual layer name.
Note that if creating a model using create_dl_model each layer of
the created network must have a unique name.
The parameter ActivationType sets the type of the activation.
Every activation mode defines a pointwise function.
Supported activation types are:
-
'abs': Absolute value.
\[\begin{eqnarray*} Abs(x) := |x| \end{eqnarray*}\] -
'acos': Arccosine activation.
-
'asin': Arcsine activation.
-
'atan': Arctangent activation.
-
'ceil': Rounds the input up to the nearest integer.
-
'celu': Continuously differentiable exponential linear unit (Celu) activation, which is defined as follows:
\[\begin{eqnarray*} Celu(x) := \left\{\begin{array}{ll} \alpha (e^{\frac{x}{\alpha}} - 1) & \textnormal{if } x < 0, \\ x & \textnormal{else.} \end{array} \right. \end{eqnarray*}\]Setting the generic parameter 'alpha' determines the value \(\alpha\) (default: 1.0). For \(\alpha=1\) a Celu activation is identical to an ELU.
-
'clip': Clip the input to a given interval:
\[\begin{eqnarray*} Clip(x) := \left\{\begin{array}{ll} min & \textnormal{if } x\leq min, \\ max & \textnormal{if } x\geq max,\\ x & \textnormal{else.} \end{array} \right. \end{eqnarray*}\]Setting the generic parameters 'min' and 'max' determines the values \(min\) and \(max\), respectively.
-
'cos': Cosine activation.
-
'cosh': Hyperbolic cosine activation.
-
'elu': Exponential linear unit (ELU) activation, which is defined as follows:
\[\begin{eqnarray*} ELU(x) := \left\{\begin{array}{ll} \alpha (e^x - 1) & \textnormal{if } x < 0, \\ x & \textnormal{else.} \end{array} \right. \end{eqnarray*}\]Setting the generic parameter 'alpha' determines the value \(\alpha\) (default: 1.0).
-
'erf': Gauss error function, which is defined as follows:
\[\begin{eqnarray*} erf(x) := \frac{2}{\sqrt{\pi}} \int_{0}^{x} e^{-t^2} \,dt \end{eqnarray*}\] -
'exp': Exponential function:
\[\begin{eqnarray*} Exp(x) := e^x \end{eqnarray*}\] -
'floor': Rounds the input down to the nearest integer.
-
'gelu': Gaussian error linear unit (Gelu) activation, which is defined as follows:
\[\begin{eqnarray*} Gelu(x) := 0.5 * x * (1 + erf(\frac{x}{\sqrt{2}})) \end{eqnarray*}\]Setting the generic parameter 'approximate' to 'tanh' determines whether an approximate function estimation is used:
\[\begin{eqnarray*} Gelu(x) := 0.5 * x * (1 + tanh(\sqrt{\frac{2}{\pi}} * (x + 0.044715 * x^3))) \end{eqnarray*}\] -
'hard_sigmoid': HardSigmoid activation:
\[\begin{eqnarray*} HardSigmoid(x) := \max(0, \min(1, \alpha x + \beta)) \end{eqnarray*}\]Setting the generic parameters 'alpha' and 'beta' determines the values \(\alpha\) (default: 0.2) and \(\beta\) (default: 0.5).
-
'hard_swish': HardSwish activation:
\[\begin{eqnarray*} HardSwish(x) := x * HardSigmoid(x) \end{eqnarray*}\]with fixed \(\alpha=\frac{1}{6}\) and \(\beta=0.5\).
-
'log': Natural logarithm:
\[\begin{eqnarray*} Log(x) := \log(x) \end{eqnarray*}\] -
'mish': Mish activation:
\[\begin{eqnarray*} Mish(x) := x * \tanh{(Softplus(x))} \end{eqnarray*}\] -
'neg': Negative of input:
\[\begin{eqnarray*} Neg(x) := -x \end{eqnarray*}\] -
'pow': Power function:
\[\begin{eqnarray*} Pow(x) := x^{exponent} \end{eqnarray*}\]Setting the generic parameter 'exponent' determines the value \(exponent\).
-
'reciprocal': Reciprocal function:
\[\begin{eqnarray*} Reciprocal(x) := \frac{1}{x} \end{eqnarray*}\] -
'relu': Rectified linear unit (ReLU) activation. By setting a specific ReLU parameter, another type can be specified instead of the standard ReLU:
-
Standard ReLU, defined as follows:
\[\begin{eqnarray*} ReLU(x) := \max(0, x) \end{eqnarray*}\] -
Bounded ReLU, defined as follows:
\[\begin{eqnarray*} ReLU(x) := \left\{\begin{array}{ll} 0 & \textnormal{if } x\leq 0, \\ x & \textnormal{if } 0<x\leq\beta,\\ \beta & \textnormal{else.} \end{array} \right. \end{eqnarray*}\]Setting the generic parameter 'upper_bound' will result in a bounded ReLU and determines the value of \(\beta\).
-
Leaky ReLU, defined as follows:
\[\begin{eqnarray*} ReLU(x) := \left\{\begin{array}{ll} \alpha x & \textnormal{if } x\leq 0, \\ x & \textnormal{else.} \end{array} \right. \end{eqnarray*}\]Setting the generic parameter 'alpha' results in a leaky ReLU and determines the value \(\alpha\). When the generic parameter 'upper_bound' is set, 'alpha' will be ignored and the result will be a bounded ReLU.
-
-
'round': Rounds the input to the nearest integer. Values with ‘.5’ are rounded to the nearest even integer, e.g.: \(Round(2.5) = 2\), \(Round(3.5) = 4\).
-
'sigmoid': Sigmoid activation, which is defined as follows:
\[\begin{eqnarray*} Sigmoid(x) := \frac{1}{1 + e^{-x}} \end{eqnarray*}\] -
'sin': Sine activation.
-
'sinh': Hyperbolic sine activation.
-
'softplus': Softplus activation function, which is defined as follows:
\[\begin{eqnarray*} Softplus(x) := \log(e^{x} + 1) \end{eqnarray*}\] -
'softsign': Softsign activation function, which is defined as follows:
\[\begin{eqnarray*} Softsign(x) := \frac{x}{1 + |x|} \end{eqnarray*}\] -
'sqrt': Square root of the input:
\[\begin{eqnarray*} Sqrt(x) := \sqrt{x} \end{eqnarray*}\] -
'swish': Swish activation function, which is defined as follows:
\[\begin{eqnarray*} Swish(x) := x * sigmoid(\alpha x) = \frac{x}{1 + e^{-\alpha x}} \end{eqnarray*}\]Setting the generic parameter 'alpha' determines the value \(\alpha\) (default: 1.0).
-
'tan': Tangent activation.
-
'tanh': Tanh activation, which is defined as follows:
\[\begin{eqnarray*} Tanh(x) := 1 - \frac{2}{e^{2x} + 1} \end{eqnarray*}\] -
'thresholded_relu': Thresholded ReLU, defined as follows:
\[\begin{eqnarray*} ThresholdedReLU(x) := \left\{\begin{array}{ll} 0 & \textnormal{if } x\leq \alpha, \\ x & \textnormal{else.} \end{array} \right. \end{eqnarray*}\]Setting the generic parameter 'alpha' determines the value \(\alpha\) (default: 1.0).
The following generic parameters GenParamName and the corresponding
values GenParamValue are supported:
-
'is_inference_output': Determines whether
apply_dl_modelwill include the output of this layer in the dictionaryDLResultBatcheven without specifying this layer inOutputs('true') or not ('false').Default: 'false'
-
'upper_bound' ('relu'): Float value defining an upper bound for a rectified linear unit. If the activation layer is part of a model which has been created using
create_dl_model, the upper bound can be unset. To do so, useset_dl_model_layer_paramand set an empty tuple for 'upper_bound'.Default: []
-
'alpha' ('relu', 'elu', 'celu','thresholded_relu', 'swish', 'hard_sigmoid'): Float value defining the alpha parameter of a leaky ReLU, thresholded ReLU, ELU, CELU, Swish or HardSigmoid-activation.
Restriction: The value of 'alpha' must be positive or zero for all activations except for
ActivationType'thresholded_relu'. This parameter is incompatible with and overridden by 'upper_bound' forActivationType'relu'.Default: 0.2 for 'hard_sigmoid', else 1.0
-
'beta' ('hard_sigmoid'): Float value defining the beta parameter of a HardSigmoid activation.
Default: 0.5
-
'min' ('clip'): Float value defining the min parameter of a clip activation.
-
'max' ('clip'): Float value defining the max parameter of a clip activation.
-
'approximate' ('gelu'): This value determines whether an approximate function estimation is used.
List of values: 'tanh', 'false'
Default: 'false'
-
'exponent' ('pow'): Float value defining the exponent parameter of a pow activation.
Certain parameters of layers created using this operator
create_dl_layer_activation can be set and retrieved using
further operators.
The following tables give an overview, which parameters can be set
using set_dl_model_layer_param and which ones can be retrieved
using get_dl_model_layer_param or get_dl_layer_param. Note, the
operators set_dl_model_layer_param and get_dl_model_layer_param
require a model created by create_dl_model.
| Layer Parameters | set |
get |
|---|---|---|
'activation_type' (ActivationType) |
x |
x |
'input_layer' (DLLayerInput) |
x |
|
'name' (LayerName) |
x |
x |
'output_layer' (DLLayerActivation) |
x |
|
| 'shape' | x |
|
| 'type' | x |
| Generic Layer Parameters | set |
get |
|---|---|---|
| 'is_inference_output' | x |
x |
| 'num_trainable_params' | x |
|
| 'alpha' | x |
x |
| 'beta' | x |
x |
| 'upper_bound' | x |
x |
| 'min' | x |
x |
| 'max' | x |
x |
| 'approximate' | x |
x |
| 'exponent' | x |
x |
Execution information🔗
Execution information
-
Multithreading type: reentrant (runs in parallel with non-exclusive operators).
-
Multithreading scope: global (may be called from any thread).
-
Processed without parallelization.
Parameters🔗
DLLayerInput (input_control) dl_layer → (handle)
Feeding layer.
LayerName (input_control) string → (string)
Name of the output layer.
ActivationType (input_control) string → (string)
Activation type.
Default: 'relu'
List of values: 'abs', 'acos', 'asin', 'atan', 'ceil', 'celu', 'clip', 'cos', 'cosh', 'elu', 'erf', 'exp', 'floor', 'gelu', 'hard_sigmoid', 'hard_swish', 'log', 'mish', 'neg', 'pow', 'reciprocal', 'relu', 'round', 'sigmoid', 'sin', 'sinh', 'softplus', 'softsign', 'sqrt', 'swish', 'tan', 'tanh', 'thresholded_relu'
GenParamName (input_control) attribute.name(-array) → (string)
Generic input parameter names.
Default: []
List of values: 'alpha', 'approximate', 'beta', 'exponent', 'is_inference_output', 'max', 'min', 'upper_bound'
GenParamValue (input_control) attribute.value(-array) → (string / integer / real)
Generic input parameter values.
Default: []
Suggested values: 'true', 'false', 'tanh'
DLLayerActivation (output_control) dl_layer → (handle)
Activation layer.
Module🔗
Deep Learning Professional