U
|
_c ã @ s d Z ddlZddlmZ ddlmZ ddlmZmZ ddlm Z G dd dej
|
Z G d d
|
d
|
ej
|
Z d d Z d d Zdd ZdddZdS )z9
|
@author: liaoxingyu
|
@contact: sherlockliao01@gmail.com
|
é N)Únn)Ú BatchNorm2d)ÚReLUÚ LeakyReLU)Ú Parameterc s4 e Zd Z fddZdd Zdd Zdd Z ZS ) ÚTLUc s0 t t| ¡ || _tt |¡| _| ¡ dS )z9max(y, tau) = max(y - tau, 0) + tau = ReLU(y - tau) + tauN) Úsuperr Ú__init__Ú num_featuresr ÚtorchÚTensorÚtauÚreset_parameters)Úselfr
|
©Ú __class__© ú./layers/frn.pyr s z TLU.__init__c C s t j | j¡ d S ©N)r ÚinitÚzeros_r ©r r r r r s zTLU.reset_parametersc C s dj f | jS )Nznum_features={num_features}©ÚformatÚ__dict__r r r r Ú
|
extra_repr s zTLU.extra_reprc C s t || j d| jdd¡¡S )Né )r Úmaxr Úviewr
|
)r Úxr r r Úforward s z TLU.forward©Ú__name__Ú
|
__module__Ú __qualname__r r r r Ú __classcell__r r r r r s r c s6 e Zd Zd fdd Zdd Zdd Zd d
|
Z ZS ) ÚFRNçíµ ÷Æ°>Fc sv t t| ¡ || _|| _|| _tt |¡| _ tt |¡| _
|
|rVtt d¡| _ n| dt |g¡¡ | ¡ dS )zë
|
weight = gamma, bias = beta
|
beta, gamma:
|
Variables of shape [1, 1, 1, C]. if TensorFlow
|
Variables of shape [1, C, 1, 1]. if PyTorch
|
eps: A scalar constant or learnable variable.
|
r ÚepsN)r r& r r
|
Úinit_epsÚis_eps_leanabler r r ÚweightÚbiasr( Úregister_bufferr )r r
|
r( r* r r r r ! s z FRN.__init__c C s8 t j | j¡ t j | j¡ | jr4t j | j| j ¡ d S r )
|
r r Úones_r+ r r, r* Ú constant_r( r) r r r r r 7 s zFRN.reset_parametersc C s dj f | jS )Nz+num_features={num_features}, eps={init_eps}r r r r r r = s zFRN.extra_reprc C s` | d¡jddgdd}|t || j ¡ ¡ }| j d| jdd¡| | j d| jdd¡ }|S )ai
|
0, 1, 2, 3 -> (B, H, W, C) in TensorFlow
|
0, 1, 2, 3 -> (B, C, H, W) in PyTorch
|
TensorFlow code
|
nu2 = tf.reduce_mean(tf.square(x), axis=[1, 2], keepdims=True)
|
x = x * tf.rsqrt(nu2 + tf.abs(eps))
|
# This Code include TLU function max(y, tau)
|
return tf.maximum(gamma * x + beta, tau)
|
é é T)ÚdimÚkeepdimr )
|
ÚpowÚmeanr Úrsqrtr( Úabsr+ r r
|
r, )r r Znu2r r r r @ s ,z FRN.forward)r' Fr! r r r r r& s r& c C s | }d}d}d}| ¡ D ]x\}}|rnt|ttfrnt|trR| |t|jd¡ nt | |t |jd¡ n| |t
|
|¡ |}|}t|t}q|S )z5
|
Convert 'BatchNorm2d + ReLU' to 'FRN + TLU'
|
NF©r
|
) Únamed_childrenÚ
|
isinstancer r r Ú
|
add_moduler& r
|
ÚNotImplementedErrorr Ú bnrelu_to_frn)ÚmoduleÚmodZ before_nameZ before_childÚ is_before_bnÚnameÚchildr r r r= V s$
|
|
ÿ r= c C s | }d }| ¡ D ]t\}}t||rrt||rrt|trP|j}| |t|jd¡ t|tt fr| |t
|
|d¡ q| |t ||¡ q|S )Nr8 ) r9 ÚhasattrÚgetattrr: r r
|
r; r& r r r Úconvert)r> Ú flag_namer? Z before_chrA rB r r r rE s s
|
rE c C sT | }| ¡ D ]B\}}t|dr<t|| | |t||¡ q | |t||¡ q |S )NÚis_convert_frn)r9 rC Údelattrr; Ú remove_flags)r> rF r? rA rB r r r rI s
|
|
rI ©r1 é rK r0 rG c sÆ t t dgfdd}dg fdd}t|trL|g}fdd|D } |¡ |¡ | }| ¡ ¡ D ] }| ¡ q D ] }| ¡ qtdtdS ) NFc s& fdd} |¡¡ d S )Nc sb t tjtjfskr( d¡ d S d o<t | ttf}|rNt| d t | t¡ d S ©NFéÿÿÿÿT) r: r Ú
|
SequentialÚ
|
ModuleListÚappendr r Úsetattrr ©r ÚinputÚoutputZ is_converted)rF r@ Úmodelr> r r Úhook s
|
z;bnrelu_to_frn2.<locals>.register_forward_hook.<locals>.hook)rP Úregister_forward_hook©r> rV )rF Ú forard_hooksr@ rU ©r> r rW s z-bnrelu_to_frn2.<locals>.register_forward_hookc s& fdd} |¡¡ d S )Nc sb t tjtjfskr( d¡ d S d o8t | t}|rJt| d t | ttf¡ d S rL ) r: r rN rO rP r rQ r r rR )rF Úis_before_relurU r> r r rV ¥ s
|
z<bnrelu_to_frn2.<locals>.register_backward_hook.<locals>.hook)rP Úregister_backward_hookrX )Úbackward_hooksrF r[ rU rZ r r\ ¤ s z.bnrelu_to_frn2.<locals>.register_backward_hookc s g | ]}t j f| qS r )r Úrand)Ú.0Zin_size)Ú
|
batch_sizer r Ú
|
<listcomp>µ s z"bnrelu_to_frn2.<locals>.<listcomp>)rF ) Úlistr: ÚtupleÚapplyÚsumÚbackwardÚremoverE rI ) rU Ú
|
input_sizer` rF rW r\ r rT Úhr )r] r` rF rY r@ r[ rU r Úbnrelu_to_frn2 s(
|
|
|
|
|
rj )rJ r0 rG )Ú__doc__r r Ztorch.nn.modules.batchnormr Útorch.nnr r Ztorch.nn.parameterr ÚModuler r& r= rE rI rj r r r r Ú<module> s 6
|
