B
|
Á_È ã @ s¤ d Z ddlZddlZddlm mZ ddlmZ dddddd gZG d
|
d dejZ G d d dej
|
Z
|
G d d dej Z G d d de Z G dd de Zdd ZdS )z9
|
@author: liaoxingyu
|
@contact: sherlockliao01@gmail.com
|
é N)ÚnnÚ BatchNormÚIBNÚGhostBatchNormÚFrozenBatchNormÚ SyncBatchNormÚget_normc s e Zd Zd fdd Z ZS ) r çñhãµøä>ç¹?Fç ð?ç c sb t j|||d |d k r*tj | j|¡ |d k rBtj | j|¡ | j | ¡ | j | ¡ d S )N)ÚepsÚmomentum)ÚsuperÚ__init__r ÚinitÚ constant_ÚweightÚbiasÚrequires_grad_) ÚselfÚ num_featuresr r Ú weight_freezeÚ bias_freezeÚ weight_initÚ bias_initÚkwargs)Ú __class__© ú./layers/batch_norm.pyr s zBatchNorm.__init__)r r
|
FFr r )Ú__name__Ú
|
__module__Ú __qualname__r Ú __classcell__r r )r r r s c s e Zd Zd fdd Z ZS ) r çñhãµøä>ç¹?Fç ð?ç c sb t j|||d |d k r*tj | j|¡ |d k rBtj | j|¡ | j | ¡ | j | ¡ d S )N)r r )r r r r r r r r )r r r r r r r r )r r r r " s zSyncBatchNorm.__init__)r$ r% FFr& r' )r r! r" r r# r r )r r r ! s c s$ e Zd Z fddZdd Z ZS )r c sL t t| ¡ t|d }|| _|| }tj|dd| _t||f|| _ d S )Né T)Úaffine)
|
r r r ÚintÚhalfr ÚInstanceNorm2dÚINr ÚBN)r ÚplanesÚbn_normr Zhalf1Zhalf2)r r r r , s z IBN.__init__c C sH t || jd¡}| |d ¡ ¡}| |d ¡ ¡}t ||fd¡}|S )Né r )ÚtorchÚsplitr+ r- Ú
|
contiguousr. Úcat)r Úxr3 Zout1Zout2Úoutr r r Úforward4 s
|
z IBN.forward)r r! r" r r8 r# r r )r r r + s c s& e Zd Zd fdd Zdd Z ZS )r r1 c s> t j|f| || _| dt |¡¡ | dt |¡¡ d S )NÚ running_meanÚ running_var)r r Ú
|
num_splitsÚregister_bufferr2 ÚzerosÚones)r r r; r )r r r r = s zGhostBatchNorm.__init__c
|
C sô |j \}}}}| js| jsÌ| j | j¡| _| j | j¡| _t | d|| j ||¡| j| j| j
|
| j¡| j | j¡d| j | j ¡ ||||¡}tj| j | j| j¡dd| _tj| j | j| j¡dd| _|S t || j| j| j
|
| j d| j | j ¡S d S )NéÿÿÿÿTr )ÚdimF)ÚshapeÚtrainingÚtrack_running_statsr9 Úrepeatr; r: ÚFÚ
|
batch_normÚviewr r r r r2 Úmeanr )r ÚinputÚNÚCÚHÚWÚoutputsr r r r8 C s
|
zGhostBatchNorm.forward)r1 )r r! r" r r8 r# r r )r r r < s c sN e Zd ZdZdZd fdd Zdd Z fdd Zd
|
d Ze d d Z
|
Z S )r a(
|
BatchNorm2d where the batch statistics and the affine parameters are fixed.
|
It contains non-trainable buffers called
|
"weight" and "bias", "running_mean", "running_var",
|
initialized to perform identity transformation.
|
The pre-trained backbone models from Caffe2 only contain "weight" and "bias",
|
which are computed from the original four parameters of BN.
|
The affine transform `x * weight + bias` will perform the equivalent
|
computation of `(x - running_mean) / sqrt(running_var) * weight + bias`.
|
When loading a backbone model from Caffe2, "running_mean" and "running_var"
|
will be left unchanged as identity transformation.
|
Other pre-trained backbone models may contain all 4 parameters.
|
The forward is implemented by `F.batch_norm(..., training=False)`.
|
é çñhãµøä>c s* t j|fddd| || _|| _d S )NT)r r )r r r r )r r r r )r r r r g s zFrozenBatchNorm.__init__c C s~ |j rX| j| j| j ¡ }| j| j| }| dddd¡}| dddd¡}|| | S tj || j| j| j| jd| jdS d S )Nr1 r? F)rB r )
|
Ú requires_gradr r: r Úrsqrtr r9 ÚreshaperE rF )r r6 Úscaler r r r r8 l s zFrozenBatchNorm.forwardc
|
s¾ | dd ¡}|d ks |dk r\|d |kr<t | j¡||d < |d |kr\t | j¡||d < |d k r¢|dk r¢t t¡} | d
|
| d¡¡¡ ||d | j 8 < t |||||||¡ d S )NÚversionr( r9 r: rO z,FrozenBatchNorm {} is upgraded to version 3.Ú.)Úgetr2 Ú
|
zeros_liker9 Ú ones_liker: ÚloggingÚ getLoggerr ÚinfoÚformatÚrstripr r Ú_load_from_state_dict)
|
r Ú
|
state_dictÚprefixÚlocal_metadataÚstrictÚ missing_keysÚunexpected_keysÚ
|
error_msgsrU Úlogger)r r r r_ s
|
z%FrozenBatchNorm._load_from_state_dictc C s d | j| j¡S )Nz*FrozenBatchNorm2d(num_features={}, eps={}))r] r r )r r r r Ú__repr__ s zFrozenBatchNorm.__repr__c C s´ t jj}|j|jf}|}t||r|| |j}|jrZ|jj
|
¡ ¡ |j_ |j j
|
¡ ¡ |j _ |j j |j _ |jj |j_ |j|_n4x2| ¡ D ]&\}}| |¡}||k r| ||¡ qW |S )a
|
Convert BatchNorm/SyncBatchNorm in module into FrozenBatchNorm.
|
Args:
|
module (torch.nn.Module):
|
Returns:
|
If module is BatchNorm/SyncBatchNorm, returns a new module.
|
Otherwise, in-place convert module and return it.
|
Similar to convert_sync_batchnorm in
|
https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/batchnorm.py
|
)r ÚmodulesÚ batchnormÚ BatchNorm2dr Ú
|
isinstancer r) r ÚdataÚcloneÚdetachr r9 r: r Únamed_childrenÚconvert_frozen_batchnormÚ
|
add_module)ÚclsÚmoduleZ bn_moduleÚresÚnameÚchildÚ new_childr r r rq s
|
|
|
|
z(FrozenBatchNorm.convert_frozen_batchnorm)rP ) r r! r" Ú__doc__Ú_versionr r8 r_ rh Ú classmethodrq r# r r )r r r U s c K s> t | tr2t| dkrdS tttdd td| } | |f|S )aZ
|
Args:
|
norm (str or callable): either one of BN, GhostBN, FrozenBN, GN or SyncBN;
|
or a callable that thakes a channel number and returns
|
the normalization layer as a nn.Module
|
out_channels: number of channels for normalization layer
|
|
Returns:
|
nn.Module or None: the normalization layer
|
r Nc [ s t d| ¡S )Né )r Ú GroupNorm)ÚchannelsÚargsr r r Ú<lambda>Í ó zget_norm.<locals>.<lambda>)r. ZGhostBNZFrozenBNZGNZsyncBN)rl ÚstrÚlenr r r r )ÚnormÚ out_channelsr r r r r » s
|
)ry rZ r2 Útorch.nn.functionalr Ú
|
functionalrE Ú__all__rk r r ÚModuler r r r r r r r Ú<module> s
|
|
f
|
