B
|
_0 ã @ s d Z ddlZddlZddlmZ ddlm mZ ddddgZG dd dej Z
|
G dd dej Z G d d
|
d
|
ej j ZG d d dej ZG d d dej ZdS ) z4
|
@author: xingyu liao
|
@contact: liaoxingyu5@jd.com
|
é NÚMishÚSwishÚMemoryEfficientSwishÚGELUc s$ e Zd Z fddZdd Z ZS )r c s t ¡ d S )N)ÚsuperÚ__init__)Úself)Ú __class__© ú./layers/activation.pyr s z Mish.__init__c C s |t t |¡¡ S )N)ÚtorchÚtanhÚFÚsoftplus)r Úxr
|
r
|
r Úforward s z Mish.forward)Ú__name__Ú
|
__module__Ú __qualname__r r Ú __classcell__r
|
r
|
)r r r s c @ s e Zd Zdd ZdS )r c C s |t |¡ S )N)r Úsigmoid)r r r
|
r
|
r r s z Swish.forwardN)r r r r r
|
r
|
r
|
r r s c @ s$ e Zd Zedd Zedd ZdS )ÚSwishImplementationc C s |t |¡ }| |¡ |S )N)r r Úsave_for_backward)ÚctxÚiÚresultr
|
r
|
r r # s
|
zSwishImplementation.forwardc C s, | j d }t |¡}||d|d| S )Nr é )Úsaved_variablesr r )r Ú grad_outputr Z sigmoid_ir
|
r
|
r Úbackward) s
|
|
z SwishImplementation.backwardN)r r r Ú staticmethodr r r
|
r
|
r
|
r r " s r c @ s e Zd Zdd ZdS )r c C s
|
t |¡S )N)r Úapply)r r r
|
r
|
r r 1 s z MemoryEfficientSwish.forwardN)r r r r r
|
r
|
r
|
r r 0 s c @ s e Zd ZdZdd ZdS )r zZ
|
Paper Section 3.4, last paragraph notice that BERT used the GELU instead of RELU
|
c C s6 d| dt t dtj ¡|dt |d¡ ¡ S )Ng à?r é g÷Hmâä¦?é )r r ÚmathÚsqrtÚpiÚpow)r r r
|
r
|
r r : s z GELU.forwardN)r r r Ú__doc__r r
|
r
|
r
|
r r 5 s )r( r$ r Útorch.nnÚnnÚtorch.nn.functionalÚ
|
functionalr Ú__all__ÚModuler r ÚautogradÚFunctionr r r r
|
r
|
r
|
r Ú<module> s
|
