U
|
_0 ã @ s d Z ddlZddlZddlmZ ddlm mZ ddddgZG dd dej Z
|
G dd dej Z G d d
|
d
|
ej j ZG d d dej ZG d d dej ZdS ) z4
|
@author: xingyu liao
|
@contact: liaoxingyu5@jd.com
|
é NÚMishÚSwishÚMemoryEfficientSwishÚGELUc s$ e Zd Z fddZdd Z ZS )r c s t ¡ d S ©N)ÚsuperÚ__init__)Úself©Ú __class__© ú./layers/activation.pyr s z Mish.__init__c C s |t t |¡¡ S r )ÚtorchÚtanhÚFÚsoftplus©r Úxr r r Úforward s z Mish.forward)Ú__name__Ú
|
__module__Ú __qualname__r r Ú __classcell__r r r
|
r r s c @ s e Zd Zdd ZdS )r c C s |t |¡ S r )r Úsigmoidr r r r r s z Swish.forwardN©r r r r r r r r r s c @ s$ e Zd Zedd Zedd ZdS )ÚSwishImplementationc C s |t |¡ }| |¡ |S r )r r Úsave_for_backward)ÚctxÚiÚresultr r r r # s
|
zSwishImplementation.forwardc C s, | j d }t |¡}||d|d| S )Nr é )Úsaved_variablesr r )r Ú grad_outputr Z sigmoid_ir r r Úbackward) s
|
|
z SwishImplementation.backwardN)r r r Ú staticmethodr r# r r r r r " s
|
r c @ s e Zd Zdd ZdS )r c C s
|
t |¡S r )r Úapplyr r r r r 1 s z MemoryEfficientSwish.forwardNr r r r r r 0 s c @ s e Zd ZdZdd ZdS )r zZ
|
Paper Section 3.4, last paragraph notice that BERT used the GELU instead of RELU
|
c C s6 d| dt t dtj ¡|dt |d¡ ¡ S )Ng à?r é g÷Hmâä¦?é )r r ÚmathÚsqrtÚpiÚpowr r r r r : s z GELU.forwardN)r r r Ú__doc__r r r r r r 5 s )r, r( r Útorch.nnÚnnÚtorch.nn.functionalÚ
|
functionalr Ú__all__ÚModuler r ÚautogradÚFunctionr r r r r r r Ú<module> s ü
|
