---
title: [DL Hacks 実装]Attention is All You Need
tags: 
author: [Deep Learning JP](https://www.docswell.com/user/DeepLearning2023)
site: [Docswell](https://www.docswell.com/)
thumbnail: https://bcdn.docswell.com/page/YE65P8G1JV.jpg?width=480
description: 2018/2/19  Deep Learning JP:  http://deeplearning.jp/hacks/
published: February 19, 18
canonical: https://www.docswell.com/s/DeepLearning2023/K982NV-dl-hacks-attention-is-all-you-need
---
# Page. 1

![Page Image](https://bcdn.docswell.com/page/YE65P8G1JV.jpg)

1
,.
11
.
.
. 11
.
.
.
1
1
J
G
.
G
S
1
P
.
N
. 1
GY
.
US
L
AI
K
V


# Page. 2

![Page Image](https://bcdn.docswell.com/page/GE5YK9VLE4.jpg)

:
•
•
.2 n
– /
– n
–
A
– ln
rN
– . .2
22
. /:
ci
:
A
b
2
N
2
gfd
N o
ab
N s
k m
a
G
p
gfd
t
Y
w
2
u
A
N
b
Nm
ci
k
gfd
N
e
h


# Page. 3

![Page Image](https://bcdn.docswell.com/page/972GW2337R.jpg)

&
A 4B3A
23:7 B
- 45
3
1
56
2
7
.5 B A
5 4
4
A53AB 5
AA5 A
SR
M
M A5
6
I


# Page. 4

![Page Image](https://bcdn.docswell.com/page/DJYVLY18JM.jpg)

DIG
IA D
•
g
r
N
vsg
V
T
–
o
M7
Q
s
M
gnRe
(
g
• 7
TQ M
o lM
D
•
q
ld o
M
g l
•
SiM. 29 ) g1892q Q M
g
–
q
Q
• 6
A 7
GG DI
•
DK C IA D C 8
v
8 9
g
M II DIA D
•
•
–
II DIA D
).
kVe
G C
IL G O
D I 8
D
a
q Qo dcMu
c8 9 g
q
Q
tv
V
Nsq Q
Q
d
T o
d
q R
VPo
e N w
q
o
- 0) . q Q
UUo
g
1 GDAD
Q
g
e
DK8 8 O
M
Tn
e
Qo
vsc
t
tv g8 9 q
q
c o


# Page. 5

![Page Image](https://bcdn.docswell.com/page/V7NM4P29E8.jpg)

B E
• s
g
z
– ) D
. EB
–
ID . D
–
–
P
..a
T
M
•
D
pn
i
e
ba
x dN
kro
w
U
2
kro
d
b P
t
u
g
Sf
dN
L ml
hg Mf
ml
–
C BD D B
BI
DG B C B
B
DG B
f
•
ml
–
(
A C
DD D
B . DEB
E
–
AB
B
B
CDB D
CE
B D
•
c b
–
DBE DEB
DD D
D
)
– yv C
DD D i
ml
–
x
•
DD
Sf
w
i
-
B
i
f
f
BI
i
BI


# Page. 6

![Page Image](https://bcdn.docswell.com/page/YJ9NQ313J3.jpg)

2D
A ,F
• rgl
D
–
1 EIH
4D H D
– 6
.
– 6
-
D
– 6
( -
– 6
)
D
DF
D
D
D=H
mz
A.
2IAH
HD
HD
6
F
HIF
w[
Fcfa
D
D
D
F
H
A
A
EIH
>
n
>
s
N3
D
x
F
h p
D
i
,HH H D
5
F DF
AM HD
F DF
AM HD
D
F
y
R
6
> hP
u
F dbe
]
I A
DF
F
5
I A
3
5
Rto
Fcfa
2IAH
AM HD
HD
A
HD
A
N3
,HH H D
2IAH
F DF
AM
F DF
AM
5
,HH
HD
HD
I A D
HD
HD
EIH 6
DF F
A
5
5
2
F
I A1
I A
4


# Page. 7

![Page Image](https://bcdn.docswell.com/page/GJ8KGM4LED.jpg)

(
3
• (
3 ]
1
–
– - 3A1
– 1
3 ]
•
1 3
[D
dt
– (
3 R
RN
A
–
A
1
go
3
.1
N A
) 13
6
1
[D
aM
13
3 R
) 13
3
3
uy S
1
N rh H
E(
3
e
e
fE A ) 13
sk
3
Em cS
L
i nl
3
5
2
1
4


# Page. 8

![Page Image](https://bcdn.docswell.com/page/LJLZG36Q7R.jpg)

.
•
6
3
5
2
1
4


# Page. 9

![Page Image](https://bcdn.docswell.com/page/47MVQDGK7W.jpg)

.
•
6
3
5
2
1
4


# Page. 10

![Page Image](https://bcdn.docswell.com/page/P7R284Z6J9.jpg)

) 2
• ko
2=
l
2
, 21
–
2=
2
l S
t S
= 1
) 2
ko !D 2
ko !
a
n
! ) 2
gP
D 2
ko
v!"
q
s
• 11Pa 2
x
a Du cdem n
• !" f
i D#$ % f
P
D 3
ko
f P D
Pa yr
a ]D !" s p
– ko (ko
– ko (ko
i D 23 ) 2
i D [ -ko (2
&'()*+,(
./0
12
)
ko
go
a
ko (12. i
2
D
p
DA
2


# Page. 11

![Page Image](https://bcdn.docswell.com/page/PJXD829DEX.jpg)

* ) ,HH
H
) 1 H /
,HH
H
– ru
= ,HH H _ehT
dad cT
mN N
(
#0#
lo p
)
*
.K
n
.K
H=
H=
- H 3
E
D H SMqs
H H= D H
2#
E#
H ,HH
#
H
Mtih
Q PV
#
q
A
H #
H
H
# K
- H 3
H
H
) # K
H=
H ,HH
H
_fN


# Page. 12

![Page Image](https://bcdn.docswell.com/page/3JKXKMPDED.jpg)

-.-
--
-


# Page. 13

![Page Image](https://bcdn.docswell.com/page/LE35ZYXPJ5.jpg)

2
-
d
–
–
–
M
c
a
3
• D 3
•
-3
2
fEM
A M
.EM
M
. 3 2
−∞ EM
e
S
A
g
i
A
A
Mh


# Page. 14

![Page Image](https://bcdn.docswell.com/page/8EDDR523EG.jpg)

2 .32
.
2 3
M
– 2 3
.32 3
–
f
3
.2-E
2 3
2 .32
2 .32 .2 3
2 .32
2 .32
gM
3
3
E
3
3
M
e
2 .32
D
2 .32 ka
i
A
.2-
dc


# Page. 15

![Page Image](https://bcdn.docswell.com/page/V7PXWG1PE8.jpg)

-.-
--
-


# Page. 16

![Page Image](https://bcdn.docswell.com/page/2JVL86YV7Q.jpg)

,
-
• ) .
( .
3D
– k N(
– EED.
3
NP
3
E
D EE P.
iP d
1
C
Pa
e
F
1
E
-
cf
C


# Page. 17

![Page Image](https://bcdn.docswell.com/page/5EGX5N91JL.jpg)

44
4
3 4 I >
>
44
•
–
WT •
3 4
.
>
D
44 E 3 4 I >
E U
OM
LS
3 4 I >
44
44
>
E
3 4 I
<->
<->
,
voc
44
>
<->
4
E
..
44
3 4 I
L


# Page. 18

![Page Image](https://bcdn.docswell.com/page/4JQ8ZQWN7P.jpg)

55
•
(
))[ ))R
siE
–
–
–
–
n S
55
R
3
dl
25
ERA
55
(
25
b
v
A
P
3 5
3225
.
AE
o S
tC siE
g
[
]aA
a
N
ce
P
m
S
v
n


# Page. 19

![Page Image](https://bcdn.docswell.com/page/K7423NK3J1.jpg)

-
• RNN CNN
1.
2.
3.
•


# Page. 20

![Page Image](https://bcdn.docswell.com/page/LJ1M15XZEG.jpg)

=R O OI
=R O OI 2 T
O 1 T
OI
–
7=
)AGO GB. ) N MM PO SGOTGO G
RS (,
TPLGOS
–
7=
)AGO HRB. ( 7 SGOTGO GS O S M T TPLGOS OTP (
– rzbtcy mad
k ilh[e u VP cSPUR G T RIGTva
4 R W RG O
(
)
G UMG
– - 8 5250 :
3: S
– 1 SG NP GM. )SG
TR O OI STG 1 I NP GM.
SG
– 1 SG NP GM.
STG S
PURS 1 I NP GM. (
T N GR
– 0
TR O OI STG
STG S (
N !" = 0.9, !( = 0.98 ϵ = 10,-
GIUM R
T PO
– 2RP PUT. RGS U M bw
– 0TTGOT PO RP PUT
nf]bopa RP PUT R TG/
– 6 DGM NPPT OI. ./0 = 0.1
g
s
S


# Page. 21

![Page Image](https://bcdn.docswell.com/page/GJW385N6J2.jpg)

/6
- 48 6 1=
– 6
– 6
–
6
=
O
0.1
S
E TRU
2
UTRUL
OM
A
2
B
A
2
B


# Page. 22

![Page Image](https://bcdn.docswell.com/page/4EZV8N3RJ3.jpg)

G
– , B: D
0 GD
• +
– ,DB BG
– 1G G
NMO
=
4D
B
D =
=
& ! = 0.6
=
G
=
'
BH D
+D :
=
= BD
.G
:
B
=
SR5 BB
4D
B 6


# Page. 23

![Page Image](https://bcdn.docswell.com/page/Y765P8X1EV.jpg)

2
– !"
–
. 2
– D
6
6


# Page. 24

![Page Image](https://bcdn.docswell.com/page/G75YK9XL74.jpg)

2
– !"
–
. 2
– D
6
6


# Page. 25

![Page Image](https://bcdn.docswell.com/page/9J2GW2X3JR.jpg)

.
3
–
transfomer
– Recurrent Neural Network Grammar


# Page. 26

![Page Image](https://bcdn.docswell.com/page/DEYVLY88EM.jpg)

•
– Self attention
encoder-encoder attention


# Page. 27

![Page Image](https://bcdn.docswell.com/page/VJNM4PQ978.jpg)

&
on
• on l
.
– af
–
–
.
– P
–
NDN
.
on
m
e
I
.
B
IicE
I
Xh
A


# Page. 28

![Page Image](https://bcdn.docswell.com/page/YE9NQ3D3E3.jpg)

-
03/17
i
s s I D E BE i
ha
c
•
–
P
S
P
e
N
v
i
t326
–
• .
d N
i
dgXN 26i
v
–
r
p
–
– .
–
P
•
II
gf E : B Md
cN II DIAEDgfh
i E
E i
yP i
o
gfn
b
d
D I
cv
re
i
ilv
E
hT cN:
g
gid
e
i)ai
AD
B
DAD
(&
gf
upg
rw
v
&
&
h
r
h
r
egr
hN
h
g
:
cN p
m
v P
P d
d
idN_i
r
I B
N
i
ie
d


# Page. 29

![Page Image](https://bcdn.docswell.com/page/GE8KGM3L7D.jpg)

) 0D
e e
A
•
–
Nd
A
–
xsvxi
– (
dc
e Id C
a
f
%
3X
f
f
tow m
I C
I
S
C
f
d
nEl
IG C
S
I
y
S
f
u ruv
P


# Page. 30

![Page Image](https://bcdn.docswell.com/page/LELZG3VQJR.jpg)

--
D
• .
Ab
–
–
b b
•
• .
–
–
bIl l >
b
c
f
c
b
b
N
Ab
0 Nc
0
X
N
c
c
e
b
c
x
n
e
P
b
n
eT
b
N m
l bNi N
l
T
1
1
Ab
E m
b
N
l
N
TeT
n
go
ld
swp
tv
urv