-- Views
June 30, 20
スライド概要
2020/06/19
Deep Learning JP:
http://deeplearning.jp/seminar-2/
DL輪読会資料
8FBLMZ4VQFSWJTFE%JTFOUBOHMFNFOU DEEP LEARNING JP [DL Papers] 8JUIPVU$PNQSPNJTFT QSFTFOUFS,FJ"LV[BXB http://deeplearning.jp/
ॻࢽใ w λΠτϧ8FBLMZ4VQFSWJTFE%JTFOUBOHMFNFOU8JUIPVU$PNQSPNJTFT w ஶऀ'SBODFTDP-PDBUFMMP #FO1PPMF (VOOBS3ÅUTDI #FSOIBSE4DIÖMLPQG 0MJWJFS#BDIFN .JDIBFM5TDIBOOFO w 'JSTUͷ-PDBUFMMP͞Μl$IBMMFOHJOH$PNNPO"TTVNQUJPOTJOUIF 6OTVQFSWJTFE-FBSOJOHPG%JTFOUBOHMFE3FQSFTFOUBUJPOTz CFTUQBQFSBXBSE BU*$.- ͷਓ w ձٞ*$.- w Ұطͱ͏͍ͰݴଘʹڀݚൺͯσʔληοτʹඞཁͳࢣڭʢΞϊςʔγϣϯʣͷগͳ ͍ɼཧੳ͖ͷEJTFOUBOHMFNFOUख๏ͷఏҊ
എܠ%JTFOUBOHMFE3FQSFTFOUBUJPO w %JTFOUBOHMFE3FQSFTFOUBUJPOɼػցਂֶशʹ͓͚Δॏཁͳ֓೦ w ΏΔ͍ఆٛ؍ଌσʔλΛߏ͢ΔҼࢠΛ͏·͘ଊ͍͑ͯΔදݱ w จʹΑͬͯ৭ʑͳఆ͕ٛ͋Δɽྫ &BTUXPPEBOE8JMMJBNT 4IVFUBM w -PDBUFMMPFUBM -PDBUFMMPFUBM հจ ͷఆٛ#JKFDUJWFCBTFE w ਅͷҼࢠz = [z1, . . . , zd]TͱɼϞσϧͷજࡏมz ̂ = [z1̂ , . . . , zd̂ ]TͷؒʹʢॱংΛআ͍ͯʣ ̂ ରͷରԠ͕ଘࡏ͢Δͱ͖ʹɼz%JTFOUBOHMFE3FQSFTFOUBUJPOͰ͋Δ w ̂ )Λຬͨ͢ ͋ΔؔfͱQFSNVUBUJPOπ͕ଘࡏ͠ɼͯ͢ͷiʹରͯ͠ɼzi = fi(zπ(i) w ҙʢൃදऀͷཧղʣਅͷҼࢠzɼΛ࠷େʹ͢Δ͔ͭɼਓؒͷʹײԊ͏Ҽࢠ ΛzਅͷzͱݺΜͰ͍Δ͚ͩʢ࣮ࡍʹγϛϡϨʔλʹΞΫηεͰ͖Δ߹͋Δ͚Ͳʜʣ
എ͠ͳࢣڭܠͷ%JTFOUBOHMFNFOUෆՄೳ
<-PDBUFMMPFUBM>
w
ཧతͳઆ໌ਅͷʢEJTFOUBOHMF͞ΕͨʣҼࢠͷP(z)ʹରͯ͠ҎԼΛຬͨ͢f͕ແʹଘࡏ
J p(z) = p( f(z))ʢೋͭͷ֬มz, f(z)ͷ͕Ұகʣ
∂fi(u)
JJ ͯ͢ͷ࣍ݩi, jʹ͍ͭͯ
for a . e . u ∈ supp(z)
∂uj
w
ཁɼP( f(z))ͱ͍͏·͘͠ͳ͍ʢFOUBOHMF͍ͯ͠Δʣ͕ແଘࡏͯ͠ɼͦͷp( f(z))ͱp(z)ࣝผෆՄೳֶशʹ
ΑͬͯͲͪΒ͕ಘΒΕΔ͔อূͰ͖ͳ͍
w
ྫ શͳͰ͠ͳࢣڭ%JTFOUBOHMFNFOUΛୡ͠Α͏ͱ͢Δβ7"& 'BDUPS7"&ͳͲ͕͋Δ
w
BHHSFHBUFEQPTUFSJPSq(z) = q(z | x)pdata(x)dxͷ֤͕࣍ݩಠཱʹͳΔΑ͏ͳ੍Λ༩͍͑ͯΔ
∫
w
Ծʹq(z)ͷ֤͕࣍ݩಠཱͰɼͦΕ͕p(z)ʢͷ֤࣍ݩʣͱCJKFDUJWFͳؔΛ͔࣋ͭɼp( f(z))ͱCJKFDUJWFͳؔΛ࣋
͔ͭʹҰக͢Δ͔อূͰ͖ͳ͍ɽͭ·Γɼ֤࣍ݩͷಠཱੑΛอূͰ͖ͯͦΕ͕ਓؒͷͱײ߹͢Δอূͳ͍ɽ
w
ҙp(z)ਅͷҼࢠͷɽϞσϧʢ7"&ʣͷࣄલͰͳ͍
എ͠ͳࢣڭܠͷ%JTFOUBOHMFNFOUෆՄೳ <-PDBUFMMPFUBM> w w ࣮ݧతͳઆ໌શͳͰ͠ͳࢣڭϞσϧબ͕͍͠ w ֶशࡁΈϞσϧͷදͲ͕ݱͷఔʮEJTFOUBOHMFNFOU͞Ε͍ͯΔ͔ʯॳظͳͲʹ͘ڧґଘʢࠨਤʣ w ͠ͳࢣڭͷࢦඪʢ&-#0࠶ߏ ࠩޡ,-߲ͳͲʣͱʮEJTFOUBOHMFNFOUͷࢦඪʯʹ૬ؔͳ͍ʢӈਤʣ w β7"&ͳͲͷطଘڀݚϞσϧબʹҼࢠͷϥϕϧΛ͍ͬͯΔ ༨ஊl3FQSPEVDJOHUIFTFFYQFSJNFOUTSFRVJSFTBQQSPYJNBUFMZ(16ZFBST /7*%*"1 z
എܠऑʹࢣڭΑΔ%JTFOUBOHMFNFOU w -PDBUFMMPFUBMɼ%JTFOUBOHMFNFOUʹ͓͍ͯɼᶃશͳ͠ͳࢣڭෆՄೳᶄ͋ࢣڭ Γߴίετͳ͜ͱ͔Βɼऑࢣڭͷར༻Λਪͨ͠ w ͦࢣڭΕͧΕͷ؍ଌ͕ͲͷཁૉΛ͔࣋ͭͱ͍͏ϥϕϧ w ऑࢣڭશͳࢣڭΑΓίετͷ͍ϥϕϧɽ৭ʑͳܗଶ͕͋Γ͏Δ w ྫ 4IVFUBMʹΑΔྨ
ऑࢣڭͷྫ)PTPZB w ਤ%͋Δάϧʔϓ͕ڞ௨ͷҼࢠΛ࣋ͭͱΘ͔ͬͯΔͱ͖ʢऑࢣڭʣʹɼڞ௨ͷҼࢠʹରԠ͢Δ જࡏม[ΛɼͦͷҼࢠΛ࣋ͭ؍ଌͷάϧʔϓ͔Βਪ͠ɼͦͷଞͷҼࢠ֤ը૾͔Βਪ͢Δ w ҙೋͭͷҼࢠʢάϧʔϓͰڞ༗ඇڞ༗ʣΛ%JTFOUBOHMF͢Δख๏Ͱ͋Γɼβ7"&ͷΑ͏ʹ શͯͷ֤͕࣍ݩʑҼࢠʹରԠ͢ΔͷͱҟͳΔ
հจͷ֓ཁ w ϖΞσʔλ(x1, x2)Λ༻͍ͨEJTFOUBOHMFEදֶݱश w σʔλʹର͢ΔԾఆɼσʔλऩूίετ͕طଘڀݚΑΓখ͍͞ w (x1, x2)ʹର͢Δ੍ɼڞ௨͢Δk ≥ 1ݸͷཁૉΛ࣋ͭ͜ͱɽ۩ମతʹԿΛڞ༗ ͍ͯ͠ΔͳͲͷϥϕϧ͍Βͳ͍ w ্هͷϖΞσʔλ͕EJTFOUBOHMFEදݱͷֶशʹेͳ͜ͱΛཧతʹࣔ͠ɼ·ͨ ࣮༻తͳख๏ΛఏҊ͢Δ w ఏҊख๏Ͱ֫ಘͨ͠EJTFOUBOHMFEදݱͷޙଓλεΫʹ͓͚Δ༗ޮੑΛ࣮ࣔ͢Ͱݧ w w ޙଓλεΫઢܗΫϥεྨɼڞมྔγϑτͷͰݩͷ൚ԽੑೳͳͲ ಛʹڞมྔγϑτͱEJTFOUBOHMFNFOUͷؔΛ࣮ݧతʹࣔͨ͠ͷ͓ͦΒ͘ॳ
հจͷ֓ཁ w ϖΞσʔλ(x1, x2)Λ༻͍ͨEJTFOUBOHMFEදֶݱश w σʔλʹର͢ΔԾఆɼσʔλऩूίετ͕طଘڀݚΑΓখ͍͞ w (x1, x2)ʹର͢Δ੍ɼڞ௨͢Δk ≥ 1ݸͷཁૉΛ࣋ͭ͜ͱɽ۩ମతʹԿΛڞ༗ ͍ͯ͠ΔͳͲͷϥϕϧ͍Βͳ͍ w ্هͷϖΞσʔλ͕EJTFOUBOHMFEදݱͷֶशʹेͳ͜ͱΛཧతʹࣔ͠ɼ·ͨ ࣮༻తͳख๏ΛఏҊ͢Δ w ఏҊख๏Ͱ֫ಘͨ͠EJTFOUBOHMFEදݱͷޙଓλεΫʹ͓͚Δ༗ޮੑΛ࣮ࣔ͢Ͱݧ w w ޙଓλεΫઢܗΫϥεྨɼڞมྔγϑτͷͰݩͷ൚ԽੑೳͳͲ ಛʹڞมྔγϑτͱEJTFOUBOHMFNFOUͷؔΛ࣮ݧతʹࣔͨ͠ͷ͓ͦΒ͘ॳ
σʔλʹର͢ΔԾఆ w σʔλͷਅͷੜաఔΛҎԼ ͷΑ͏ʹԾఆ͍ͯ͠Δ p(z) = d Πi=1p(zi), ⋆ p(z̃) = d Πi=1p(z̃i) ⋆ S ∼ p(S), x1 = g (z), x2 = g ( f(z, z̃, S)) w fɼi ∉ SͷཁૉΛz̃iʹஔ ͢ΔΑ͏ͳૢ࡞ʢਤΛࢀর
հจͷ֓ཁ w ϖΞσʔλ(x1, x2)Λ༻͍ͨEJTFOUBOHMFEදֶݱश w σʔλʹର͢ΔԾఆɼσʔλऩूίετ͕طଘڀݚΑΓখ͍͞ w (x1, x2)ʹର͢Δ੍ɼڞ௨͢Δk ≥ 1ݸͷཁૉΛ࣋ͭ͜ͱɽ۩ମతʹԿΛڞ༗ ͍ͯ͠ΔͳͲͷϥϕϧ͍Βͳ͍ w ্هͷϖΞσʔλ͕EJTFOUBOHMFEදݱͷֶशʹेͳ͜ͱΛཧతʹࣔ͠ɼ·ͨ ࣮༻తͳख๏ΛఏҊ͢Δ w ఏҊख๏Ͱ֫ಘͨ͠EJTFOUBOHMFEදݱͷޙଓλεΫʹ͓͚Δ༗ޮੑΛ࣮ࣔ͢Ͱݧ w w ޙଓλεΫઢܗΫϥεྨɼڞมྔγϑτͷͰݩͷ൚ԽੑೳͳͲ ಛʹڞมྔγϑτͱEJTFOUBOHMFNFOUͷؔΛ࣮ݧతʹࣔͨ͠ͷ͓ͦΒ͘ॳ
ੳఆཧʢཁʣ ∫∫∫ w ԾఆϖΞσʔλ(x1, x2) ∼ p(x1, x2) = p(x1, x2, z, z̃, S)dzdz̃dS͕ແʹಘΒΕΔ w Ծఆڞ༗͢ΔҼࢠͷ | S | = k͕༩͑ΒΕ͍ͯΔ w ԾఆS, S′ ∼ p(S)ʹରͯ͠ɼP(S ∩ S′ = {i}) > 0, ∀i ∈ [d] w ݁ՌੜϞσϧΛֶशͯ͠ࣄલp(zî )ɼσίʔμʔq(x1 | z)̂ ɼp(S)̂ ͕ಘΒΕΔͱ͢Δɽ BHHSFHBUFEQPTUFSJPSq(z)̂ = q(z ̂ | x1)p(x1)dxCJKFDUJWJUZͷҙຯͰ%JTFOUBOHMF͞Ε͍ͯΔ ∫
ੳͬ͘͟Γͨ͠ূ໌ͷࢦ w ·ͣSʢx1, x2ʹڞ௨͢ΔҼࢠʣΛݻఆֶͯ͠शͨ͠߹Λߟ͑Δɽ ̂ ͱzi∉T ̂ ͕ಘΒΕΔ w ͜ͷஈ֊ͰɼҼࢠzi∈Sͱzi∉SʹରԠ͢Δzi∈T w ཁɼڞ௨͢ΔҼࢠͦΕҎ֎ͷҼࢠͷ%JTFOUBOHMFNFOU͕Ͱ͖Δ w छͷEJTFOUBOHMFNFOUͱ͍͏ҙຯͰ)PTPZBͱࣅ͍ͯΔɽσʔλʹର͢ΔԾఆɼϖΞ σʔλ(x1, x2) ∼ p(x1, x2 | S)͕ಘΒΕΔɼͱ͍͏ͷͰଟಉ͡ w ࣍ʹɼSΛϥϯμϜʹαϯϓϦϯάͨ͠߹Λߟ͑Δ w ڞ௨͢ΔҼࢠ͕ϥϯμϜʹݱΕΔͷͰɼશͯͷҼࢠ͕EJTFOUBOHMFNFOU͞ΕΔ w ʢ͜͜ͷূ໌গ͍͑ͯ͠ͳ͍෦͕͋Δɽʣ
ఏҊख๏ w ࣮ͷํ w ਅͷࣄޙͰ p (zi | x1) = p (zi | x2) p (zi | x1) ≠ p (zi | x2) ∀i ∈ S ͕ΓཱͭͷͰɼ͜ͷ੍ΛϞσ ∀i ∈ S̄ ϧͷࣄޙʹ༩͍͑ͨ w ͔͠͠ɼڞ௨͢Δཁૉͷk͕ʢఆͰʣ༩͑ΒΕͳ͍ͱɼS͕ਪఆͰ͖ͳ͍ɽ ࣮༻্ɼkΘ͔Βͳ͍͔͠Εͳ͍͠ɼఆͰͳ͍ʢϖΞ͝ͱʹҧ͏͔ ͠Εͳ͍ʣ͔͠Εͳ͍ w ͦ͜ͰɼkʢͱSʣώϡʔϦεςΟοΫͰϖΞ͝ͱʹਪఆ͢Δ
ఏҊख๏ max 𝔼(x1,x2)𝔼q̃ϕ(ẑ | x1) log (pθ (x1 | z)̂ ) ϕ,θ +𝔼q̃ϕ(ẑ | x2) log (pθ (x2 | z)̂ ) −βDKL (q̃ϕ (ẑ | | x1) | p(z)̂ ) XIFSF q̃ϕ (zî | x1) = a (qϕ (zî | x1), qϕ (zî | x2)) q̃ϕ (zî | x1) = qϕ (zî | x1) −βDKL (q̃ϕ (ẑ | | x2) | p(z)̂ ) w ϖΞ(x1, x2)ʹରͯ͠ɼͦΕͧΕ࠶ߏͱ,-߲Λద༻ʢ7"&෩ʣ w ̂ qϕ (zî | x1), qϕ (zî | x2)͕ಉ͡ʹͳΔΑ͏ʹฏؔۉaΛͤ·טΔ ਪɼڞ௨͢Δཁૉi ∈ Sʹؔͯ͠ w w ฏؔۉɼͷΛߟ͑Δ͔ʢ)PTPZBʣੵͷΛߟ͑Δ͔ʢ#PVDIBDPVSUFUBMʣ ڞ௨͢Δཁૉ͔Ͳ͏͔ɼ,-߲ͷେ͖͞δi = DKL (qϕ (zî | x1) ∥qϕ (zî | x2))͔ΒώϡʔϦεςΟοΫʹஅ ∀i ∈ S ̂ FMTF
հจͷ֓ཁ w ϖΞσʔλ(x1, x2)Λ༻͍ͨEJTFOUBOHMFEදֶݱश w σʔλʹର͢ΔԾఆɼσʔλऩूίετ͕طଘڀݚΑΓখ͍͞ w (x1, x2)ʹର͢Δ੍ɼڞ௨͢Δk ≥ 1ݸͷཁૉΛ࣋ͭ͜ͱɽ۩ମతʹԿΛڞ༗ ͍ͯ͠ΔͳͲͷϥϕϧ͍Βͳ͍ w ্هͷϖΞσʔλ͕EJTFOUBOHMFEදݱͷֶशʹेͳ͜ͱΛཧతʹࣔ͠ɼ·ͨ ࣮༻తͳख๏ΛఏҊ͢Δ w ఏҊख๏Ͱ֫ಘͨ͠EJTFOUBOHMFEදݱͷޙଓλεΫʹ͓͚Δ༗ޮੑΛ࣮ࣔ͢Ͱݧ w w ޙଓλεΫઢܗΫϥεྨɼڞมྔγϑτͷͰݩͷ൚ԽੑೳͳͲ ಛʹڞมྔγϑτͱEJTFOUBOHMFNFOUͷؔΛ࣮ݧతʹࣔͨ͠ͷ͓ͦΒ͘ॳ
࣮ݧ w w σʔληοτE4QSJUFT $BST% 4NBMM/03# .1*% w Ҽࢠ͕શͯΞϊςʔγϣϯ͞Ε͍ͯΔʢධՁʹͷΈ͏ʣ w ؍ଌҼࢠ͔Βܾఆతͳࣸ૾͕ͱΕΔʢҼࢠҎ֎ͷϊΠζ͕ͳ͍ʣ ϕʔεϥΠϯ w ͠ͳࢣڭβ7"& "OOFBMFE7"& 'BDUPS7"& β5$7"& %*17"& w ऑࢣڭʢBWFSBHF͕ؔҧ͏ೋͭʣ w (7"& )PTPZB .-7"& #PVDIBDPVSUFUBM
࣮͠ͳࢣڭݧख๏ͱͷൺֱ w Ϟσϧ ͕ఏҊख๏ɼϞσϧબͳ͠ͷ߹ฏۉతʹߴ͍είΞ w ͜ΕΒͷਤϞσϧબͳ͕ͩ͠ɼ͠͠ͳࢣڭͷख๏͕ϞσϧબͷͨΊʹҼ ࢠϥϕϧใΛͬͯɼ$PNFUJUJWFͳੑೳ
࣮ݧ4VQFSWJTJPOQFSGPSNBODF5SBEFP⒎T w தԝͱӈͷਤ͕໘ന͍ w ӈͷਤͰɼL ڞ༗͞ΕΔ ཁૉͷ ΛϥϯμϜʹͨ͠ ͱ͖ʹɼLΛΒͳ͍Ͱ BEBQUJWFʹਪఆ͢ΔఏҊख ๏ VOLPXO ͱɼҰ͚ͭͩਖ਼ ղͷڞ༗͞ΕΔཁૉΛͬ ͨऑࢣڭख๏(7"& LOPXO Λൺֱ͍ͯ͠Δ w ෆશͳࢣڭΛ༩͑ΒΕͨ (7"&ΑΓɼLʹؔͯ͠ શʹBEBQUJWFͳఏҊख๏ͷ ํ͕ྑ͍
࣮ݧ%PXOTUSFBN5BTLT w ࠨਤɼఏҊख๏Ͱ࠶ߏͱࠩޡઢܗΫϥεྨͷੑೳʹ૬͕ؔݟΒΕͨ͜ͱΛද͢ w w ಄ʹ͓ͨ͠͠Α͏ʹɼβ7"&ͰͰ͖ͳ͍ தԝͷਤɼਓతʹ࡞Γग़ͨ͠DPWBSJBUFTIJGUͷͱͰͷ൚Խੑೳͱɼࠨ࣠ͷࢦඪʢ࠶ߏͲͳࠩޡʣͷ૬ؔؔΛද͢
࣮ݧ%PXOTUSFBN5BTLT w 'BJSOFTT"CTUSBDU7JTVBM3FBTPOJOHͰɼ࠶ߏ֤ͱࠩޡλεΫͷࢦඪͰ ૬͕ؔݟΒΕͨ w Ͱ͠ͳࢣڭϞσϧબ͕Ͱ͖Δ
ײ w ͱͯ໘ന͍จͩͬͨͱݸਓతʹࢥ͍·͢ w ੳ͕͔ͬ͠Γͯ͠Δ w ख๏γϯϓϧ w ݁Ռ͕͍ڧ w কདྷ՝ͱͯ͠ɼSFBMXPSMEEBUBͷఏҊख๏ͷద༻͕͋͛ΒΕ͍ͯΔ͕ɼͳʹؾΔͱ͜Ζ w ఏҊख๏ɼ݁ہkඞཁͱ͠ͳ͍ͷͰɼϖΞ(x1, x2)͕ҰͭҎ্ͷཁૉΛڞ༗͍ͯ͠Εద༻Ͱ͖Δ w ͦΕऑͳࢣڭͷ͔ɼશͳ͑ͯ͠ͱ͠ͳࢣڭΔͷ͔ɽ w ʮϖΞ(x1, x2)͕ҰͭҎ্ͷཁૉΛڞ༗͍ͯ͠Δ͔ʯͱ͍͏Ξϊςʔγϣϯ࣮༻తͳͷ͔ɽଞͷʢίεύͷྑ ͍ʣऑࢣڭͷ༩͑ํ͋Δͷ͔
3FGFSFODFT w &BTUXPPE $JBOBOE8JMMJBNT $ISJT"GSBNFXPSLGPSUIFRVBOUJUBUJWFFWBMVBUJPOPGEJTFOUBOHMFE SFQSFTFOUBUJPOT*$-3 w -PDBUFMMP ' #BVFS 4 -VDJD . (FMMZ 4 4DIÖMLPQG # BOE#BDIFN 0$IBMMFOHJOHDPNNPO BTTVNQUJPOTJOUIFVOTVQFSWJTFEMFBSOJOHPGEJTFOUBOHMFESFQSFTFOUBUJPOT*O*OUFSOBUJPOBM$POGFSFODF PO.BDIJOF-FBSOJOH w #PVDIBDPVSU % 5PNJPLB 3 BOE/PXP[JO 4.VMUJMFWFMWBSJBUJPOBMBVUPFODPEFS-FBSOJOHEJTFOUBOHMFE SFQSFTFOUBUJPOTGSPNHSPVQFEPCTFSWBUJPOT*O"""*$POGFSFODFPO"SUJpDJBM*OUFMMJHFODF w )PTPZB )(SPVQCBTFEMFBSOJOHPGEJTFOUBOHMFESFQSFTFOUBUJPOTXJUIHFOFSBMJ[BCJMJUZGPSOPWFM DPOUFOUT*O1SPDFFEJOHTPGUIF5XFOUZ&JHIUI*OUFSOBUJPOBM+PJOU$POGFSFODFPO"SUJpDJBM*OUFMMJHFODF *+$"* QQr w 4IV 3 $IFO : ,VNBS " &SNPO 4 BOE1PPMF #8FBLMZTVQFSWJTFEEJTFOUBOHMFNFOUXJUI HVBSBOUFFT*OUFSOBUJPOBM$POGFSFODFPO-FBSOJOH3FQSFTFOUBUJPOT