Detection of Football Spoilers on Twitter

152 Views

September 26, 18

スライド概要

This is a presentation slide in CollabTech2018.

profile-image

明治大学 総合数理学部 先端メディアサイエンス学科 中村聡史研究室

シェア

埋め込む »CMSなどでJSが使えない場合

関連スライド

各ページのテキスト
2.

SNSs SNSs are widely prevalent as a communication tool. 2

3.

SNSs SNSs are widely prevalent as a communication tool. For example Message exchange, browsing a friend's doing now and so on. 3

4.

Sharing Events in Real-time 4

5.

Sharing Events in Real-time 5

6.

Sharing Events in Real-time 6

7.

Spoiler Problem Recording 7 I have to go to work!!

8.

Spoiler Problem Going back to home... 8 I wonder what Sam is doing now.

9.

Spoiler Problem Lando Calrissian is return!!!!! Spoiler!! Obi-Wan Kenobi is appear!!! Optimistic people want to hear the bad news first, while pessimists ask for the good. Realists just start drinking. I don’t have a dirty mind, I have a sexy imagination. 9

10.

Spoiler Problem Not the best result we expected. Sports is very shocking. Bélgica 0-0 Japón Anything could happen. Tonight is the night. I’ll keep my fingers crossed for Belgian always look down Japanese. This is kalma. 10

11.

Example(2) 11 By another interested event The festival seems interesting!

12.

Example(2) They will easily encounter spoilers. 12

13.

Example(3) 13 By push notifications @you My baby has come!!. Tapped!

14.

Example(3) Spoiler... 14

15.

Example(4) Recording 15 I'll go to bed now. VS 3:00 AM in Japan

16.

Example(4) In the next morning, half asleep... 16

17.

Example(4) Not the best result we expected. Spoiler!! Bélgica 0-0 Japón Anything could happen. Tonight is the night. I’ll keep my fingers crossed for Belgian always look down Japanese. This is kalma. 17

18.

Example(5) Because of time difference 18

19.

Example(5) An example of time difference spoiler. 19

20.

Solution for Spoiler Problem People sometimes stay away from the Internet and choose a self imposed isolation. 20

21.

Solution for Spoiler Problem 21 People sometimes stay away from the Internet and choose a self imposed isolation. It affects the maintenance of relationships with their friends.

22.

Solution for Spoiler Problem 22 People sometimes stay away from the Internet and choose a self imposed isolation. We need automatic method for hiding spoilers. It affects the maintenance of relationships with their friends.

23.

Related Researches Spoiler Alert [Boyd-Graber et al. 2013] targeted movie reviews and conducted an evaluation of machine learning approaches to find spoilers in social media posts. Spoiler detection in TV program tweets [Jeon et al. 2015] proposed a method of detecting spoilers in comments on Twitter about television programs. 23

24.

Related Researches Spoiler Alert [Boyd-Graber et al. 2013] targeted movie reviews and conducted an evaluation of machine learning approaches to find spoilers in social media posts. How to detect sports spoilers on SNSs with high accuracy was Spoiler detection in TVinsufficient program tweets [Jeon et al. 2015] proposed a method of detecting spoilers in comments on Twitter about television programs. 24

25.

[Our goal in this work] Examining methods for detecting football spoilers with high accuracy on Twitter. 25

26.

Contents 1. Generating spoiler dataset 2. Experiment of spoiler detection 26

27.

Generating Spoiler Dataset 27 We collect tweets of 9 games by the Japan national football team. Match Score Day 2015 Women’s World Cup “Japan vs. England” JPN 2 – 1 ENG 07/01/15 2015 Women’s World Cup “Japan vs. United States” JPN 2 – 5 USA 07/05/15 2015 EAFF East Asian Cup “Japan vs. South Korea” JPN 1 – 1 KOR 08/05/15 2015 Women’s EAFF East Asian Cup “Japan vs. China” JPN 2 – 0 CHN 08/08/15 2015 EAFF East Asian Cup “Japan vs. China” JPN 1 – 1 CHN 08/09/15 World Cup Qualifiers “Japan vs. Cambodia” JPN 3 – 0 KHM 09/03/15 World Cup Qualifiers “Japan vs. Afghanistan” JPN 6 – 0 AFG 09/08/15 Friendlies “Japan vs. Iran” JPN 1 – 1 IRI 10/13/15 World Cup Qualifiers “Japan vs. Singapore” JPN 3 – 0 SIN 11/12/15

28.

Generating Spoiler Dataset 5 labelers helped us label the tweets. Evaluation system 28

29.

Generating Spoiler Dataset 5 labelers helped us label the tweets. Evaluation system 29

30.

Generating Spoiler Dataset 30 5 labelers helped us label the tweets. 1000 tweets × 9 games × 5 labelers = 45000 data Evaluation system

31.

Some Examples of The Dataset 31 We defined spoiler’s as tweets labeled more than half number. We got 1651 spoiler tweets. Tweet Elapsed time Label “Ooooh! Kagawa scores a goal!!!” 20 spoiler “Already allowed two goals (´Д` )” 0 spoiler “Now kick off” 0 non-spoiler “Hmm. A missed pass is no good” 20 non-spoiler “I saw a sweeping victory for the first time in a very long time” 120 spoiler

32.

Some Examples of The Dataset 32 We defined spoiler’s as tweets labeled more than half number. We got 1651 spoiler tweets. Tweet “Ooooh! Kagawa scores a goal!!!” Words themselves Elapsed timelike spoiler... Label are 20 spoiler “Already allowed two goals (´Д` )” 0 spoiler “Now kick off” 0 non-spoiler “Hmm. A missed pass is no good” 20 non-spoiler “I saw a sweeping victory for the first time in a very long time” 120 spoiler

33.

Terms Used Frequently in Spoiler Winning Terms 33 Losing TF-IDF Terms Tying TF-IDF Terms TF-IDF [Player] 0.742 [Player] 0.531 [Player] 0.627 [Team] 0.442 [Team] 0.475 [Team] 0.552 0.261 Break the deadlock 0.238 Goal 0.226 [Num]th points 0.173 Allowing goals 0.238 Tie 0.201 [Num]points 0.131 Parry 0.224 Match 0.151 [Num] 0.117 Second 0.112 [Num]-[Num] 0.136 Win 0.106 Score 0.112 End 0.125 Match 0.099 Too 0.112 National 0.110 Goal ※[Player]: player names, [Team]: team names, [Num]: number

34.

Terms Used Frequently in Spoiler Winning Terms 34 Losing TF-IDF Terms Tying TF-IDF Terms TF-IDF [Player] 0.742 [Player] 0.531 [Player] 0.627 [Team] 0.442 [Team] 0.475 [Team] 0.552 0.261 Break the deadlock 0.238 Goal 0.226 [Num]th points 0.173 Allowing goals 0.238 Tie 0.201 [Num]points 0.131 Parry 0.224 Match 0.151 [Num] 0.117 Second 0.112 [Num]-[Num] 0.136 Win 0.106 Score 0.112 End 0.125 Match 0.099 Too 0.112 National 0.110 Goal ※[Player]: player names, [Team]: team names, [Num]: number Some terms have the characteristics of spoiler directly.

35.

Terms Used Frequently in Spoiler Winning Terms 35 Losing TF-IDF Terms Tying TF-IDF Terms TF-IDF [Player] 0.742 [Player] 0.531 [Player] 0.627 [Team] 0.442 [Team] 0.475 [Team] 0.552 0.261 Break the deadlock 0.238 Goal 0.226 [Num]th points 0.173 Allowing goals 0.238 Tie 0.201 [Num]points 0.131 Parry 0.224 Match 0.151 [Num] 0.117 Second 0.112 [Num]-[Num] 0.136 Win 0.106 Score 0.112 End 0.125 Match 0.099 Too 0.112 National 0.110 Goal ※[Player]: player names, [Team]: team names, [Num]: number

36.

Terms Used Frequently in Spoiler Winning Terms 36 Losing TF-IDF Terms Tying TF-IDF Terms TF-IDF [Player] 0.742 [Player] 0.531 [Player] 0.627 [Team] 0.442 [Team] 0.475 [Team] 0.552 0.261 Break the deadlock 0.238 Goal 0.226 [Num]th points 0.173 Allowing goals 0.238 Tie 0.201 [Num]points 0.131 Parry 0.224 Match 0.151 [Num] 0.117 Second 0.112 [Num]-[Num] 0.136 Win 0.106 Score 0.112 End 0.125 Match 0.099 Too 0.112 National 0.110 Goal ※[Player]: player names, [Team]: team names, [Num]: number

37.

Terms Used Frequently in Spoiler Winning Terms 37 Losing TF-IDF Terms Tying TF-IDF Terms TF-IDF [Player] 0.742 [Player] 0.531 [Player] 0.627 [Team] 0.442 [Team] 0.475 [Team] 0.552 0.261 Break the deadlock 0.238 Goal 0.226 [Num]th points 0.173 Allowing goals 0.238 Tie 0.201 [Num]points 0.131 Parry 0.224 Match 0.151 [Num] 0.117 Second 0.112 [Num]-[Num] 0.136 Win 0.106 Score 0.112 End 0.125 Match 0.099 Too 0.112 National 0.110 Goal The terms of spoilers differed depending on the game status.

38.

Experiment of Spoiler Detection 38 We examined methods for detecting spoilers with high accuracy. Compared methods 3 word based methods Evaluation points Precision, Recall, F-measure

39.

Our Proposed method(SVM with Status of Match) 39 The method uses different learning models of SVM depending on game status. Kick-off 0-1 0-2 Game 1: 1-0 Game 2: 1-0 Game 3: 1-1 Full time

40.

Our Proposed method(SVM with Status of Match) 40 The method uses different learning models of SVM depending on game status. Kick-off 0-1 0-2 Game 1: 1-0 Game 2: Winning model 1-1 1-0 Game 3: Winning model Full time

41.

Our Proposed method(SVM with Status of Match) 41 The method uses different learning models of SVM depending on game status. Kick-off 0-1 Game 1: 0-2 Losing model Full time Losing model 1-0 Game 2: Winning model 1-1 1-0 Game 3: Winning model

42.

Our Proposed method(SVM with Status of Match) 42 The method uses different learning models of SVM depending on game status. 0-1 Kick-off Game 1: 0-2 Losing model Tying model Full time Losing model 1-0 Game 2: Tying model 1-1 1-0 Game 3: Tying model Winning model Winning model Tying model

43.

Our Proposed method(SVM with Status of Match) 43 The method uses different learning models of SVM depending on game status. 0-1 Kick-off Game 1: 0-2 Losing model Tying model Full time Losing model 1-0 Game 2: Tying model Winning model 1-1 1-0 Game 3: Target Game: Tying model Tying model Winning model Detect Tying model 1-0 Winning model Detect (A system monitors game status)

44.

Methods Used in Experiment 44 3 word-based methods (player names, team names, number are normalized) Pattern matching (from Jeon’s research) [Jeon et al. 2015] Tweets that containing spoiler keywords were judged as spoilers Spoiler keywords were terms with 0.100 or higher of a TF-IDF value SVM (from Jeon’s research) [Jeon et al. 2015] The SVM model detected spoilers Features for SVM were generated using a Bag of Words of each tweet SVM with Status of Match (Our proposed method) The SVM model detected spoilers Features for SVM were generated using the Bag of Words of each tweet

45.

Bag of Words 45 We conducted word division After it, we selected noun, verb, adjective, adverb, interjection and generated the Bag of Words’ features For example Features for SVM of tweets about “Japan is winning” and “Fight! Japan!!” Tweet Japan win fight Japan is winning 1 1 0 Fight! Japan!! 1 0 1

46.

How to calculate the result value Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 46 Game 8 Game 9

47.

How to calculate the result value Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 47 Game 8 Generate spoiler keywords or SVM model Game 9

48.

How to calculate the result value Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 48 Game 8 Detect spoiler keywords or SVM model Game 9

49.

How to calculate the result value Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 49 Game 8 Detect spoiler keywords or SVM model Value 1 Game 9

50.

How to calculate the result value Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 Generate spoiler keywords or SVM model Value 1 50 Game 8 Game 9

51.

How to calculate the result value Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 Detect spoiler keywords or SVM model Value 1 51 Game 8 Game 9

52.

How to calculate the result value Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 Detect spoiler keywords or SVM model Value 1 Value 2 52 Game 8 Game 9

53.

How to calculate the result value Game 1 Value 1 Value 2 Game 2 Value 3 Game 3 Value 4 Game 4 Value 5 Game 5 Value 6 Game 6 Value 7 53 Game 7 Value 8 Value 9 Game 8 Game 9

54.

How to calculate the result value Game 1 Value 1 Value 2 Game 2 Value 3 Game 3 Value 4 Game 4 Value 5 9 Game 5 Value 6 Game 6 Value 7 54 Game 7 Value 8 Value 9 Game 8 Game 9 The result = value

55.

Result 55 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611

56.

The evaluation points Precision = Recall = 𝒏𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒔𝒑𝒐𝒊𝒍𝒆𝒓′ 𝒔 𝒅𝒆𝒕𝒆𝒄𝒕𝒆𝒅 𝒄𝒐𝒓𝒓𝒆𝒄𝒕𝒍𝒚 𝒏𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒅𝒆𝒕𝒆𝒄𝒕𝒆𝒅 𝒕𝒘𝒆𝒆𝒕𝒔 𝒏𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒔𝒑𝒐𝒊𝒍𝒆𝒓′ 𝒔 𝒅𝒆𝒕𝒆𝒄𝒕𝒆𝒅 𝒄𝒐𝒓𝒓𝒆𝒄𝒕𝒍𝒚 𝒏𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒔𝒑𝒐𝒊𝒍𝒆𝒓 𝒕𝒘𝒆𝒆𝒕𝒔 F-measure = 2・(𝐩𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧)・(𝐫𝐞𝐜𝐚𝐥𝐥) (𝐩𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧)+(𝐫𝐞𝐜𝐚𝐥𝐥) 56

57.

Result 57 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611

58.

Result 58 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611

59.

Result 59 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611 Pattern matching often detect a spoiler only from the player name. SVM related detect a spoiler not only from the player name. e.g.: “Kagawa’s missed pass is scary because of the heavy turf” Pattern matching: Spoiler SVM related: Non-spoiler

60.

Result 60 Method Precision Recall F-measure 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611 Pattern matching Pattern matching often detect a spoiler only from the player name. SVM related detect a spoiler not only from the player name. e.g.: “Kagawa’s missed pass is scary because of the heavy turf” Pattern matching: Spoiler SVM related: Non-spoiler

61.

Result 61 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611

62.

Result 62 Method Pattern matching Precision Recall F-measure 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611

63.

Result 63 Method Pattern matching Precision Recall F-measure 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611 Mistakenly learned tweets by SVM were no longer learned for every time zone by SVM with Status of Match. e.g.: “I saw a national team match for the first time in a very long time!!” SVM: Spoiler SVM with Status of Match: Non-spoiler

64.

Result 64 Method Pattern matching Precision Recall F-measure 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611 Mistakenly learned tweets by SVM were no longer learned for every time zone by SVM with Status of Match. e.g.: “I saw a national team match for the first time in a very long time!!” Cause: Mistakenly learned tweets such as “I saw a sweeping victory for the first time in a very long time”

65.

Result 65 Method Pattern matching Precision Recall F-measure 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611 Mistakenly learned tweets by SVM were no longer learned for every time zone by SVM with Status of Match. Ex) “I saw a national team match for the first time in a very long time!!” The detection accuracy can be improved by considering Cause: Mistakenly learned of “I saw a sweeping victory thefirst statuses matches. for the time in a of very long time”

66.

Result 66 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611

67.

Result 67 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611 We don’t set up the standard for labeling. e.g.: “Nagatomo got a cramp!” was judged to be spoilers in the dataset

68.

Generating new spoiler dataset A labeler predicts the game‘s result. (We defined more confident tweet is more crucial spoiler) I wonder if Haaril has an idea. [120 minutes elapsed] Confidence: Prediction: Win Draw Lose Not sure Japan is netted the rebound. [40 minutes elapsed] Confidence: Prediction: Win Draw Loooooooooong pass [20 minutes elapsed] Lose Not sure Choose spoilers ↓ Predict the results and it’s confidence 68

69.

Result(new dataset) 69 We regarded tweets about 50 or more average confidence as spoiler and less than 50 average confidence as non-spoiler. Method Precision Recall F-measure SVM with Status of Match (old dataset) 0.698 0.565 0.611 SVM with Status of Match (new dataset) 0.831 0.880 0.852

70.

Result(new dataset) 70 We regarded tweets about 50 or more average confidence as spoiler and less than 50 average confidence as non-spoiler. Method Precision Recall F-measure SVM with Status of Match (old dataset) 0.698 0.565 0.611 SVM with Status of Match (new dataset) 0.831 0.880 0.852

71.

Result(new dataset) 71 We regarded tweets about 50 or more average confidence as spoiler and less than 50 average confidence as non-spoiler. Method Precision Recall F-measure SVM with judging only0.698 more crucial spoiler, Status ofIfMatch 0.565 0.611 (old it dataset) can be judged with more 0.8 accuracy. SVM with Status of Match (new dataset) 0.831 0.880 0.852

72.

Application to real-world contents Normal digest Non-spoiler digest 72

73.

Application to real-world contents Number of spoilers? 73 Non-spoiler digest eliminate eliminate time

74.

Current Problem Precision was about 0.5 if keeping high recall. Precision-Recall curve 74

75.

Current Problem 75 Precision was about 0.5 if keeping high recall. Precision-Recall curve Future task: We need to think other methods to design a high-recall model first and then create models realizing higher precision.

76.

Findings 1. Terms themselves have the characteristics of spoiler and the terms of spoilers differed depending on the game status. 2. SVM with Status of Match method is the best about the detection accuracy. 3. If judging only more crucial spoiler, it can be judged with more 0.8 accuracy. 76

77.

Summarize Examining methods for detecting spoilers with high accuracy on Twitter.  The detection accuracy can be improved by considering the statuses of matches.  If judging only more crucial spoiler, it can be judged with more 0.8 accuracy. [Future works]  Improvement of detection method.  Implementation of Twitter client. 77