---
title: Visual Attention Model for Manipulating Human Attention by a Robot (ICRA2014)
tags:  #視覚的注意 #attention #human-robot interaction  
author: [Tamura Lab.](https://www.docswell.com/user/tamlab)
site: [Docswell](https://www.docswell.com/)
thumbnail: https://bcdn.docswell.com/page/ZP7RQM29E9.jpg?width=480
description: Yusuke Tamura, Shiro Yano, Hisashi Osumi, "Visual Attention Model for Manipulating Human Attention by a Robot," Proceedings of the 2014 IEEE International Conference on Robotics and Automation, pp.5307-5312, 2014.
published: March 16, 22
canonical: https://www.docswell.com/s/tamlab/ZVJJ3Z-ICRA2014_tamura
---
# Page. 1

![Page Image](https://bcdn.docswell.com/page/ZP7RQM29E9.jpg)

Visual Attention Model for
Manipulating Human Attention
by a Robot
Yusuke TAMURA1, Shiro YANO2, Hisashi OSUMI1
1 Chuo
University, JAPAN
2 Ritsumeikan University, JAPAN
2014 IEEE International Conference on Robotics & Automation, Hong Kong, June 4, 2014


# Page. 2

![Page Image](https://bcdn.docswell.com/page/NPJX5VD3EX.jpg)

MOTIVATION
Smooth interaction between humans...
2. Estimation/Prediction
1. Observation
3. Decision
2


# Page. 3

![Page Image](https://bcdn.docswell.com/page/63JK6QXNJD.jpg)

MOTIVATION
Smooth interaction between humans...
2. Estimation/Prediction
0. Elicitation
1. Observation
3. Decision
2


# Page. 4

![Page Image](https://bcdn.docswell.com/page/4LE36P5ZE5.jpg)

MOTIVATION
Smooth interaction between humans...
2. Estimation/Prediction
0. Elicitation
1. Observation
3. Decision
Manipulation of other’s attention is important for
smooth interaction
2


# Page. 5

![Page Image](https://bcdn.docswell.com/page/V8EDVLD4EG.jpg)

MOTIVATION
Smooth interaction between robot and human...
2. Estimation/Prediction
0. Elicitation
1. Observation
3. Decision
3


# Page. 6

![Page Image](https://bcdn.docswell.com/page/4V7PV2XV78.jpg)

MOTIVATION
Smooth interaction between robot and human...
2. Estimation/Prediction
0. Elicitation
1. Observation
3. Decision
A robot has to understand how its behaviors affect
human’s attention.
3


# Page. 7

![Page Image](https://bcdn.docswell.com/page/L2JVZDLREQ.jpg)

OBJECTIVE
4


# Page. 8

![Page Image](https://bcdn.docswell.com/page/Z5EG4YX6JL.jpg)

OBJECTIVE
4


# Page. 9

![Page Image](https://bcdn.docswell.com/page/V4JQGM827P.jpg)

OBJECTIVE
Development of a model of human attention
for manipulating human attention by a robot
(robot’s) behavior
attention area
environment
4


# Page. 10

![Page Image](https://bcdn.docswell.com/page/DK745V2PJ1.jpg)

RELATED WORKS
Computational visual attention model
- Saliency map (Itti et al., 1998)
- Saliency map + Face position (Cerf et al., 2008)
- Saliency map + Face position/orientation (Ozeki et al., 2011)
Input image
Intensity
Colors
Orientation Motion
Center-surround differences & normalization
Across-scale combinations & normalization
Saliency map
5


# Page. 11

![Page Image](https://bcdn.docswell.com/page/PLJ1W3MX7G.jpg)

MANIPULATION OF ATTENTION
Magicians are experts of manipulating human attention.
6


# Page. 12

![Page Image](https://bcdn.docswell.com/page/8GJW6P3KJ2.jpg)

MANIPULATION OF ATTENTION
Magicians are experts of manipulating human attention.
Magic requires a method to achieve an effect.
How the trick works
What the spectator perceives
6


# Page. 13

![Page Image](https://bcdn.docswell.com/page/W4EZYKVNJ3.jpg)

MANIPULATION OF ATTENTION
Magicians are experts of manipulating human attention.
Magic requires a method to achieve an effect.
How the trick works
What the spectator perceives
Magicians manipulate spectators’ attention in order to prevent
them from uncovering the methods by various skills.
Eye gaze (Kuhn et al., 2009)
Hand movement (Otero-Millan et al., 2011)
Face, hands, and gaze (Akashi et al., 2013)
6


# Page. 14

![Page Image](https://bcdn.docswell.com/page/KY76DQ59EV.jpg)

MODEL OF VISUAL ATTENTION
Hypothesis:
Human visual attention is influenced by
visual saliency and
others’ face, hands and gaze.
7


# Page. 15

![Page Image](https://bcdn.docswell.com/page/5G753RYDJ4.jpg)

MODEL OF VISUAL ATTENTION
Hypothesis:
Human visual attention is influenced by
visual saliency and
others’ face, hands and gaze.
Saliency map
7


# Page. 16

![Page Image](https://bcdn.docswell.com/page/V9J2ZDYMER.jpg)

MODEL OF VISUAL ATTENTION
Hypothesis:
Human visual attention is influenced by
visual saliency and
others’ face, hands and gaze.
Saliency map
Manipulation map
7


# Page. 17

![Page Image](https://bcdn.docswell.com/page/5DEYR2GPJM.jpg)

MODEL OF VISUAL ATTENTION
Hypothesis:
Human visual attention is influenced by
visual saliency and
others’ face, hands and gaze.
Saliency map
Manipulation map
Attention map
7


# Page. 18

![Page Image](https://bcdn.docswell.com/page/2VJND8GM78.jpg)

SALIENCY MAP
Visual saliency of input images
Input image
Intensity
Colors (x2)
Orientation (x4)
Motion
Center-surround differences & normalization
Across-scale combinations & normalization
Saliency map
8


# Page. 19

![Page Image](https://bcdn.docswell.com/page/GYE9GKWWJ3.jpg)

MANIPULATION MAP
Attention area (Akashi et al., 2013)
Concentration
Distraction
Manipulation map describes
- Position of face
- Position of left/right hand
- Direction of gaze
9


# Page. 20

![Page Image](https://bcdn.docswell.com/page/5GE8VRNRED.jpg)

FACE & HAND
xR
Face-Hand map
xF
F (x) =
X
i
1
p
exp
2⇡ |Si |
⇢
1
(x
2
xi )T Si 1 (x
xi )
xL
10


# Page. 21

![Page Image](https://bcdn.docswell.com/page/MLEL5KP27R.jpg)

GAZE
x
d
g xF
Gaze map
1
G(x) = p
2⇡
exp
d
✓
d(x)
2 d2
2
◆
11


# Page. 22

![Page Image](https://bcdn.docswell.com/page/54JM3249EW.jpg)

MANIPULATION MAP
+
Face-hand map
Gaze map
Manipulation map
12


# Page. 23

![Page Image](https://bcdn.docswell.com/page/ZPJRQMW979.jpg)

ATTENTION MAP
Input&image
Face1Hand&map
Gaze&map
Saliency&map
Manipulation&map
Attention map
Attention&map
13


# Page. 24

![Page Image](https://bcdn.docswell.com/page/NPEX5VP37X.jpg)

MAGIC VIDEO
A magician performs “vanishing a ball” trick.
method: a false transfer of the ball
effect: vanish of the ball
1920 x 1080
14


# Page. 25

![Page Image](https://bcdn.docswell.com/page/63EK6QDNED.jpg)

SCENES OF THE VIDEO
1: Raise RH
2: Show the ball in RH
3: Raise LH
4: “Pass” the ball from RH to LH
5: Lower RH and raise LH
6: “Vanish” the ball in LH
8: Open LH
9: Lower LH
7: Change the orientation of LH
15


# Page. 26

![Page Image](https://bcdn.docswell.com/page/4L736PRZ75.jpg)

SCENES OF THE VIDEO
1: Raise RH
2: Show the ball in RH
3: Raise LH
4: “Pass” the ball from RH to LH
5: Lower RH and raise LH
6: “Vanish” the ball in LH
8: Open LH
9: Lower LH
7: Change the orientation of LH
15


# Page. 27

![Page Image](https://bcdn.docswell.com/page/V87DVLY47G.jpg)

SCENES OF THE VIDEO
1: Raise RH
2: Show the ball in RH
3: Raise LH
4: “Pass” the ball from RH to LH
5: Lower RH and raise LH
6: “Vanish” the ball in LH
8: Open LH
9: Lower LH
7: Change the orientation of LH
15


# Page. 28

![Page Image](https://bcdn.docswell.com/page/4VJPV26VJ8.jpg)

SCENES OF THE VIDEO
1: Raise RH
2: Show the ball in RH
3: Raise LH
4: “Pass” the ball from RH to LH
5: Lower RH and raise LH
6: “Vanish” the ball in LH
8: Open LH
9: Lower LH
7: Change the orientation of LH
15


# Page. 29

![Page Image](https://bcdn.docswell.com/page/L2EVZD3R7Q.jpg)

SCENES OF THE VIDEO
1: Raise RH
2: Show the ball in RH
3: Raise LH
4: “Pass” the ball from RH to LH
5: Lower RH and raise LH
6: “Vanish” the ball in LH
8: Open LH
9: Lower LH
7: Change the orientation of LH
15


# Page. 30

![Page Image](https://bcdn.docswell.com/page/Z57G4Y36EL.jpg)

SCENES OF THE VIDEO
1: Raise RH
2: Show the ball in RH
3: Raise LH
4: “Pass” the ball from RH to LH
5: Lower RH and raise LH
6: “Vanish” the ball in LH
8: Open LH
9: Lower LH
7: Change the orientation of LH
15


# Page. 31

![Page Image](https://bcdn.docswell.com/page/V4EQGM52JP.jpg)

SCENES OF THE VIDEO
1: Raise RH
2: Show the ball in RH
3: Raise LH
4: “Pass” the ball from RH to LH
5: Lower RH and raise LH
6: “Vanish” the ball in LH
8: Open LH
9: Lower LH
7: Change the orientation of LH
15


# Page. 32

![Page Image](https://bcdn.docswell.com/page/DKJ45V9PE1.jpg)

SCENES OF THE VIDEO
1: Raise RH
2: Show the ball in RH
3: Raise LH
4: “Pass” the ball from RH to LH
5: Lower RH and raise LH
6: “Vanish” the ball in LH
8: Open LH
9: Lower LH
7: Change the orientation of LH
16


# Page. 33

![Page Image](https://bcdn.docswell.com/page/PLE1W3KXJG.jpg)

SCENES OF THE VIDEO
1: Raise RH
2: Show the ball in RH
3: Raise LH
4: “Pass” the ball from RH to LH
5: Lower RH and raise LH
6: “Vanish” the ball in LH
8: Open LH
9: Lower LH
7: Change the orientation of LH
16


# Page. 34

![Page Image](https://bcdn.docswell.com/page/8GEW6PDKE2.jpg)

SCENES OF THE VIDEO
1: Raise RH
2: Show the ball in RH
3: Raise LH
4: “Pass” the ball from RH to LH
5: Lower RH and raise LH
6: “Vanish” the ball in LH
8: Open LH
9: Lower LH
7: Change the orientation of LH
16


# Page. 35

![Page Image](https://bcdn.docswell.com/page/W47ZYKGNE3.jpg)

SCENES OF THE VIDEO
1: Raise RH
2: Show the ball in RH
3: Raise LH
4: “Pass” the ball from RH to LH
5: Lower RH and raise LH
6: “Vanish” the ball in LH
8: Open LH
9: Lower LH
7: Change the orientation of LH
16


# Page. 36

![Page Image](https://bcdn.docswell.com/page/KYJ6DQY97V.jpg)

RESULT: SALIENCY MAP
17


# Page. 37

![Page Image](https://bcdn.docswell.com/page/5GJ53RGDE4.jpg)

RESULT: SALIENCY MAP
17


# Page. 38

![Page Image](https://bcdn.docswell.com/page/4LE36PPQE5.jpg)

RESULT: MANIPULATION MAP
18


# Page. 39

![Page Image](https://bcdn.docswell.com/page/V8EDVLLWEG.jpg)

RESULT: MANIPULATION MAP
18


# Page. 40

![Page Image](https://bcdn.docswell.com/page/4V7PV22X78.jpg)

RESULT: PROPOSED MODEL
19


# Page. 41

![Page Image](https://bcdn.docswell.com/page/L2JVZDD3EQ.jpg)

RESULT: PROPOSED MODEL
19


# Page. 42

![Page Image](https://bcdn.docswell.com/page/Z5EG4YYYJL.jpg)

MEASUREMENT OF GAZE
Measuring participants’ gaze points while watching
the magic video
23-­‐inch	
  screen
Participants:
5 healthy volunteers (21-24 yrs)
Measurement:
faceLAB4.2 (60Hz)
cameras
(FCB-­‐EX480B,	
  SONY)
20


# Page. 43

![Page Image](https://bcdn.docswell.com/page/V4JQGMM67P.jpg)

EVALUATION INDEX
gaze%point:%xg
attention%value:%
Attention
mapA(xg )
evaluation%index:%
A(xg )
A(x)
saliency%value:%
Saliency
mapS(xg )
evaluation%index:%
Attention'map
attention
value:
Saliency'map
saliency
value:
evaluation index:
evaluation index:
S(xg )
S(x)
21


# Page. 44

![Page Image](https://bcdn.docswell.com/page/DK745VVMJ1.jpg)

RESULTS
Saliency map
Proposed model
Gaze point
Scene
1
2
3
4
22


# Page. 45

![Page Image](https://bcdn.docswell.com/page/PLJ1W33Y7G.jpg)

RESULTS
Saliency map
Proposed model
Gaze point
Scene
1
2
3
4
22


# Page. 46

![Page Image](https://bcdn.docswell.com/page/8GJW6PP1J2.jpg)

RESULTS
Saliency map
Proposed model
Gaze point
Scene
1
2
3
4
22


# Page. 47

![Page Image](https://bcdn.docswell.com/page/W4EZYKKXJ3.jpg)

RESULTS
Saliency map
Proposed model
Gaze point
Scene
5
6
7
8
23


# Page. 48

![Page Image](https://bcdn.docswell.com/page/KY76DQQPEV.jpg)

RESULTS
Saliency map
Proposed model
Gaze point
Scene
5
6
7
8
23


# Page. 49

![Page Image](https://bcdn.docswell.com/page/5G753RRPJ4.jpg)

RESULTS
Saliency map
Proposed model
Gaze point
Scene
5
6
7
8
23


# Page. 50

![Page Image](https://bcdn.docswell.com/page/V9J2ZDDZER.jpg)

Ratio to overall average
EVALUATION
5
**
4
3
2
1
0
Proposed
Saliency
** p<0.01
The proposed model can better explain human
visual attention than the original saliency map.
24


# Page. 51

![Page Image](https://bcdn.docswell.com/page/5DEYR22NJM.jpg)

SUMMARY
A model of human attention considering visual saliency
and human face, hands and gaze was developed.
The model can well explain human attention.
FUTURE WORKS
Developing an inverse model to calculate appropriate
behaviors for any intended attention area.
Implementing the model into a small humanoid robot.
25