Smile Through Your Fear and Sadness - Psychology

PSYCHOLOGICAL SCIENCE

Research Report

Smile Through Your Fear and
Sadness

Transmitting and Identifying Facial Expression Signals Over
a Range of Viewing Distances

Fraser W. Smith and Philippe G. Schyns

University of Glasgow

ABSTRACT—It is well established that animal communica- shaped by the processes of evolution, and so we must consider
tion signals have adapted to the evolutionary pressures of the costs and benefits of facial expression signaling in the evo-
their environment. For example, the low-frequency vo- lutionary past (Schmidt & Cohn, 2001; see also Susskind
calizations of the elephant are tailored to long-range et al., 2008).
communications, whereas the high-frequency trills of birds
are adapted to their more localized acoustic niche. Like the For example, the constraint of predator avoidance might have
voice, the human face transmits social signals about the shaped facial expressions for successful distal recognition (e.g.,
internal emotional state of the transmitter. Here, we ad- of a fearful expression), whereas the evolution of language and
dress two main issues: First, we characterized the spectral close communication might have shaped a more proximal form
composition of the facial features signaling each of the six of signaling (e.g., sadness at the distances typical of verbal
universal expressions of emotion (happiness, sadness, fear, communication). So, understanding how expression recognition
disgust, anger, and surprise). From these analyses, we operates over a range of viewing distances typical of social
then predicted and tested the effectiveness of the trans- signals transmission is a precondition to understanding the
mission of emotion signals over different viewing distances. distal or proximal nature of their possible adaptive advantage.
We reveal a gradient of recognition over viewing distances
constraining the relative adaptive usefulness of facial ex- Consider ‘‘fear,’’ for which there has been great interest in the
pressions of emotion (distal expressions are good signals neuroimaging community, in relation to its adaptive value (de
over a wide range of viewing distances; proximal expres- Gelder, Morris, & Dolan, 2005; Morris, de Gelder, Weiskrantz, &
sions are suited to closer-range communication). Dolan, 2001; Vuilleumier, Armony, Driver, & Dolan, 2003;
Whalen et al., 2004). It has been proposed that fearful stimuli
Since Eckman and Friesen’s seminal research, we know that six use a fast, subcortical pathway in the brain (e.g., Vuilleumier
universal categories of facial expression (happiness, sadness, et al., 2003; Morris et al., 2001). This pathway comprises mag-
fear, disgust, anger, and surprise) are recognized across different nocellular cells that have faster temporal resolution and are
cultures (Ekman & Friesen, 1975; Izard, 1971, 1994). Such sensitive to the coarse-scale information (i.e., low spatial fre-
expressions represent a crucial means of social communication quency, LSF) of the visual input (e.g., see Livingstone & Hubel,
in humans because they transmit mental states. That is, our own 1987). Thus, this pathway is an ideal route to process visual
facial expressions signal to others our likely intentions and, in signals quickly at a distance. In fact, Vuilleumier et al. (2003)
turn, their facial expressions indicate their intentions toward us. confirmed that the LSF of ‘‘fearful’’ faces (the facial information
It is supposed that such universal human behavior has been visible from far away) elicits a stronger response of the amygdala
(though there is still considerable debate surrounding this
Address correspondence to Fraser W. Smith, Centre for Cognitive subcortical route; e.g., see Johnson, 2005).
Neuroimaging and Department of Psychology, 58 Hillhead St.,
University of Glasgow, Glasgow G12 8QB, United Kingdom, e-mail: Note that we are not suggesting that signal transmission is
[email protected]. the only factor that might have shaped the evolution of facial
expressions. For example, Susskind et al. (2008) suggested
specific muscle configurations could confer sensory gains to
the sender (e.g., by constricting air intake in ‘‘disgust’’ vs.

1202 Copyright r 2009 Association for Psychological Science Volume 20—Number 10

Fraser W. Smith and Philippe G. Schyns

enhancing intake in ‘‘fear’’). Here, however, we focus on the of the stimulus (see Fig. 1). Thus, if a given expression should be
outstanding issue of the possible shaping of facial emotions for recognized from a certain range of viewing distances, then
transmission across varying viewing distances typical of distal the facial information critical to categorize this expression (i.e.,
and proximal social interactions. the diagnostic features for that expression) should be repre-
sented across the corresponding range of the SF spectrum.
Facial expressions, like all visual signals, are analyzed in the
brain by a number of spatial-frequency (SF) channels (e.g., see Therefore, our research agenda is, first, to characterize the
Sowden & Schyns, 2006). Depending on the evolutionary im- spectral composition of the information that is diagnostic of each
portance of proximal or distal recognition for a given expression, of the six universal expressions of emotion, plus neutral (we call
it should be represented across a corresponding range of the SF this the diagnostic SF spectrum). We derived these from a new
spectrum. This is necessary because changes in viewing dis- analysis of two 7-alternative forced-choice (7AFC) expression
tance modulate the SF content of the stimulus projecting on the experiments, each performed using the Bubbles paradigm
observer’s retina. As the stimulus moves further away, layers of (Schyns, Petro, & Smith, 2007; Smith, Cottrell, Gosselin, &
high SFs of the stimulus are progressively peeled off, ultimately Schyns, 2005; see Gosselin & Schyns, 2001). Second, from the
leaving on the retina the information represented in the low SFs diagnostic spectra, we predicted the recognition of normal

A B CDE F

Distal B′ C′ D′ E′ F′
1.07° 0.538° 0.269° 0.134° 0.07°
2°
B′ C′ D′ E′ F′
Proximal
B CDE F
A

Fig. 1. Illustration of the stimulus-generation process for the viewing-distance experiment. Examples of original images are shown on the left (labeled

‘‘A’’). All filtering operations were performed using the Laplacian Pyramid (see the text). This process is illustrated here for a distal expression (i.e., an

expression that is well recognized over a wide range of viewing distances; top two rows) and a proximal expression (i.e., an expression better suited to

close-range communication; bottom two rows). We first removed the highest spatial frequencies (SFs) present in each image (those expressed over 2
pixels), generating the images labeled ‘‘B.’’ These images were then down-sampled by a factor of 2 to obtain reduced-size images (‘‘B0’’). Note that
there was no loss of face SF information, despite the reduction in size. We repeated this process recursively to generate five reduced-size images (‘‘B0’’–
‘‘F0’’). The reduced-size images were used in the viewing-distance experiment to simulate increasing viewing distance. The numbers represent the

visual angle (height) of each stimulus in the experiment.

Volume 20—Number 10 1203

Transmitting Facial Signals Over Distances

pictures of expressions over a range of viewing distances and a proportion of information use per SF band and plotted these as
directly tested such predictions in a new 7AFC expression ex- histograms representing the diagnostic SF spectrum.
periment. We show that the diagnostic spectra for expression
recognition covary with the viewing distance from which a given Results and Discussion
expression can be successfully recognized. Specifically, in Figure 3 presents the results of the Bubbles analysis: The figure
contrast to predictions regarding the adaptive nature of ‘‘fear,’’ shows the significant face information used for each expression
we show that ‘‘happiness’’ and ‘‘surprise’’ are favored for suc- (rows) and SF band (columns 2–5). The final column presents a
cessful distal recognition. We discuss these results in the context bar graph showing the diagnostic SF spectrum for each ex-
of the pressures of communication (distal and proximal) and pression. We classified expressions as distal if the SF band usage
sensory acquisition (Susskind et al., 2008) that are proposed to was above the midpoint of the scale (2.5) and as proximal if the
subtend the evolution of facial expressions of emotions. average SF band usage was below the midpoint of the scale.
Using these criteria, we designated happiness, surprise, disgust,
BUBBLES ANALYSIS and anger as distal and fear, neutral, and sadness as proximal.

Method What consequences does such a pattern of results have for
successful recognition of given expressions at varying viewing
Data Collection distances? Distal expressions should be better recognized
Data were collated from two experiments stemming from our across a wider range of viewing distances than proximal ex-
laboratory (Schyns et al., 2007; Smith et al., 2005). Both of these pressions because, as the stimulus shrinks on the retina with
experiments utilized the same stimulus set (10 actors each greater viewing distance, the low-frequency representation of
posing the six basic facial expressions plus neutral, FACS the diagnostic features (e.g., see the bottom portion of the hap-
coded; Dailey, Cottrell, & Reilly, 2001; Ekman & Friesen, piness face, band 4, Fig. 3) becomes the only remaining stimulus
1978), the same Bubbles methodology (see Fig. 2), and the same information to resolve the task. We directly tested these pre-
task (7AFC expression categorization). The data of Smith et al. dictions in a new 7AFC expression experiment. We presented
consisted of 14 observers performing 1,200 trials per expres- whole faces of different physical sizes and asked participants to
sion, whereas the data of Schyns et al. consisted of 4 observers categorize the expression. In addition, we also manipulated the
performing 3,000 trials per expression. amount of time for which each face was presented, though we do
not focus on that here.
Analysis
On each trial of both experiments, the observer was exposed to a EXPERIMENT
random subset of SF information from the chosen face image (see
Fig. 2). The task was to categorize the sparse stimulus in terms of Method
expression. Thus, on each trial, the randomly located Gaussian
apertures define a three-dimensional mask (two of image space, Participants
the third of SF) that reveals a sparsely sampled face. To reveal Six Glasgow University students (2 males, 4 females) took part in
the facial information diagnostic of each expression, we simply the present experiment in return for a small payment. All had
summed together, independently for each SF band, all the normal or corrected-to-normal vision.
samples leading to correct responses (across all observers) for a
given expression and divided by the sum of all the samples Apparatus
shown for that expression. These probabilities were then trans- The images were displayed on a ProNitron 17/550 monitor,
formed into z scores to locate statistically significant pixels driven by an Apple Macintosh G4, by means of the MATLAB
(Chauvin, Worsley, Schyns, Arguin, & Gosselin, 2005). We then Psychophysics Toolbox (Brainard, 1997; Pelli, 1997).
filtered a representative face stimulus with the diagnostic in-
formation in each SF band to produce the effective stimulus for Stimuli, Design, and Procedure
each expression (see Fig. 3, column 1). This experiment utilized the same stimulus set as the Bubbles
experiments (though we used the closed-mouth rather than the
Diagnostic SF Spectrum open-mouth version of happiness). We simulated increasing
To measure the quantity of available information used at each SF physical distance between a signaler and a receiver by shrinking
band, for a given expression, we summed the number of sig- image size with the Laplacian Pyramid (Burt & Adelson, 1983),
nificant pixels present at each SF band for that expression and a technique that recursively removes the highest SFs of an image
divided that sum by the standard deviation of the Gaussian at the while down-sampling the residual image by a corresponding
relevant band. To compare the relative use of SF band across amount (see Fig. 1). We used the Laplacian Pyramid because it
expressions, we expressed the quantity of information in terms of removes one octave of SF between images of different sizes,
which corresponds roughly to a similar decrement in spectral

1204 Volume 20—Number 10

Fraser W. Smith and Philippe G. Schyns

SF Decomposition

Original 120 (0.36) 60 (0.7) 30 (1.4) 15 (2.9) 7.5 (5.1)
Image
(Fear)

Sample Locations (per Band)

Information Samples (per Band) Final
Stimulus

Fig. 2. Example of stimulus generation for the Bubbles experiment. First (top row), an original face was decomposed into five nonoverlapping spatial-
frequency (SF) bands of 1 octave each (starting at 120–60 cycles per face). Each band was then independently sampled with randomly positioned
Gaussian apertures (with standard deviation starting at 0.36 cycles/degree of visual angle for the higher SFs), each revealing 6 cycles per face irre-
spective of band. The second row illustrates these randomly positioned apertures for each SF band, and the third row shows the face information
samples that result from multiplying the band-specific face information (top row) with the band-specific sample locations (second row). Addition of the
randomly sampled face information from each SF band produced one stimulus image (final stimulus). Further details are available in Schyns, Petro,
and Smith (2007) and Smith, Cottrell, Gosselin, and Schyns (2005). The numbers at the top indicate the highest cycle represented in each band and (in
parentheses) the standard deviation of the Gaussian aperture at each band.

energy. To minimize any loss of information associated with tion times were combined in a fully factorial design with a total
image shrinking compared to actual changes in viewing dis- of 2,520 trials (70 images  6 distances  6 presentation times).
tance, we placed the monitor at a viewing distance of 3.3 m (see Each trial began with a fixation cross (500 ms), followed by
Loftus & Harley, 2004), ensuring a pixel density of 200 pixels presentation of a randomly selected image at a given viewing
per degree of visual angle. From a fixed viewing distance of 3.3 distance for a specified presentation time, followed by a mask
m and initial image size of 380 Â 240 pixels, the simulated (for 150 ms) consisting of phase noise with the same amplitude
viewing distances ranged from 3.3 to 105.6 m (3.3, 6.6, 13.2, spectra as that of an average face. The next trial began after the
26.4, 52.8, and 105.6 m). participant responded. Participants were instructed to judge
the expression of the face displayed to the best of their ability
We also manipulated presentation time on a logarithmic scale (training before the main experiment ensured participants could
ranging from 16 to 512 ms (16, 32, 64, 128, 256, and 512 ms). successfully classify the faces by expression).
The six different viewing distances and six different presenta-

Volume 20—Number 10 1205

Transmitting Facial Signals Over Distances

120–60 60–30 30–15 15–7.5 Proportion
All Bands 1 2 3 4 Information
Use per Band

Proportion Proportion Proportion Proportion Proportion Proportion Proportion 1.0

Happiness .5

Surprise .0 1 2 3 4 5
1.0

.5

.0
12345

1.0

Anger .5

.0 1 2 3 4 5
1.0

Disgust .5

.0 1 2 3 4 5
1.0

Sadness .5

.0 1 2 3 4 5
1.0

Neutral .5

.0 1 2 3 4 5
1.0

Fear .5

.0 1 2 3 4 5
Band

Fig. 3. Results of the Bubbles analysis. The significant face information used to classify each ex-
pression is shown in a separate row. The first column shows the diagnostic spatial-frequency (SF)
information collapsed across all the SF bands sampled during the experiment. The next four columns
show the diagnostic information from each band separately (we do not show diagnostic images for the
fifth SF band because only one expression had diagnostic information in this band). The bar graphs
show the diagnostic SF spectrum for each expression (see the main text). The numbers at the top of
the figure indicate the range of cycles per face present in each band. The numbers below these ranges
indicate the (arbitrary) number of the band.

Results and Discussion is in the lower SFs (happiness, surprise, and disgust), across
Figure 4 presents d0 sensitivity measures of performance as a several physical distances. This pattern is highlighted by a
function of expression and viewing distance averaged across significant interaction between expression and viewing dis-
presentation time and participants. tance, F(29.4, 147.1) 5 8.30, p < .001. Importantly, fear is not
well recognized across the range of viewing distances, falling in
Our prediction of proximal and distal expressions agrees with the middle of the seven expressions. We quantified the rela-
the data: Proximal expressions, whose average SF band usage is tionship between the two analyses reported by binarizing, and
in the higher SFs (neutral, fear, and sadness), tend to be more then correlating, our two experimental outcome variables: The
impaired than distal expressions, whose average SF band usage

1206 Volume 20—Number 10

Fraser W. Smith and Philippe G. Schyns

4.0 ‘‘catastrophic’’ transformation of the face happens (the mouth
opens, revealing the teeth); this transformation is, itself, repre-
Happiness sented in the LSF of the diagnostic spectra (see Fig. 3). The
3.5 Surprise reasons for such catastrophic transformations are puzzling.
Disgust Happy expressions have been proposed to signal a willingness to
Fear engage in reciprocal altruism (Schmidt & Cohn, 2001), but this
3.0 Neutral does not seem to provide, in itself, any reason for distal recog-
nition to be favored. Nonetheless, some authors have argued for
Anger a homology between facial expressions of happiness in humans
2.5 Sadness and the silent bared-teeth display in nonhuman primates (e.g.,
Sensitivity (d ′) Preuschoft, 1992; Van Hooff, 1972), even though the latter is
2.0 associated with signaling fear or appeasement (Schmidt & Cohn,
2001). Perhaps this provided a pressure toward distal recogni-
1.5 tion for happy expressions.

1.0 Surprise, on the other hand, is a transient signal indicating
something unexpected that rapidly changes into another emo-
0.5 tion (Ekman & Friesen, 1975). Why should this expression be
characterized by an LSF-rich signal? One answer comes from
0.0 the work of Susskind et al. (2008), who investigated the sensory
benefit for the sender in transmitting an emotion signal. Their
102 101 work suggests that expressing surprise should result in the
Distance (m) greatest benefit to the sender, greater even than fear, because
surprise also involves opening the mouth, thereby enhancing
Fig. 4. Sensitivity (d0) for each expression as a function of viewing dis- sensory preparedness for rapid reaction. Thus, perhaps it is
tance. Sensitivity was averaged across observers and presentation times. surprise, not fear, which needs to be recognized from far away,
It was computed independently for each expression, viewing distance, because it is the better signal of an unexpected event to which
and presentation time as the observer’s sensitivity (d0) in discriminating the receiver should also react.
each expression from the other expressions in the tested set. Results for
proximal expressions are coded with solid lines, and results for distal Such considerations must, in any case, remain speculative
expressions are coded with dashed lines. Error bars show 1 standard until we have an analysis of both how the different facial ex-
error across participants. pressions evolved their specific patterns of muscular movements
and the range of naturalistic contexts in which each expression
first variable classifies an expression’s SF usage as either distal occurs (see Schmidt & Cohn, 2001; Susskind et al., 2008). Fa-
or proximal (Experiment 1), and the second classifies each ex- cial muscle configuration is critical because it determines the SF
pression as either above or below mean recognition distance for representation of each expression and hence the range of
a sensitivity value of 2 (half the range of the sensitivity scale, viewing distances from which it can be successfully recognized.
Experiment 2). The correlation is significant (j 5 0.75, p .05; The range of natural contexts within which different expressions
see Howell, 2002). Expressions that are, on average, biased are utilized, on the other hand, speaks directly to the evolu-
toward lower SF information tend to be biased toward above- tionary function of expression signaling. Without such anal-
mean recognition distance. yses, we risk forgetting that the proximal or distal character of a
given expression may be incidental to its evolutionary function.
GENERAL DISCUSSION
We believe that there is much scope in an approach that first
We have shown that the diagnostic SF spectra predict the per- defines the signal and observer characteristics of important
formance of observers in recognizing different facial expressions evolutionary categories and then examines how the brain has
across a range of viewing distances. The psychophysical data adapted to process these characteristics.
reveal a gradient of recognition proceeding as follows: sadness,
anger, fear, disgust, surprise, and happiness. That sadness is Acknowledgments—We thank Marie Smith. This work was
poorly recognized is not surprising, because there is no obvious supported by Economic and Social Research Council Grant
survival benefit to detect it from far away. It is more surprising RES-060-25-0010.
that anger, a signal conveying threat, and fear, a signal con-
veying potential danger, are both poorly recognized across a REFERENCES
range of viewing distances. One might have expected evolu-
tionary pressures toward distal recognition in each case. Brainard, D.H. (1997). The Psychophysics Toolbox. Spatial Vision, 10,
433–436.
In contrast, happiness and surprise are our most distal ex-
pressions. Both expressions are LSF rich (see Fig. 3) and involve
the mouth more than any other expressions. Note that, in sur-
prise and happiness expressions (when the teeth are shown), a

Volume 20—Number 10 1207

Transmitting Facial Signals Over Distances

Burt, P.J., & Adelson, E.H. (1983). The Laplacian Pyramid as a Morris, J.S., de Gelder, B., Weiskrantz, L., & Dolan, R.J. (2001).
compact image code. IEEE Transactions on Communications, 31, Differential extrageniculostriate and amygdala responses to
532–540. presentation of emotional faces in a cortically blind field. Brain,
124, 1241–1252.
Chauvin, A., Worsley, K.J., Schyns, P.G., Arguin, M., & Gosselin, F.
(2006). Accurate statistical tests for smooth classification images. Pelli, D.G. (1997). The VideoToolbox software for visual psycho-
Journal of Vision, 5, 659–667. physics: Transforming numbers into movies. Spatial Vision, 10,
437–442.
Dailey, M., Cottrell, G.W., & Reilly, J. (2001). California Facial
Expressions, CAFE. Unpublished digital images, University of Preuschoft, S. (1992). Laughter and smile in Barbary macaques
California, San Diego, Computer Science and Engineering (Macaca sylvanus). Ethology, 91, 220–236.
Department, San Diego, CA.
Schmidt, K.L., & Cohn, J.F. (2001). Human facial expressions as ad-
de Gelder, B., Morris, J.S., & Dolan, R.J. (2006). Unconscious fear aptations: Evolutionary questions in facial expression research.
influences emotional awareness of faces and voices. Proceedings Yearbook of Physical Anthropology, 44, 3–24.
of the National Academy of Sciences, USA, 102, 18682–18687.
Schyns, P.G., Petro, L., & Smith, M.L. (2007). Dynamics of visual
Ekman, P., & Friesen, W.V. (1975). Unmasking the face. Englewood information integration in the brain to categorize facial expres-
Cliffs, NJ: Prentice Hall. sions. Current Biology, 17, 1580–1585.

Ekman, P., & Friesen, W.V. (1978). The Facial Action Coding System Smith, M.L., Cottrell, G.W., Gosselin, F., & Schyns, P.G. (2005).
(FACS): A technique for the measurement of facial action. Palo Transmitting and decoding facial expressions. Psychological
Alto, CA: Consulting Psychologists Press. Science, 16, 184–189.

Gosselin, F., & Schyns, P.G. (2001). Bubbles: A technique to reveal the Sowden, P.T., & Schyns, P.G. (2006). Channel surfing in the visual
use of information in recognition tasks. Vision Research, 41, brain. Trends in Cognitive Sciences, 10, 538–545.
2261–2271.
Susskind, J.M., Lee, D.H., Cusi, A., Feiman, R., Grabski, W., & An-
Howell, D.C. (2002). Statistical methods for psychology (5th ed.). Pa- derson, A.K. (2008). Expressing fear enhances sensory acquisi-
cific Grove, CA: Duxbury. tion. Nature Neuroscience, 11, 739–740.

Izard, C. (1971). The face of emotion. New York: Appleton-Century- Van Hooff, J.A.R.A.M. (1972). A comparative approach to the phy-
Crofts. logeny of laughter and smiling. In R.A. Hinde (Ed.), Non-verbal
communication (pp. 209–242). New York: Oxford University
Izard, C.E. (1994). Innate and universal facial expressions: Evidence Press.
from developmental and cross-cultural research. Psychological
Bulletin, 115, 288–299. Vuilleumier, P., Armony, J.L., Driver, J., & Dolan, R.J. (2003). Distinct
spatial frequency sensitivities for processing faces and emotional
Johnson, M.H. (2005). Subcortical face processing. Nature Reviews expressions. Nature Neuroscience, 6, 624–631.
Neuroscience, 6, 766–774.
Whalen, P.J., Kagan, J., Cook, R.G., Davis, F.C., Kim, H., Polis, S.,
Livingstone, M.S., & Hubel, D.H. (1987). Psychophysical evidence for et al. (2004). Human amygdala responsivity to masked fearful eye
separate channels for the perception of form, color, movement, whites. Science, 306, 2061.
and depth. Journal of Neuroscience, 7, 3416–3468.
(RECEIVED 10/7/08; REVISION ACCEPTED 3/22/09)
Loftus, G.R., & Harley, E.M. (2004). Why is it easier to identify
someone close than far away? Psychonomic Bulletin & Review, 12,
43–65.

1208 Volume 20—Number 10