The Reading the Mind in the Eyes test (RME) was developed by Simon Baron-Cohen to test aspects of his extreme male brain theory of autism. Recently, it has become popular with researchers studying affective empathy and autism, and is now available online in various places, mostly as a diagnostic tool for adults on the autism spectrum.  It has been revised, and the current version contains 36 items, each with four possible responses in a multiple choice format. Unfortunately, even in the revised version, the test items are highly gender biased which could affect its accuracy even though it has decent test-retest reliability.  I have analyzed the content of the test items, and broken them down by gender in order to highlight this problem and bring it to the attention of psychology researchers.

Restricted range of female images

Here are examples of the range of male eyes depicted in the RME:

The images are cropped black and white stills of actors in movies.  These are not taken from real life, and are not representative of the population although the items are balanced in terms of number of male and female eyes (18 female and 18 male).  There is a much greater range of ages represented by the male faces as compared to the female faces.  This may be due to the restricted pool (movies) that the items were drawn from.  All the female eyes are model-like in their symmetry, and all are made up with eye makeup and have plucked eyebrows.  In other words, all the female eyes are conforming to a gender stereotype of feminine beauty as seen in movies.

Restricted range of female emotional expression

I made a list of all the response choices (77 different choices) and found that there were only 48 different choices presented to the test subject as possible responses for the female eyes while there were 54 different choices presented for the male eyes.  Thus the range of possible emotions for the female eyes was restricted as compared to the male eyes.

Similarly, among the correct responses, there were 18 different correct responses for the male eyes (a unique response for each item), and only 15 for the female eyes since three of the female items (“fantasizing”, “preoccupied”, and “interested”) were duplicated.  Again, a greater emotional range  was available to describe the male eyes.  There was apparently no effort made to ensure parity between the range and types of emotions expressed in the male and female items.

Sexualized content of female items

Among the correct responses for female eyes, we find: “desire”, “flirtatious”, “interested” (twice), and “playful”.  Similar emotions in the male eyes are described as “anticipating” and “friendly”.  The correct responses for the female eyes have a strong bias toward expressing sexual interest directed toward the viewer.  This perception is reinforced by the camera angles and lighting that were used in the movies from which the items were drawn.

Also among the correct responses for the female eyes are: “contemplative”, “fantasizing” (twice), “preoccupied” (twice), and “reflective”.  The eyes in those items are not expressing much emotion.  Similar items for male eyes were described as “pensive” and “thoughtful”.  All of these indicate inwardly directed attention.  Fantasizing is not really an emotional state; it is an activity during which one could experience a variety of emotional states. When someone's attention is directed inward, less communication about their emotional state is directed outward making the actual emotional states depicted in these items ambiguous.

Answering these ambiguous items correctly is more about eliminating answers than making a distinction between “thoughtful” and “fantasizing”.  Also, ambiguous items may muddy the results since test subjects may be more likely to project their own emotional states or be more suggestible to response items. Six female items (out of 18) were ambiguous as compared to two (out of 18) for the male items, so fully 33% of the female items were ambiguous compared to 11% of the male items.  This cluster of similar items among the correct answers for the female eyes further restricts the range of emotions expressed. 

Using the RME to look for gender differences in empathy could be very problematic.  One study concludes that men have more trouble reading the emotions of women than of other men based on their performance on the RME.  Given that there is a significantly higher percentage of ambiguous items among the female eyes, and the range of emotions expressed in the female items is limited, I'm not sure that is a valid conclusion.  If the RME is to be used for testing hypotheses about autism and gender or gender and affective empathy, it needs to be redesigned with careful attention to gender parity in both the range of different ages and facial types presented as well as the range and types of emotions represented.

Sources

Baron-Cohen S. The extreme male brain theory of autism. Trends in Cognitive Science. 2002; 6(6):248-254.

Baron-Cohen S, Wheelwright S, Hill J, Raste Y, Plumb I.  “Reading the Mind in the Eyes” test revised version: a study with normal adults, and adults with Asperger syndrome or high-functioning autism. Journal of Child Psychology and Psychiatry. 2001; 42: 241-251.

Best C, Minshew N, Strauss M. Gender discrimination of eyes and mouths by individuals with autism. Autism Research. 2010; 3(2): 88-93.

Fernández-Abascal E, Cabello R, Fernández-Berrocal P, Baron-Cohen S. Test-retest reliability of the 'Reading the Mind in the Eyes' test: a one-year follow-up study. Molecular Autism. 2013; 4:33.

Schiffer B, Pawliczek C, Müller B, Gizewski E, Walter H. Why Don't Men Understand Women? Altered Neural Networks for Reading the Language of Male and Female Eyes. PLOS One. 8(4): e60278.