Magnetoencephalographic Signals Identify Stages in Real-Life Decision Processes

We used magnetoencephalography (MEG) to study the dynamics of neural responses in eight subjects engaged in shopping for day-to-day items from supermarket shelves. This behavior not only has personal and economic importance but also provides an example of an experience that is both personal and shared between individuals. The shopping experience enables the exploration of neural mechanisms underlying choice based on complex memories. Choosing among different brands of closely related products activated a robust sequence of signals within the first second after the presentation of the choice images. This sequence engaged first the visual cortex (80-100 ms), then as the images were analyzed, predominantly the left temporal regions (310-340 ms). At longer latency, characteristic neural activetion was found in motor speech areas (500-520 ms) for images requiring low salience choices with respect to previous (brand) memory, and in right parietal cortex for high salience choices (850-920 ms). We argue that the neural processes associated with the particular brand-choice stimulus can be separated into identifiable stages through observation of MEG responses and knowledge of functional anatomy.


INTRODUCTION
Traditionally, laboratory research into cognitive processes has attempted to simplify the context. Thus, decisions to be made, or items to be recognized, learned, or remembered are separated from the normal complex experiential web within which perception takes place and memories are made and acted upon in day-to-day life. This tradition, which dates back to Ebbinghaus, has been followed in most of the more recent approaches to imaging the neural processes engaged in decisionmaking; such as in simple lexical decision tasks (e.g. Embick et ai., 2001), or go/no-go decisions in movement tasks (e.g. Filipovic et al., 2000). The reasons are straightforward; decisions based on individual autobiographic experience are, by definition, specific. Prior personal experiences, which help shape real life decision-making, will nearly always involve episodic, semantic, and even procedural elements, and are therefore by their nature idiosyncratic.
Thus, it is difficult to devise an experimental procedure to investigate the neural correlates of such real-life situations that can be applied equivalently across many subjects. Nonetheless, to have such paradigms is highly desirable as it 242 S. BRAEUTIGAM ET AL. becomes more and more clear that the mechanisms underlying decision-making, in which diverse factors such as social context or differences in personality may be important, should be studied in contexts related to everyday life situations (e.g. Eysenck & Keane, 2000, 486-487;Oaksford, 1997).
Some interesting studies into decision-making related to real-life situations have been published. One imaging study (Maguire et al., 1997) investigated the neural mechanisms engaged when London taxi drivers plan/decide which route to take for a given destination. Emotional decisions have been studied in relation to the subject's state of anxiety using unpleasant linguistic stimuli (Tabert et al., 2001).
The approach pursued here draws on the observation that in industrialized societies, common experiences of urban dwellers can be exploited to provide stereotyped experiments with a broader 'real-life' context. Thus, most adults have some experience of supermarket shopping and choosing specific products and items from an array of competing brands. A variety of individual factors, such as age, gender, financial constraints, familiarity with the displayed items, advertising, and previous experience and/or preference for certain brands/products, influence their behavior.
Although an earlier MEG investigation of the neural correlates of advertising stimuli has been reported (Ioannides et al., 2000), the present paper is, to our knowledge, the first attempt to study the neural systems associated with the very moment that a consumer choice is being initiated and/or made. This moment is defined here as the onset of a stimulus requiring subjects to make a consumer choice. The overall aim is to explore whether the sequence of MEG responses can reveal the recruitment of the generic systems needed to effect the choice. The link with choice behavior is made through correlation between the signals and a behavioral measure, salience, calculated on the basis of prior behavioral responses. The responses were obtained from a questionnaire (as commonly used in advertising/marketing, e.g. Ambler & Burne, 1999) about brand familiarity and brand preference and are presumed to reflect the combined effect of all these potential factors affecting choice.

Subjects
Eight right-handed healthy native-Englishspeaking adults 4 females and 4 males), aged between 30 and 63 years) participated in this study. Subjects had normal or corrected-to-normal vision and signed an informed consent form before the experiment (Helsinki Declaration). The procedures were described from a pilot study on a larger number of volunteers, during which no brain signals were recorded.
All experiments were carried out in the Low Temperature Laboratory of Helsinki University of Technology. The tasks were delivered in the form of video clips projected on a screen within a magnetically shielded room. Participants were seated under the helmet-shaped detector at a distance of about 80 cm from the presentation screen. The subjects used their right hands to press keys on a small keypad according to task condition. Standard verbal instructions were given before each task. There was no emphasis on speed, but participants were asked to press the key as soon as they had reached a decision.

Choice task
Participants were presented with footage of the interior of a familiar supermarket in England where all subjects shopped, at least occasionally. The footage comprised 18 scenes of walking along the aisles and shelves. Each scene showed a selection of common consumer items belonging to a certain category (e.g. diary products, soft drinks) as placed Video Video c) Example stimuli used in the color control task (colored shapes were used in the experiment). All images had the same size on screen.
on the shelves on the day of filming. The scenes cued subjects on the category of products that would be shown in five static images after each video scene (Fig. 1). These images, constituting the actual stimuli, were shown for 5 sec each, tbllowed by a 3 sec inter-stimulus-interval. Each image showed three products of the relevant category arranged in a row of items on a neutral background (e.g. three types of butter). A total of 90 (18 x 5) one-out-of-three choices were to be made (90 x 3 270 items). The video lasted about 18 rain.
Participants were asked to indicate, after each image, which of the three items they would purchase if given the choice. Subjects expressed their choice by pressing with their index, middle, or ring finger corresponding to the left, middle or right item shown in the image. They were instructed to press with their thumb when they felt they could not make a choice. Subjects were asked to ignore price differentials and were informed that they would be given a shopping voucher (GBP 50), which could be used to purchase products selected during the MEG experiment.

Questionnaire
At the end of the experiments, participants filled out a questionnaire on which they used a 5point scale to indicate their familiarity with or usage of each of the 270 consumer items. With this ordering, the questionnaire could not have an impact on the MEG recording. For each subject, these questionnaire returns were used to calculate a measure S of the salience of a chosen item within the context of a given image according to s v, --jOy, + ), where the Vc represents the questionnaire score of the item chosen and V and V2 represent the sores of the non-chosen items. S ranges from -4 to 4. The maximum is achieved if the item chosen (V,) scores 5, whereas the two non-chosen items (V and V) each score 1. The minimum signifies the reverse situation.
Example: Subject chooses the still mineral water [middle] in the left image of Fig. 1A. Her questionnaire gives score 4 for this item and scores and 2 for the left and right items respectively.
For subsequent analysis, data epochs from the choice task were median split into two groups, according to decreasing values of S and the groups denoted by high and low respectively.
In these data, the group of high salience trials is mainly comprised of images for which the score of the item chosen, Vc, is higher than the scores corresponding to the non-chosen items V and Vz (90%). It also contains trials for which Vc V and Vc >(V + 1). The low group contains all images with equal scores, or nearly equal scores (Vc=V and Vc=Vz+l; in total 70%) and images in which at least one item has a higher score than the one chosen.

Height control task
In this task, participants were presented with a random sequence of 60 (5 sec presentation, 3 sec inter-stimulus-interval) images drawn from those used in the choice task, this time without interleaving scenes of the supermarket. The task was to indicate which of the three items (left, middle, or right) was the shortest, again by pressing one of the respective keys (pressing with the thumb was possible if subjects felt they could not discriminate between the heights of the items). The images were selected such that the shortest item could be discriminated easily, and there was an equal probability for the shortest item being in one of the three positions. The presentation lasted for about 8 rain. The use of 60 images was based on the pilot study, which had suggested a higher frequency of no-choice responses than actually occurred.

Color control task
In this task, participants were presented with a random sequence of 60 images (5 sec presentation, 3 sec inter-stimulus-interval) showing three simple geometrical objects arranged in a row. The task was to indicate which of the three objects (left, middle, or right) was red, again by pressing one of the respective keys. A small proportion of images did not feature an object in red, where subjects had to press with their thumbs. The experiment lasted about 8 min.

Data acquisition
Neuromagnetic responses following image presentation were recorded using a VectorView TM MEG system (H/mliinen, 1997), which is based around a helmet-shaped array of 102 pairs of firstorder gradiometer sensors. The outputs of each pair of detectors are most sensitive to the tangential current flow in the region directly below the detectors. The local root-mean-square (rms) signal summed over the two readings is a measure of the strength of that current.
Electronic markers on the video tape were frame-locked to the appearance of each image and fed into the data acquisition system for synchronization The data were sampled at 600Hz (0.01 to 200Hz anti-alias filter). We controlled for artifacts by recording the electro-oculogram and the electrocardiogram. We identified movement of the head by measuring the head's position before and after each experiment.

Time-series analysis
For each subject, all epochs were averaged according to task and high-low salience conditions within the interval-200 to 1000 ms (t=0 denotes stimulus onset). This gives a total of five types of average evoked response for this study (choice, height, color, choice-low-salience, and choicehigh-salience). Before the analysis, average signals were further filtered (0.2 to 30Hz) and normalized to the signal variance within a baseline interval (0 to 200ms) before stimulus arrival. The variations in the baseline signal variance were small across tasks and conditions.
Significant differences between pairs of evoked responses were sought using a time-dependent measure P(t). This measure identifies latencies where, across the subject group as a whole, significant differences between the responses from two types of evoked response are found. For each latency for which a significant difference was identified, spatial maps of root-mean-square signals were calculated as measures of the corresponding neuronal activity. The measure has been used successfully by Braeutigam et al. (2001a) and is repeated here for convenience.
where N=204 denotes the number of channels, prob is the significance level of the quantity in brackets (Batschelet, 1981), and w is the level of significance of a paired Wilcoxon test (Conover, 1980) of the pairs of evoked responses from all subjects in the ith channel.
2.8. Source analysi,,: Anatomical MRI scans were available for four male subjects. We calculated source estimates only for these four individuals. Both equivalent current dipole calculations and current density estimation were performed using Curry TM software (Curry, 1999). We obtained all density estimates using a minimum norm algorithm with L-curve regularization restricting currents to (reconstructed) cortical surfaces within a best-fit spherical volume con-ductor. Before source estimation, the data were preprocessed using the same baselines as those used for the time-series analyses and subjected to a principal component analysis. The cortical source components obtained in this way best describe the observed differences between signals. Based on the data available, an assessment of significance in source-space was not possible. In addition, identification of possibly deep sources could not be consistently achieved.

Behavioral responses
All eight subjects completed the three tasks successfully with a negligible number of missed key presses. Subjects maintained a constant head position within the accuracy of the positioning system (<5ram). A small number of epochs (<1%) had to be rejected because of heart and/or eyeblink artifacts.
In total, participants positively chose an item in 74% of all stimuli, pressing the appropriate key on average 2620 ms after stimulus onset. Overall, the results of the questionnaire mirrored the actual choices made during the experimental run in that items chosen scored significantly higher than nonchosen items (2.9 and 1.9 respectively, p <0.01). The distinction between low and high salience stimuli was reflected in a significant difference of 570ms in average response times (p <0.01). Subjects responded faster to high than to low salience stimuli. Note that the number of non-choice key presses (thumb) significantly anti-correlated with questionnaire scores across product categories ( -0.51, p <0.05). Despite this finding, the occurrence of non-choices was too varied across subjects to allow further behavioral or magnetoencephalographic analysis of these trials.
In both control tasks, decision-making was significantly faster than in the choice condition (p<0.01). Subjects responded by pressing the appropriate key 860 ms and 660 ms after stimulus onset in the height and color tasks, respectively. The occurrence of 'cannot discriminate' presses was negligible in both control tasks (< 0.5%).

Overview: Evoked responses
Based on this initial observation, statistical analysis using the measure P(t) identified four characteristic differential effects, for which neural activity is modulated by either task condition or salience. All effects are robust in signal-space, supported in at least six out of eight subjects (all four effects were present in five subjects), and statistically significant across the group of subjects.
In the choice task, the first appreciable evoked responses were observed over occipital primaryvisual cortices at about 100 ms after presentation of the images. Consistent evoked responses were observed at latencies up to about 200 ms over extra-striate and parietal regions for a given subject, but the responses varied between subjects. At longer latency, where inter-subject variability was greater, evoked activity is seen in rough sequence over anterior temporal, pre-frontal, frontal, occipital, and parietal areas during the period from 300 ms to about 1000 ms after stimulus onset ( Fig. 2A). At about 1100 ms evoked responses begin to decrease rapidly. The consistency of neural activity with respect to the inducing stimulus is lost, and the evoked responses become too weak to be analyzed (Fig. 2B).
In both control tasks, evoked responses were broadly similar to the choice tasks for latencies up to about 300 ms; starting over occipital primaryvisual cortices and extending to extra-striate, parietal, and left anterior-temporal regions. Subsequently, at around 400 ms, evoked activity decreased rapidly, consistent with a variety of studies requiring the discrimination of (complex) visual stimuli (e.g. Swithenby et al., 1998). At longer latency (>500 ms), evoked responses were associated with the motor activity required by the ensuing key presses, which had been made much earlier (860 and 660 ms, respectively) in the control tasks than in the choice condition. Such responses were highly varied across subjects and without any (stimulus-locked) pattern.
3.3 Differential responses: Choice versus control tasks The measure P(t) was used to identify intervals in time when the evoked responses elicited by the choice tasks differed significantly (P(t) <0.01) from the two control tasks across subjects. The analysis was restricted to 0 to 450 ms after stimulus onset. At longer latency, comparisons are impossible to interpret because of the onset of the motor activity found in the control tasks. Two intervals, V and T, were identified where the responses were significantly modulated by the task conditions (see Fig. 3 and Fig. 4; V and T respectively; these labels correspond to the location of main effects in each interval, i.e. primary visual and temporal cortices).
V: Neuronal activity observed over primary visual cortices at around 90 ms after stimulus onset. This was consistent with a localized (dipolar) source in primary visual cortex, within the limitations of the source analysis. Signal amplitudes were highest in the choice task, second highest in the height control task, and weakest in the color control task. The differences between the evoked amplitudes elicited by the two control tasks are also significant. T: Neuronal activity over left temporal cortices at around 325 ms after stimulus onset. The bulk of this activity is generated in left anterior temporal cortices, extending, to variable degree, to ventral and medial temporal areas. Some generators were also found in left superior/middle frontal gyri, orbital gyri, and right extra-striate cortex. Within this latency range, signal amplitudes following presentation of the images were higher when choosing an item as opposed to either determining the shortest item or the red item. When comparing the two control tasks, we found no such effect. A) Local rms-signal at latency 100 to 900 ms after the choice stimulus (rms-signal has been summed over an interval of 10 ms). The sequence shown is based on data from two subjects (first subject 100-400 ms; second subject 500-900 ms) to illustrate the significant stages of neural activity. Due to inter-subject variability, not all regions mentioned in the text are visible. For the presentation of data, the helmet-shaped array of detectors has been projected into two dimensions (left ear on the left, front at the top). Inset: The helmet-shaped array of detectors. Each plate symbolizes two orthogonal, first-order gradiometers most sensitive to directly underlying neural currents. B) Global rms-signal (arbitrary units) after the choice stimulus, summed over all subjects and channels.

Differential responses" High versus low salience stimuli (choice task)
The measure P(t) was used to identify intervals in time when the evoked responses elicited in the choice tasks differed significantly (P(t) <0.01) between the high and low salience images measured across subjects. The analysis was restricted to 0 to 1000 ms after stimulus onset-the range of latencies when most of the stimulus locked evoked activity is seen. Two intervals, F and P, were identified when the responses were significantly modulated by item salience conditions (see Fig. 3 and Fig. 4; F and P respectively; these labels correspond to the location of main effects in each interval, i.e. frontal and parietal cortices).  3: Summary of differential effects across tasks and conditions. The spatial distribution of significance is shown in the top row, where gray-scale coding symbolizes the number of channels and time points (within a given interval) for which w,(t) < 0.01 holds. The number of subjects (out of 8) is given in which each differential effect is observable. V: the same effects are observed in both choice versus color and height versus color comparisons. T: the same effect is observed in the choice versus color comparison.
F: Neuronal activity observed over left inferior frontal cortices at around 510 ms after stimulus onset. Within this latency range, signal amplitudes following low-salience stimuli were higher than those following high-salience stimuli. The results of the source analysis suggest that this activity is generated predominantly in cortical regions homologous to Broca's speech area, together with some evidence for activation of secondary visual cortices. A possible contribution from the cerebellum or brainstem as suggested by the significance plot could not be resolved. P: Neuronal activity observed over right parietal cortices at around 885 ms after stimulus onset. Within this latency range, signal amplitudes following high-salience stimuli were higher than Fig. 4: Source estimates. The images have been selected to illustrate those main source locations which are likely to give rise to the differential effects, and which have been consistently identified across subjects, i.e. occipital (V; dipolar sources), left temporal (T), left lateral frontal (F), and right posterior parietal (P). These source areas are consistent in at least three out of the four subjects for which MRIs were available. At long latency, other areas may become strongly active but there is considerable inter-subject variability. The putative functional significance has been indicated. It is noted that due to variability not all generators mentioned in the text are visible. For the purpose of presentation, each view has been rotated by a few degrees independently.
those following low-salience stimuli. Source estimates suggest a strong contribution from right, posterior parietal cortices. At this latency, nondifferential generators in secondary visual, extrastriate, orbital, and cerebellar regions are also involved to varying degrees across subjects. No salience related effects were found at latencies before F, i.e. the choice evoked responses associated with differential effects V and T above are insensitive to the high-low salience distinction. Within the limitation of the source analysis, there is some overlap between the frontal generators associated with T and F, but the frontal sources for F are clearly more lateralized than those identified forT.

DISCUSSION
Eight subjects completed an experiment designed to study the neural correlates of a type of common real-life behavior, making purchasing decisions for common consumer items during supermarket shopping. Such decisions are individual and the outcome of a variety of factors, including prior experience (learning and memory), the effects of advertising and financial constraints, as well, perhaps, as more transient processes such as mood, season, or time of day. Age, gender, class, and other broad variables are also known to affect such decision-making. The behavioral responses of subjects in our simulated shopping environment suggest that they did indeed engage in higher-order cognitive processes that reasonably might be linked to making purchasing decisions. Thus, the response times were much longer than would be expected for purely geometric discrimination tasks, and the questionnaire responses correlated strongly with the choices made during the MEG recording sessions. Also, the pattern found for non-choices (that is, when confronted with products in the three-item choice situation, none of which were of interest) points to a plausible behavior in that the subjects did not choose when familiarity/preference was low.
The implications of the salience measure are relevantit provides a comparative rating of the three consumer items in each image. Salience is only high when one item is strongly preferred to or is much more familiar than the other two. If all three items are (nearly) equal in familiarity or desirability, even if the familiarity/desirability is strong, then the salience is low. Salience is also low if the chosen item has competed successfully with two items with higher scores (presumably because of greater familiarity) on the questionnaire. Thus, in each case, low salience stimuli are those in which there may be some form of (perceived) ambiguity or perplexity in making a choice. The longer response times for low salience stimuli seem to reflect this putatively harder choice.
The MEG results averaged across all subjects reveal a robust temporal sequence of neural responses, which follow the presentation of those images requiring the expression of subject choice. This sequence emerges from two separate comparisons of choice versus control and high versus low salience.
The initial, primary visual cortex response V, at around 90 ms following stimulus onset, was compatible with the timing found in a variety of studies involving visual responses (Halgren et al., 1994;Yoneda et al., 1995). This primary visual response was stronger in the choice than in the control conditions. One interpretation would be that a complex stimulus must be strongly represented in striate cortex for subsequent higher analysis. This view would be in accordance with recent findings that a high working memory load in a task requiring visual selective attention is associated with increased activity in occipital cortices (de Fockert et al., 2001). Nevertheless, in the first control task, where product images are discriminated between on the basis of height, the signal amplitudes were also higher than those in the color discrimination task based around simple objects. Presumably, therefore, the strength of the cortical representation is related to both the complexity of the images and the demands of the task.
The later response T, at 325 ms in left temporal cortices, is also stronger in the consumer choice task. The effect is clearly induced by this task in that the responses following the height task do not differ from the responses obtained from the color task. These regions are known from a variety of intra-cranial, MEG, and functional imaging studies to be engaged in semantic processing and the memory-based interpretation of visually presented material (McCarthy et al., 1995;Braeutigam et al., 200 b: Damasio et al., 1996;Nyberg et al., 1996). The finding is thus compatible with the hypothesis that at this time, the images are being recognized and compared with data recalled from long-term memories of the relevant brands and products. Such memories must be complex with episodic and, in many cases, affective and cognitive elements. The memories may involve actual experience of using, purchasing, or seeing advertisements for the specific brands. Activity in the right extra-striate cortex may further aid object recognition as part of this process (Allison et al., 1994).
Any processes of comparison occurring at this latency, however, seem to be of a rather general character as there is no dependence on the salience measure. Working memory is likely to be involved because some of the generators locating in left frontal regions match recent observations of visual selective attention (de Fockert et al., 2001).
The differences in the responses to high and low salience images was reflected in the MEG data from around 510 ms and maps initially onto Broca's area, F. There is prior evidence of silent vocalization occurring in interpreting such visual presentations . The stronger signal from the low salience stimuli, where the subjects may face difficulty in making a decision, may indicate an increased tendency to vocalize as a strategy aiding decision-making in the absence of easily retrieved preference. Post-hoc scrutiny of images did not suggest that this putative vocalization is linked to obvious features provided in the images, such as color, shape, or linguistic (text) information.
Finally, a characteristic response is found at 885 ms in the right parietal cortex P, in high salience conditions, and thus where the subject has a strong familiarity with or preference for one of the three brands/products. Whilst this strongly lateralized parietal signal cannot be conclusively explained here, a number of insights from other sources bear on this finding.
The parietal cortex receives converging input from many sources, making it available for second-order mapping, e.g. it is engaged in relating spatial to other representations (Anderson & Zipser, 1990), notably during memory retrieval. Lesions of the right parietal affect a person's capacity to produce speech with normal prosody and emotion (Heilman et al., 1975;Ross & Mesulam, 1979). Damasio has broadened these observations into a specific 'somatic marker hypothesis' according to which damage to the right parietal cortex (Damasio, 2000;Charlton, 2000) results in anosognosia; where intentionality is profoundly damaged. High salience stimuli might relate to decisions in which the outcome is strongly consistent with some form of intention based on previous experience. In this context, it may be relevant that, at this latency, left lateral prefrontal activity was observed in three of the eight subjects, which might be related to mechanisms of reward expectancy (Watanabe, 1996). For this integrative and representational view of the responses associated with stimuli of high salience, it may be relevant that long lasting parietal waves associated with recall linguistic stimuli have been reported recently (Kane et al., 2000).
A further, but not necessarily alternative, explanation draws on the role of the right parietal in selective and sustained attention processes (Cabeza & Nyberg, 1997;Vallar, 1997), as well as higher levels of motor control (e.g. Krakauer & Ghez, 2000). Accordingly, right parietal activity may signify a (final) attentional focus on the item already chosen to visually 'hold' it during the ensuing or already initiated motor control that is necessary for the key-press. Currently, it is unclear whether this implies that such right parietal activity could, in principle, follow low salience stimuli as well.
Clearly, the neural mechanisms underlying such shopping choices are complex. They may draw on the specificity of an individual's past experience and engage many interacting psychological and social processes not explored here (notably gender) with, doubtless, appropriate brain correlates. Yet, this study has provided evidence that relevant behavioral measures (salience) that are associated with choosing consumer items may translate into differential neural activity at specific stages following stimulus presentation. In this context, it might be interesting to explore in future work the neural responses that are possibly locked to key presses, i.e. the moment a choice is being translated into action. This experimental design may be a step toward examining brain mechanisms engaged in closer approximations to real life situations. Indeed, as an increasing number of people actually shop on the Internet, real life and this study situation may come ever closer.