EVA - Emotive Agents, Disordered Minds
An approach to Affective Computing
To be human is to experience subjective well-being that varies across the hours, months, and years of our lives. From time to time these variations are intense. Traceable, perhaps, to an obvious cause - the death of a child, loss of a lifelong career, winning the lottery jackpot - they intrude significantly on us. Often, however, the changes are mild, a background murmur against the press and buzz of schedules, commitments and goals. Their mild ebb and flow, exciting less attention, "just is", demanding less from our searches for cause and meaning.
Is it possible that this complex mix of temporal change in our affect can be quantified, and mapped in a way that enables models of affective computing (Picard 1997) to resemble human mood and emotion? We believe that the answer to this question is yes, and that it stands squarely before all who would recreate convincingly human mental processes in silico. At the same time, such new computational models might improve our understanding of human subjective experience per se, especially those experiences central to our sense of subjective well-being, to the pathologies of our well-being, and to how we deal with them.
Depression's Dark Shadow
If you would have me weep, first weep yourself.
Horace, Ars Poetica
Among the pathologies of human subjectivity, depression is an often devastating, yet surprisingly common, blemish on the relation between self and world. The National Institutes of Mental Health report that, per annum, some 18 million Americans suffer from depressive illness, with effects rippling not only through patients' lives but also through their workplace, families, and an overburdened health care system. The Harvard Mental Health Letter puts the number of Americans affected by depression severe enough to warrant medical attention at more than 11 million, with annual costs to the U.S. economy of US$44 billion. Statistics Canada figures for 1995 show a similar epidemiological profile, with women of all ages more likely victims than men by a factor of about 1.5-2:1, and prevalence among younger people (17-44 years) of either sex.
Although psychiatrists identify a number of depression subtypes (Brown et al. 1994, Frank et al. 1994, Lewis 1997), the common thread is a pronounced, insistent sense of despair and hopelessness. Appetite may diminish, along with energy; concentration becomes harder, memory poorer; along with thought, movement slows; activities once pleasurable may lose their interest. Depression's etiology remains poorly understood, however. A person can enter depression abruptly and for no reason apparent to them, while others exposed to similar circumstances emerge unscathed. Genetics, family environment, personality, and life experiences all seem to be involved, in a manner that is complex and still poorly understood (Hepburn and Eysenck 1989, Andreason 1997). In the Emotive Virtual Agent, or EVA, project, we are studying how self-adapting in silico entities equipped with complex patterns of variable affect might help improve the diagnosis and treatment of depression's many forms.
Clinical Narratives
The concept of narrative also figures prominently in our approach to the EVA project. This is because, universally, the first encounter between patient and physician begins with a story, in which the patient tells of the events and experiences that led them finally to seek medical help (Lumsden and Whiteside 1987). A properly told story is a major step in establishing the correct diagnosis and appropriate treatment. Physicians are trained to help patients tell the relevant story, a personal history dwelling on the time course, traits, and impact of their symptoms, the fingerprints a disease leaves on mind and body in the form of altered structure and function. A patient's history of their presenting illness is the physician's first window onto the disease they must diagnose and manage. This of course includes the clinical encounter with depression. We are therefore interested in developing computer-based processes that can combine the affective dynamics of depression with the ability to tell us the story of their major depressive episodes.
Remarkably, narrative plays a therapeutic role, as well as a diagnostic one, in the clinician's treatment of depression. As of this writing (1998), the treatment options for depression include antidepressive drugs, psychotherapy, and electroconvulsive shocks - applied alone or in suitable combination. Roughly speaking, depression responds equally well to all three of these treatments. It is easier to imagine the efficacy of a physical intervention like drugs or electricity; these after all engage the brain substance itself, presumably including those brain regions that regulate mood and arousal. But what of psychotherapy, which relies on patient and doctor sitting together and talking? How does the exploration of the patient's life work its positive effects? Could new models for affective computing improve our understanding of how "talk therapy", a social form wrapped in its own complex temporal rhythms and ceremonies (Scheflen 1973, Heath 1986), works in alleviating depression, and thereby help us improve the training of the psychotherapists?
Scaling and Complexity
Two recent discoveries about human mood and emotion have been especially important in our work with EVA. First, temporal change and variation in quantitative measures of affect have been found on all time scales thus far assessed - hours, days, weeks, months, years (e.g. Robbins and Tank 1987, Cowdry et al. 1991, Hall Jr. et al. 1991, Coombs et al. 1994, Gottschalk et al. 1995, Pezard et al. 1996, Totterdell et al. 1996). This is so for otherwise healthy people, as well as people suffering from various mental illnesses, including depression. It is, we should at once also note, not just a matter of "piling up" countless hourly experiences to make a year's worth; rather, temporal patterns characterized by all these scales of duration are taking place atop one another. Thus, while affect modulates hour by hour it is also shifting a manner that bespeaks daily rhythms, and so on (Figure 1). Even in the simplest numerical terms, then, our subjective lives are astonishingly complex.
Second, when affect is tracked over a sufficiently long period (say, daily measurements for 1-2 years) and tabulated in a simple way, a remarkable pattern emerges from this apparent complexity (Gottschalk et al. 1995). When the logarithm of I, the intensity of an experience, is graphed against the logarithm of N, the frequency or number of times that intensity occurs, the plot running from the mildest experiences (most frequent) to the strongest (least frequent) is just a straight line with a negative slope (Figure 2). This can be so only if the frequency and intensity variables N and I are connected by a so-called power-law relationship in which I is proportional to N raised to a power a. The quantity a, in turn, is simply a negative value equal to the steepness of the downward slope observed on the logarithmic plot.
It is now evident that the temporal organization of affect follows this power- law in people with depression as well as in normal subjects. The slope of the line, however, is steeper on the log-log plot for depressive patients than it is for normal subjects. In depression, the magnitude of the slope parameter, a, is about twice that found in healthy individuals (Gottschalk et al. 1995). In other words, individuals with depression are significantly depressed relatively more often than normal subjects, but show comparatively less minor variation in affect. Their subjective lives, while darkened by depression's shadow, are if anything more "orderly" than those of healthy people, who experience comparatively greatly amounts of short-term, milder affect variation. What is going on?
SOC
It turns out that the power-law behavior we have described above is surprisingly common - some would argue ubiquitous - in the natural and social worlds. Processes as disparate as earthquakes, music, city sizes, and stock market jitters all show it when their event magnitudes are tabulated against how often they happen. The physicist Per Bak and his colleagues have suggested that such power laws are the signatures of complex systems responding to external stress (see Bak 1996 for an invigorating introduction). What warrants the epithet "complex" is not the arrangement of many adjacently interacting parts that make up such systems, since these parts can be very simple, or "dumb", and the rules of interaction among the parts equally modest. Rather, complexity emerges from the way the parts and their interactions support patterns of stress dissipation that range in magnitude from events involving very few of the parts (the most common) up to those affecting the every component throughout the entire system (the least common). The behavior is self-organizing and, because it brackets all possible event magnitudes, is said to be "critical" in analogy to the "critical state" of physical systems changing phase (like water going from a liquid to steam), where fluctuations in properties like the density obey power laws. The name suggested by Bak to describe such behavior is self-organized criticality, or SOC.
The current EVA architecture, EVA 1.0, uses self-organized criticality to produce a succession of affective events that follow power-law behavior. As in the human case, events of all relevant magnitudes occur. Part of our motivation, of course, is purely heuristic: The mathematics of SOC are a concise tool for building power-law scaling into the experiences and behaviors of digital agents. We have in addition asked ourselves if there is a reasonable interpretation of SOC stress dissipation, as EVA 1.0 experiences it, in terms relevant to the human experience of affect.
Hidden Goals
To see why this might be so, consider EVA's understanding of the abstract digital world it inhabits. Like all agents worthy of the name, EVA maintains a set of goals it seeks to attain; but (in a fashion more evocative of the human condition than that of ideally adapted robots) some of these goals EVA cannot reach, no matter what. EVA 1.0's "mind" is made up of this array of goals, each with an associated affect intensity induced by frustration or stress: EVA can get only part way to any unreachable goal before something happens to frustrate and abort its efforts. It then goes back to "square one", rescheduling its efforts to achieve that goal. At any moment, each goal in this set has an associated numerical value. The larger this value, the more progress EVA has made toward reaching the goal, and the higher its current priority to EVA.
In EVA 1.0, two additional traits are common to all these unreachable goals. First, each goal is associated with closely related goals, as are these in turn, and so on throughout the entire goal set. For example, in our everyday lives the goal of earning that much-desired promotion may connect closely with our goals of improving our financial security and increasing what we believe is the esteem others accord us. When EVA's efforts to achieve a goal fail, EVA temporarily turns its attention from that goal to those closely related to it, redistributing to them the priority given to the original goal. (So, failing to get that big promotion we turn (temporarily) to other sources of fiscal security and esteem.) Eventually, of course, EVA's attention returns to that goal, but only after a transient "downtime." Discouraged, EVA seeks other, related objectives.
Affect Cascades
The way EVA handles failure has some interesting consequences. The goals to which priority is distributed might already have some level of active priority; EVA's efforts to reach them may already be underway. Now receiving increased priority and attainment effort, they in turn may fail, triggering another redistribution of priority to their related goals, and so on. (Indeed, in EVA 1.0 failure is inevitable once a priority passes a specified threshold - a simple instance of "If you want it too much, you're certain to fail.") Failures, progress, and shifted priority values cascade outward from the initiating event, potentially affecting many - in principle all - of the unreachable goals.
Although all these unreachable goals currently trigger identical responses to progress and failure, this need not be so. By assigning unequal initial priorities across the goal set, and further distinguishing among goals by means of unequal priority growth rates per progress step and affect intensities per failure, we can begin to represent personality structure by means of an influence on goal-seeking behavior and affect dynamics. We are, of course, especially interested in how depressive patterns of affect, with fewer mild downswings and more large ones, might follow from changes in such parameters of the model. Such a determination is a first step in developing novel new treatments for depression; if we understand what changes in the affect dynamics cause the depressive pattern to appear, we might be in a better position to reverse it.
Second, not all of these goals are accessible to EVA's capacity for explaining its behavior and affective experience. Some are "hidden," allowing us to study the impact of unreachable goals that are outside the frame of narrative awareness and/or not expressible in narrative terms. Events involving or originating from these hidden goals are unavailable to EVA in its search for causal stories that explain the changes in its affective life.
In drawing the connection to affective experience, we have been influenced by the observation that failure to reach goals ranks among the principal triggers of negative subjective experience (review in Parkinson et al. 1996). In EVA 1.0, the magnitude of the experience is simply proportional to the size of the entire cascade of failures and priority reassignments.
Tell Us A Story
At the start of an EVA simulation, all unreachable goals have zero initial priority. As EVA responds to its world (currently composed of other EVA agents), goal-related experiences trickle in and the priorities begin to change. The system however, is self-organizing: This initial transient period evolves to a state in which EVA's goal priorities, failures, and negative affective experiences are drawn toward the SOC power-law behavior.
When EVA fails to reach a goal and the priority collapses, EVA will try to put together a simple narrative describing the failure's causal history, as well as its effects on the importance of the other unreachable goals. Although not yet sufficiently rich to mimic the clinical narratives of depressed patients, these simple stories are nonetheless very interesting because of EVA's limited frame of causal inference. Since EVA has some unreachable goals hidden from its narrative competence, collapses and cascades ("priority avalanches") arising from or involving hidden goals are "mysterious" to EVA. They "just happen" without apparent cause.
The algorithm explain (50K JPEG) is an example of a procedure for sifting simple stories like these out of goal avalanche patterns.
EVA 1.0 simply flags events like these as "unexplained" and moves on. In the clinical setting such literal gaps are uncommon; even when people "don't know" why they feel depressed, imaginative attempts to reconstruct plausible scenarios occur frequently. We are currently at work on architectures that will allow EVA to deal more creatively with these literally unexplainable gaps in its history.
Send in the Avatars
In the psychotherapy clinic as well as everyday life, we do not allow charts of numbers or transcriptions of our stories to take our place. We present ourselves. In the clinical encounter, the paralanguage of facial expression, body gestures, and aural tonality is invaluable to the physician listening to the patient recount the history of their illness.
A principal goal of our work with EVA is a humanoid interface, in which digital characters driven by EVA agents assume human-like form and express affect and affect narratives. This is an exciting challenge. At present, most virtual agents of interest to the young field of affective computing are state transition machines with relatively few goals and simple connections between failure and the expression of frustration or disappointment (Picard 1997, Trappl and Petta 1997). With EVA, as with the human mind, we must now begin to understand how to animate characters that have complex, dynamic patterns of shifting priorities among many goals - some potentially hidden as well as unreachable.
Our hypothesis is that it is not success or failure with goals or priorities taken one by one, but rather the emergent cascading patterns of priority and failure in the goal set considered as whole, that crucially determine the general subjective tone of affective experience, and its expression through face and body. We are therefore developing a humanoid interface specification for EVA in which the animation generator takes its cues from the global pattern of activity in the goal set. Our earlier experience with the C-ASE clinical training simulator suggests that there are two routes to such humanoid visualization.
Frequently, character designers build the anatomy of virtual humans using the tools and procedures of 3D computer-aided design (e.g. Parke and Waters 1996). Stunning creations like Virtual Marilyn (Magnenat Thalmann and Thalmann 1994) and Jack (Badler 1997) show how convincing humanoid behaviors like walking, smiling, and touching can be when expressed through 3D CAD. There is at present no reason to doubt that real-time 3D models will be also be effective for EVA, although the virtual embodiments pertinent to the clinic necessarily will differ from those relevant to the television screen or the battlefield (Figure 5).
Ingryd bigger (42K JPEG).
We are also interested in visualization methods based on our experience with the C-ASE clinical training simulator (Woolridge 1995). In a move more reminiscent of video gaming than 3D industrial design, C-ASE flips the CAD paradigm of humanoid digital character animation on its head: Instead endowing a supple 3D avatar with the illusion of life, the digital agent accesses a library of 2D digital video segments of human actors. 3D mimicry is traded off against a gestural richness that, to date, no simulated face or body can match. First developed for the much simpler purpose of simulating patient responses to doctors' questions, the C-ASE method remains to be explored as a means by which agents could compose novel, narrative behaviors "on the fly".
Prospects
EVA 1.0 is, for the time being, the ultimate narcissist, centered solely on the affect it experiences contingent to its frustrations and failures. Capable of obstructing or aiding other EVA agents, EVA ties into its world through transactions that, while primitively social, are important only via its current priorities. With no overt behavior other than recounting the stories of its affective experiences, EVA 1.0 is a minimal humanoid, albeit one relevant to understanding human affect and its abnormalities.
This EVA is stultifyingly direct in its story telling. Lacking an imagination by which to leap creatively across the otherwise unexplainable gaps linked to its hidden goals, EVA peppers its narratives with "Don't know!" just when the affect cascades get interesting. Real people are not so dull. In our current work we are using methods of creativity programming (Findlay and Lumsden 1988, Kreindler and Lumsden 1994, Hofstadter 1995) to help EVA bridge those narrative gaps in a more interesting and clinically relevant manner.
Faced with uncertainty, the human mind merges allusion, metaphor, and history to understand itself (Turner 1996) - a tactic impossible for an agent locked into an eternal present: No past, no story, and EVA 1.0 has no past aside from a diary of affective events. We are now generalizing EVA's design so that its initial data structures will incorporate a biography as well as the goal set. Able then to reach into a virtual past as it moves toward the virtual future sketched fleetingly in the goal priority cascades, EVA may have better affect stories to tell.
EVA 1.0's affective life is one of negative jolts, the consequences of failure to reach prized, but unattainable, goals. Would the "flip side", positive rather than negative swings, be EVA's version of happiness? Not necessarily. Much evidence now suggests that, as the months and years of our lives flow past, human subjectivity feels the brunt of two affective processes rather than a single one with two "flip sides" (Diener and Emmons 1984). It is as if the human mind brackets the world with a capacity for positive affect running concurrently with, and in parallel to, one for negative affect. Thus positive and negative may co-occur, rather than trading off against each along a single continuum from despair to elation.
If this is so, we humans have evolved as "vector" rather than "scalar" emotive beings: Our subjectivity requires, minimally, at least two independent numerical variables - a mathematical form called a two-component vector - to describe the play of our emotive "ups" and "downs". A single quantity, or "scalar," might be adequate for positive or negative affect considered in isolation from the other, but inadequate for their combined role in human consciousness. We are hopeful that the EVA architecture, which is modular and readily extended beyond scalar dynamics, will provide a testing ground for computational agency based on vector emotive processing.
Does EVA feel? The affective phenomenology is there, embodied in the cascades of binary excitation as they wash the silicon-metal hardware's microstructure. But the spartan narratives, with their barest flicker of self-expression, as yet give no clue as to what it is like to be EVA 1.0. Not human, hardly a humanoid, EVA nevertheless is a meeting place for new ideas about our well-being and its disorders.
Institute of Medical Science, University of Toronto