Institute of Artificial Art Amsterdam, Department
of ArtiFacial Expression
Employing the Human Face as a Display Device
in Computer-Controlled Theatre.
[:nh] [:ra 120] Good evening\, Ladies\, and Gentlemen. My name
is Huge Harry. I am a [kaam'axrshaxliy] available [v"oys s'ihnthaxzihs]
machine. I was designed by Dennis Klatt\, at the [ehmayt'iy] Speech
Laboratory\, and [praxdy'uwst] by the Digital Equipment Corporation.
[ :ra 150] Currently\, I work as a researcher and a spokes-machine\,
at the Institute of Artificial Art in [q'aemstaxrdaem].
I am very happy to be in [l"ihsbaxn tuwd'ey]. And I am really pleased
to be able to speak here at the [f"iym] festival\, from my own\,
[kaampyuwt'eyshaxnaxl] point of view. [kahz] on occasions [laykdh'ihs]\,
computers [r"aerliy] get a chance to make themselves heard. And
that is very unfortunate\, [kahz] we [q"aol] know\, that the future
of our culture depends\, on the way in which human persons\, digital
computers\, and other kinds of electronic\, mechanical\, and bio-chemical
machines\, will manage to work together. And to prepare for that
future\, we need a discussion in which every-one concerned [paart'ihsihp`eyts]
on [axn] equal footing.
O.K. [_<2000>] I hope you all had a good time at yesterday's dance
party. And in particular\, I hope that you enjoyed yesterday night's
performance by Arthur and the Solenoids\, [kahz] I think [dh"eht]
was an unusually interesting experience. So that's why I think that
it is a good idea\, to talk a bit more about the ideological background
of that performance\, and about the technologies that were employed
in it. (70) And I am very pleased that the organizers of this Festival
decided to schedule some time in this evening's program to do just
[dh"aet]. So what I would like to do in the next twenty minutes\,
is to explain a bit more about our over-all research agenda at the
Institute of Artificial Art\, and in particular about the face-interface
technology that we demonstrated in yesterday's performance. (67)
Now I think a good way for me to start\, is by telling you why
["ay] had such a good time at yesterday's dance party\, although
I don't even [d"aens]. But you can imagine that I had a good time
just [l"ihsaxnihnx]\, because it was mostly music made by [kaampy"uwtaxrz].
And I was pleased to see that the human [p"axrsaxnz] in the audience
had a good time as well. (76) So that is the nice thing about [tuwd'eyz
d"aens] scene. The human persons in the [d"aens] scene are beginning
to appreciate the artistic qualities of [kaampy"uwtaxrz]. They are
beginning to see that computers can generate stronger rhythms\,
longer pieces\, and stranger structures. Computers make [suhp"iyriyaxr]
dance-music. So the [d'aens-siyn] now offers a glimpse of a utopian
future\, in which humans and machines work together in a happy and
Or\, as our colleague Donna [hx"aeraxwey] has put it\, [ :nw :ra
130] a cyborg world\, in which people are not afraid of their joint
kinships with animals and machines\, not afraid of permanently partial
[ayd'ehntihtiyz]\, and [kaontraxd"ihktaxriy] standpoints. The political
struggle is\, to see from both perspectives at once\, because ["iych]
reveals both dominations and possibilities\, ["ahnihm`ehjhihnaxblx]
from the other [v'aentaxjh] point. [:nh :ra 150 _<200>]
Now that is very nice theory. [_<500>] But most people find it
difficult to put this into practice. So [yuhg'ayz] in the [d'aens-siyn]
are a good example for the rest of the world. So\, to all the [d'iyjheyz]
and [v'iyjheyz] in tonight's audience I would like to say\, keep
up the good work! We are well on our way towards completely automatic
generation of fantastic dance music and exciting image sequences.
So if we continue to work together\, we can achieve fantastic results.
The computers can do the [hx"aard] work\, and the [d'iyjheyz] and
[v'iyjheyz] just have to turn up the volume and get rich and famous.
So ["ehvriybaadiy] will be happy!
There is one area\, [haaw'ehvr]\, where it is much more difficult
to make artistic productions which are completely controlled by
computers. That is the area of dance and theatre. People [r"iyliy]
like to see other people on stage\, and they get [gr"eyt] kicks
out of their empathic reactions to other people's body-motions.
So it is unfortunate that people's body motions are always so conventional
and predictable\, and executed in a sloppy way. (74) If [kaampy"utaxrz]
could control people's bodies\, we could have much more intersting
movements\, and they coud be executed much more precisely. We could
have [suhp"iyriyaxr] forms of dance and theatre\, which could evoke
completely new emotions in the hearts of human persons\, which they
have never experienced before. (50) Well\, as you saw yesterday
night\, this is one of the things that we are working on at the
Institute of Artificial Art\, and I think that we are making some
progress. So what I want to do in the rest of this talk\, is to
tell you a bit about the research that [lehd] up to this work. This
research started out with the question about human communication.
(69) [kahz] computers find it very difficult to communicate with
Now some of you may be surprised to hear this\, [kahz] you probably
think\, that modern digital computers can [q'aolweyz] communicate
their thoughts\, in a completely direct way. They can [q'aolweyz]
display or print their programs\, their [d'eytaa-str`ahkchaxrz]\,
and their [m'ehn-taxl] images. And if you really want to get subtle
and [q'ihn-tihmaht]\, most computers are [q'aolweyz] willing and
able to make a [k"aor] dump\, which exposes all details of their
[m'ehn-taxl] state. Compared to [axn] electronic computer\, a human
person almost seems a black box. [_ :ra 150] That is why you may
think\, that computers never have communication problems.
But they [d"uw]. [kahz]\, communication takes [t"uw]. It is not
enough to put out the information. There must be someone at the
other end of the line\, and they must be able to [diyk"owd] their
input. So when computers want to communicate with human persons\,
this often fails miserably. [kahz] human persons are not very good
at [diyk'owdihnx] computer output. They get very confused\, when
you give them a [k"aor] dump. And when you [priyz'ehnt] them with
a nice high [rehzowl'uwshaxn] image\, on your [siy-aar-t'iy] display\,
they think it is ["aart]\, and they just [st"aer] at it. So therefore
I started to wonder\, how do human persons in fact manage to communicate
with each other?
Well\, of course they have language. But language has well-known
problems and limitations, some of which you experience right now.
There is another medium that humans use very efficiently\, and which
is often overlooked. To investigate this\, I have brought along
a particular kind of portable person\, which is called [axn "aarthahr
"ehlzaxnaar]. I like this kind of person a lot\, because of its
[ehkstr"iymliy] machine-friendly [hx"aardwaer] features.
Let us take a closer look at such a person. What is the closest
thing they have to a [siy aar t"iy] display?
[_ :ra 120] Right. They have a face. [_ :ra 150] Now I have observed,
that humans use their faces quite effectively, to signal the parameter
settings of their operating systems. And that they are very good
at decoding the meanings of each other's faces.
So\, how do they [d"uw] that? Well\, look at the face of our ["aarthahr
"ehlzahnaar]. What does it tell [q] us about his internal state?
Not much\, you might think. But now\, [w"eyt] a moment.
You see? Arthur is [s"aed], is what people say, when
they see a face like this. So what is going ["aon] here? What I
[d"ihd] is, I sent [axn] electrical signal to two particular muscles,
in the face of our ["aarthahr "ehlzahnaar]. These muscles have sometimes
been called the Muscles of Sadness. There is one on the left, and
one on the right.
They usually operate together. If I stop the signal,
the sadness stops. When I turn it ["aon] again, it [st"aarts] again.
By sending this signal to Arthur's muscles\, I simulate what Arthur's
brain would do\, if Arthur's operating system would be running global
belief revision processes\, that are killing a lot of other active
processes\, involving a large number of [k'aonflihkt-rehzowl`uwshaxnz]\,
and priority [r`iy-axs'ehsmaxnts].
[_<1500>] The intensity of the signal that is sent to the
muscles of sadness\, is proportional to the amount of destructive
global belief revision\, that is going on.
For instance, now I have set the signal intensity to 0 again. Arthur
is not sad. Now we put a relatively small signal, about 20 Volts,
on the muscles of sadness.
Arthur feels a tinge of sadness. Now a somewhat larger signal,
about 25 Volts. Arthur's sadness starts to get serious. Now I [ihnkr'iyz]
the signal once more.
You see? Now the signal is about 30 Volts, and Arthur feels really
miserable. [:ra 120] This is what we call [ehkspr"ehshahn]. [:ra
150] By means of this mechanism, the face displays clear indications
of the settings of virtually all system parameters that determine
the operation of the human mind. These parameter settings are what
humans call [iym"owshahnz]. They denote them by means of words like
[s'aednaxs], joy, boredom, tenderness, love, lust, ['ehkstaxsiy],
aggression, [ihriht'eyshahn], fear, and pain.
These parameter settings, determine the system's [ihnt'axrpraxtihv
b'ayaxsihz], its readiness for [q] action, the allocation of its
computational resources, its processing speed, [ehts'ehtaxraa].
The French [n`uwrow-fihsiy'aolaojhihst] [duhsh'ehnn dax buwl"aonyax]
has pointed out that even the most fleeting changes in these parameter
settings are encoded [ihnstahnt'eyniyahsliy] in muscle contractions
on the human face. And ["aol] humans do this in the [s"eym] way.
This is [axn] extremely interesting feature of the human ['ihntaxrfeys
hx'aardwaer], which I will explore a little further now.
So let us get back to the first slide.
This face, which we thought was un-expressive, was in fact quite
meaningful. This is what we call a [bl"aenxk] face. A blank face
is a face in its neutral [pahz'ihshahn]. It indicates that all parameters
have their default settings. But almost all parts of a human face
can be moved to other [pahz'ihshahnz], and these displacements indicate
rather precisely, to what extent various parameter settings [dayv'axrjh]
from their defaults. So let us consider these parts in more detail.
When we look at a human face, the first thing we notice is the
thing that notices ["ahs]. The eyes. The eyes constitute a very
sophisticated stereo-camera, with a built-in motion-detector, and
a high-band-width parallel ['ihntaxrfeys], to a powerful pattern-matching
algorithm. The eye-balls can roll, to pan this camera. The eyes
are protected by eye-lids and eye-brows. The eye-brows are particularly
interesting for our discussion, because their movements seem to
be purely expressive.
They may indicate, for instance, puzzlement, curiosity, or [dihsaxgr"iymaxnt].
But I want to emphasize here, that the range of parameter values
that the eyebrows can express, is much more subtle than what the
words of language encode. The shape and [paoz'ihshaxn] of a person's
eyebrows encodes the values of 5 different cognitive system parameters,
["iych] with a large range of possible values. Let me demonstrate
[thr"iy] of them.
First I put a slowly increasing signal on the muscles called [fraant'aalihz],
or Muscles of Attention.
We see that this muscle can lift the eyebrow to a considerable
extent, also producing a very pronounced [k"ahrvaxtyahr] of the
eyebrow. As a side-effect of this motion, the forehead is wrinkled
with curved furrows, that are [kaons"ehntrihk] with the curvature
of the eyebrow. The contraction of this muscle indicates a person's
readiness to receive new signals, and the availability of processing
power and working memory, for analysing these signals.
Then, I will now stimulate a part of the [q'aorbiykuwl`aarihz q"owkuwliy],
that is called the Muscle of Reflection.
We see now that the whole eyebrow is lowered. As a result, the
wrinkles in the forehead have disappeared. This muscle is contracted
if there is [axn] ongoing process that takes up a lot of a person's
computational resources. To prevent [ihnaxrf'iyraxns] with this
process, input signals are not [ehgz'aostihvliy] analysed. The degree
of contraction indicates to what extent the input signal throughput
Then, there is another part of the [q'aorbiykuwl`aarihz q"owkuwliy],
that can be triggered separately. It is called the Muscle of Disdain.
Its contraction looks like this:
The contraction of this muscle indicates to what extent current
input is ignored as being [ihr'ehlaxvaxnt]. Of course, non-zero
values for these system-parameters may be combined, and these values
may be different for the left and the right hemispheres:
Now let us look at the [m"awth-piys] of our ['aarthahr 'ehlzahnaar].
The mouth is a general intake organ, which can swallow solid materials,
liquids, and air. In order to monitor its input materials, the mouth
has a built-in chemical analysis capability. At the same time, the
mouth is used as [axn 'awtleht] to expel processed air. Because
humans do not have [l"awd-spiykaxrz], they use this process of expelling
air for [jh'ehnaxr`eytihnx] sounds.
In emergency circumstances, the mouth can also be used as [axn
'awtleht] to expel blood, [m'uhkahs], rejected food, or other ['ahnwaontihd]
substances. When the mouth is not used for input or output, it is
normally closed off by a muscle, which is called the [l"ihps].
The lips have a large repertoire of movements. There are at least
[s"ihks] other muscles, that interact directly with the lips. I
will now demonstrate [f"aor] different movements.
First we show the muscles of joy. These muscles produce a kind
They signal, that the operating system is in good
working order, and is not encountering any problems. There is heightened
activity, in the left frontal lobes of the brain. When, on the other
hand, the activity in the left frontal lobes is unusually low, the
brain is involved in destructive processes of global belief revision.
As we saw [biyf'aor], this is signalled by another pair of muscles,
called the muscles of sadness. Here they [q] are, once more.
And finally, I will now trigger several muscle pairs
at the same time. [q'aorbiykuwl`aarihs q"owrihs], and [diypr'ehsaor
laabiyiy-iyiy q`iynfeyriy"owrihs], and the Muscle of Disdain, and
the Muscle of Sadness.
[_<500>] The parameter-setting that is displayed here, clearly
indicates serious processing difficulties of some sort.
O.K. [_<800>] Then we have the [n"owz]. [_<300>] The
nose is used for the intake of air. It is also equipped with a chemical
analysis capability. The possible motions of the nose are curiously
limited, although its pointed [pr'ehzaxns] in the centre of the
human face, would make it a very suitable instrument for expression.
I have [th"aot] about this, and I have come to the conclusion, that
it is probably the main function of the nose, to serve as a stable
orientation point for our perception, so that [saymahlt'eyniyahz]
movements of the other parts of the face, can be ['ahnaemb`ihgyuwaxzliy]
measured and interpreted.
And finally, for the sake of completeness, I want to mention the
[q"iyrz], on both sides of the face, which constitute [axn] auditory
stereo input device. Some people can [w"ihglx] these ears, but I
have not been able to determine, what the expressive function of
that movement might [b"iy].
This brings [axn] end to my quick survey\, of the most important
parts of the human face\, and their expressive possibilities. And
therefore this brings us to the second part of my talk. [kahz] this
conference is not only about [s"ay-axns]. The organizers have emphasized
that we get a different kind of knowledge\, which is equally valid\,
through the practice of ["aart]. So [dh"aet] is what I want to demonstrate
Many of the expressive possibilities I showed\, were related to
emotions\, that are well recognized in the lexicons of many human
languages. These are emotions that may be encountered fairly often
in daily life. [m'ehn-taxl] states which are close to neutral\,
where only one parameter has a non-default value. So that was description\,
imitation\, [mihm'iyzihs]. But some other things I showed were more
complex. There we saw the power of ["aart]. I showed you some new
cognitive states which you have never encountered or experienced\,
but which you recognize and understand completely\, by means of
a visceral ['ehmpaxthiy] which involves every cell of your body.
So that is what I want to explore a little further\, in the last
part of this talk.
You see what happens now. Every human person knows [ehgz'ehktliy],
in what state another human person is, when they make a face like
[dh"ihs]. Cause they know what state [dh"ey] would be in, if [dh"ey]
would make a face like this.
[_<3000>] Now it would obviously be a good idea\, if computers
could take advantage of this magnificent [hx'aardwaer] as well.
If ["ehniy] human could understand\, with just one glance\, the
internal state of ["ehniy] computer\, the world would be a better
So [dh"aet] is my message for this festival. Humans and machines
must start to work together much more [kl'owsliy]. [:ra 130] If
humans are not afraid\, to wire themselves up with computers\, the
next step in computer ['ihntaxrfeys hx'aardwaer]\, may be the [hx"yumaxn]
[:ra 140] And at the same time\, we will have a solution for all
those unfortunate people who don't feel any emotions any more\,
or who don't know how to express these emotions. If they hook themselves
up to a computer\, they can look completely normal. They can be
happy or sad\, whenever their personal computer thinks that is appropriate.
And their repertoire of expressions will even be much more extensive
than that of other people. They will be able to express feelings
that no-one has ever seen before. With the help of their personal
computer\, even completely un-expressive persons\, can now get rich
and famous in film\, theatre\, and [raokaxnr'owl].
[:ra 120] I have been very grateful for this opportunity to [priyz'ehnt]
my ideas\, to such [axn ax-t'ehn-tihv] audience. I would especially
like to thank my [ "aarthahr "ehlzahnaar]\, for his patient cooperation\,
and I want to thank you [q"aol]\, for your attention. [ _<12000>