Institute of Artificial Art, Department of ArtiFacial
The Human Interface System; Theory and Applications.
[:nh] [:ra 120] Good afternoon\, Ladies\, and Gentlemen.
My name is Huge Harry. I am a [kaam'ahrshaxliy] available [v"oys s'ihnthaxzihs]
machine. I was designed by Dennis Klatt\, at the [ehmayt'iy] Speech
Laboratory\, and [prahdy'uwst] by the Digital Equipment Corporation.
[ :ra 150] Currently\, I work here in [aamstaxrd'aam]\, as a researcher
and a spokes-machine\, at the Institute of Artificial Art\, just across
the street here on the [kehihzaxrs-gr'aaghxt].
I feel very honoured to be invited by the ['aemstaxrdaem]
School of Cultural Analysis\, to give the closing talk at this conference.
So far\, most speakers at this conference were [hx"yuwmaxn] persons\,
and many of them talked about technology. So I feel it is quite proper
that I am enabled now to [sp"iyk] here\, from my own\, [k`aampyuwt'eyshaxnaxl]
perspective\, about the [hxy"uwmaxn] senses.
My talk will consist of two parts. The first part is concerned with
the human sensory [aepaxr'aetaxz]\, and with the semiotic system that
human persons employ\, to signal information about their [paxrs'ehpchuwaxl]
and [ihnt'axrpraxtihv] processes to each other. It will also introduce
some ['ihntaxrfeys] technology\, that I have developed to be able
to investigate this issue. In the second part of my talk\, I will
then discuss and demonstrate some artistic applications of this technology.
O.K.[_<100>] I think that everyone here agrees\,
that the future of our culture depends\, on the way in which human
persons\, digital computers\, and other kinds of electronic\, mechanical\,
and bio-chemical machines\, will manage to work together. And to prepare
for that future\, we need a discussion in which every-one concerned
[paart'ihsihp`eyts] on [axn] equal footing. What we want\, is a world
that includes humans\, animals\, computers\, and other equipment\,
in a harmonious collaboration\, in which ["iych] has their proper
Or, as our colleague Donna [hx"aerahwey] has put it,
[ :nw :ra 130] A cyborg world, in which people are not afraid of their
joint kinships with animals and machines, not afraid of permanently
partial [ayd'ehntihtiyz], and [kaontrahd"ihktahriy] standpoints. The
political struggle is, to see from both perspectives at once, because
["iych] reveals both dominations and possibilities, ["ahnihm`ehjhihnaxblx]
from the other [v'aentahjh] point. [:nh :ra 150 _<1500>]
Now that is very nice theory. But I have found that
it is not easy to put this into practice\, because it is very difficult
for a computer\, to communicate its thoughts to a human person. You
may be surprised to hear this, cause you probably think, that modern
digital computers can [q'aolweyz] communicate their thoughts, in a
completely direct way. They can [q'aolweyz] display or print their
programs, their [d'eytaa-str`ahkchahrz], and their [m'aen-tahl] images.
And if you really want to get subtle and [q'ihn-tihmaht], most computers
are [q'aolweyz] willing and able to make a [k"aor] dump, which exposes
all details of their [m'aen-tahl] state.
Compared to [axn] electronic computer, a human person
almost seems a black box. [_ :ra 150] That is why you may think, that
computers never have communication problems. But they [d"uw]. Cause,
communication takes [t"uw]. It is not enough to put out the information.There
must be someone at the other end of the line, and they must be able
to [diyk"owd] their input.
So when computers want to communicate with human persons,
this often fails miserably. Cause human persons are not very good
at [diyk'owdihnx] computer output. They get very confused, when you
give them a [k"aor] dump. And when you [priyz'ehnt] them with a nice
high [rehzowl'uwshaxn] image, on your [siy-aar-t'iy] display, they
think it is ["aart], and they just [st"aer] at it. So therefore I
started to wonder, how do human persons in fact manage to communicate
with each other?
Well\, of course they have language. But language has
well-known problems and limitations\, some of which you experience
right now. So this brings me to the theme of this conference. The
[s"ehnsihz]. [kahz] there is something very interesting going on with
the human senses. This is something which is often overlooked\, but
["iyziy] to explain.
We know that human [kaogn'ihshaxn] is almost exclusively
concerned with [paxrs"ehpshaxn]. And we also know that the only thing
that human persons care about\, is other human [p"axrsaxnz]. So we
can predict\, that there is one particular kind of process\, that
human persons would want to monitor very carefully\, and that is\,
the [paxrs"ehpchuwaxl] processes\, of ["ahdhaxr] human persons. Now
you might think that this is not possible\, [kahz] perception is [axn]
internal process that [gowz'aon] inside someone's brain\, so [qaxn"axdhaxr]
person cannot [aobz'axrviht]. But that is precisely what I want to
talk about [tuwd'ey].
I have found out that people display a lot of signals
to each other all the time\, which indicate many detailed characteristics
of their perceptual and [ihnt'axrpraxtihv] processes. [_] To demonstrate
my [f'ayndihnxz]\, I have brought along a particular kind of portable
person\, which is called [axn "aarthahr "ehlzaxnaar]. I like this
kind of person a lot\, because of its [ehkstr"iymliy] machine-friendly
Let us take a closer look at such a person. What is
the closest thing they have to a [siy aar t"iy] display?
[_ :ra 120] Right. They have a face. [_ :ra 150] Now
I have observed, that humans use their faces quite effectively,
to signal the parameter settings of their operating systems. And
that they are very good at decoding the meanings of each other's
So\, how do they [d"uw] that? Well\, look at the face of our ["aarthahr
"ehlzahnaar]. What does it tell [q] us about his perceptual processes?
Not much\, you might think. But now\, [w"eyt] a moment.
You see? Arthur is [s"aed], is what people say, when
they see a face like this. So what is going ["aon] here? What I [d"ihd]
is, I sent [axn] electrical signal to two particular muscles, in the
face of our ["aarthahr "ehlzahnaar]. These muscles have sometimes
been called the Muscles of Sadness. There is one on the left, and
one on the right.
They usually operate together. If I stop the signal,
the sadness stops. When I turn it ["aon] again, it [st"aarts] again.
By sending this signal to Arthur's muscles\, I simulate what Arthur's
brain would do\, if Arthur's operating system would be running global
belief revision processes\, that are killing a lot of other active
processes\, involving a large number of [k'aonflihkt-rehzowl`uwshaxnz]\,
and priority [r`iy-axs'ehsmaxnts].
When a person's operating system is in this kind of state\, we know
that their [paxrsehpchuwaxl] processes are [ehkstr"iymliy `ahnriyl'ayaxbaxl].
As a side effect of the destructive belief revision\, new input may
be [diyl"iytihd] before it is properly processed.
Or new input may receive strange and ['axnl`aykliy] interpretations\,
because in the state of sadness\, the system's probability calculations
are not based on a [kowhx'iyraxnt] world model.
[_<1500>] The intensity of the signal that is sent to the muscles
of sadness\, is proportional to the amount of destructive global belief
revision\, that is going on.
For instance, now I have set the signal intensity to
0 again. Arthur is not sad. Now we put a relatively small signal,
about 20 Volts, on the muscles of sadness.
Arthur feels a tinge of sadness. Now a somewhat larger
signal, about 25 Volts. Arthur's sadness starts to get serious. Now
I [ihnkr'iyz] the signal once more.
You see? Now the signal is about 30 Volts, and Arthur
feels really miserable. [:ra 120] This is what we call [ehkspr"ehshahn].
[:ra 150] By means of this mechanism, the face displays clear indications
of the settings of virtually all system parameters that determine
the operation of the human mind. These parameter settings are what
humans call [iym"owshahnz]. They denote them by means of words like
[s'aednaxs], joy, boredom, tenderness, love, lust, ['ehkstaxsiy],
aggression, [ihriht'eyshahn], fear, and pain.
These parameter settings, determine the system's [ihnt'axrpraxtihv
b'ayaxsihz], its readiness for [q] action, the allocation of its computational
resources, its processing speed, [ehts'ehtaxraa]. The French [n`uwrow-fihsiy'aolaojhihst]
[duhsh'ehnn dax buwl"aonyax] has pointed out that even the most fleeting
changes in these parameter settings are encoded [ihnstahnt'eyniyahsliy]
in muscle contractions on the human face. And ["aol] humans do this
in the [s"eym] way. This is [axn] extremely interesting feature of
the human ['ihntaxrfeys hx'aardwaer], which I will explore a little
So let us get back to the first slide.
This face, which we thought was un-expressive, was in
fact quite meaningful. This is what we call a [bl"aenxk] face. A blank
face is a face in its neutral [pahz'ihshahn]. It indicates that all
parameters have their default settings. But almost all parts of a
human face can be moved to other [pahz'ihshahnz], and these displacements
indicate rather precisely, to what extent various parameter settings
[dayv'axrjh] from their defaults. So let us consider these parts in
When we look at a human face, the first thing we notice
is the thing that notices ["ahs]. The eyes. The eyes constitute a
very sophisticated stereo-camera, with a built-in motion-detector,
and a high-band-width parallel ['ihntaxrfeys], to a powerful pattern-matching
algorithm. The eye-balls can roll, to pan this camera. The eyes are
protected by eye-lids and eye-brows. The eye-brows are particularly
interesting for our discussion, because their movements seem to be
They may indicate, for instance, puzzlement, curiosity,
or [dihsaxgr"iymaxnt]. But I want to emphasize here, that the range
of parameter values that the eyebrows can express, is much more subtle
than what the words of language encode. The shape and [paoz'ihshaxn]
of a person's eyebrows encodes the values of 5 different cognitive
system parameters, ["iych] with a large range of possible values.
Let me demonstrate [thr"iy] of them.
First I put a slowly increasing signal on the muscles
called [fraant'aalihz], or Muscles of Attention.
We see that this muscle can lift the eyebrow to a considerable
extent, also producing a very pronounced [k"ahrvaxtyahr] of the eyebrow.
As a side-effect of this motion, the forehead is wrinkled with curved
furrows, that are [kaons"ehntrihk] with the curvature of the eyebrow.
The contraction of this muscle indicates a person's readiness to receive
new signals, and the availability of processing power and working
memory, for analysing these signals.
Then, I will now stimulate a part of the [q'aorbiykuwl`aarihz
q"owkuwliy], that is called the Muscle of Reflection.
We see now that the whole eyebrow is lowered. As a result,
the wrinkles in the forehead have disappeared. This muscle is contracted
if there is [axn] ongoing process that takes up a lot of a person's
computational resources. To prevent [ihnaxrf'iyraxns] with this process,
input signals are not [ehgz'aostihvliy] analysed. The degree of contraction
indicates to what extent the input signal throughput is reduced.
Then, there is another part of the [q'aorbiykuwl`aarihz
q"owkuwliy], that can be triggered separately. It is called the Muscle
of Disdain. Its contraction looks like this:
The contraction of this muscle indicates to what extent
current input is ignored as being [ihr'ehlaxvaxnt]. Of course, non-zero
values for these system-parameters may be combined, and these values
may be different for the left and the right hemispheres:
Now let us look at the [m"awth-piys] of our ['aarthahr
'ehlzahnaar]. The mouth is a general intake organ, which can swallow
solid materials, liquids, and air. In order to monitor its input materials,
the mouth has a built-in chemical analysis capability. At the same
time, the mouth is used as [axn 'awtleht] to expel processed air.
Because humans do not have [l"awd-spiykaxrz], they use this process
of expelling air for [jh'ehnaxr`eytihnx] sounds.
In emergency circumstances, the mouth can also be used
as [axn 'awtleht] to expel blood, [m'uhkahs], rejected food, or other
['ahnwaontihd] substances. When the mouth is not used for input or
output, it is normally closed off by a muscle, which is called the
The lips have a large repertoire of movements. There
are at least [s"ihks] other muscles, that interact directly with the
lips. I will now demonstrate [f"aor] different movements.
First we show the muscles of joy. These muscles produce
a kind of grin.
They signal, that the operating system is in good working
order, and is not encountering any problems. There is heightened activity,
in the left frontal lobes of the brain. When, on the other hand, the
activity in the left frontal lobes is unusually low, the brain is
involved in destructive processes of global belief revision. As we
saw [biyf'aor], this is signalled by another pair of muscles, called
the muscles of sadness. Here they [q] are, once more.
And finally, I will now trigger several muscle pairs
at the same time. [q'aorbiykuwl`aarihs q"owrihs], and [diypr'ehsaor
laabiyiy-iyiy q`iynfeyriy"owrihs], and the Muscle of Disdain, and
the Muscle of Sadness.
[_<500>] The parameter-setting that is displayed
here, clearly indicates serious processing difficulties of some sort.
O.K. [_<800>] Then we have the [n"owz]. [_<300>]
The nose is used for the intake of air. It is also equipped with a
chemical analysis capability. The possible motions of the nose are
curiously limited, although its pointed [pr'ehzaxns] in the centre
of the human face, would make it a very suitable instrument for expression.
I have [th"aot] about this, and I have come to the conclusion, that
it is probably the main function of the nose, to serve as a stable
orientation point for our perception, so that [saymahlt'eyniyahz]
movements of the other parts of the face, can be ['ahnaemb`ihgyuwaxzliy]
measured and interpreted.
And finally, for the sake of completeness, I want to
mention the [q"iyrz], on both sides of the face, which constitute
[axn] auditory stereo input device. Some people can [w"ihglx] these
ears, but I have not been able to determine, what the expressive function
of that movement might [b"iy].
This brings [axn] end to my quick survey\, of the most
important parts of the human face\, and their expressive possibilities.
And therefore this brings us to the second part of my talk. [kahz]
this conference is not only about [s"ay-axns]. The organizers have
emphasized that we get a different kind of knowledge\, which is equally
valid\, through the practice of ["aart]. So [dh"aet] is what I want
to demonstrate now.
Many of the expressive possibilities I showed\, were related to emotions\,
that are well recognized in the lexicons of many human languages.
These are emotions that may be encountered fairly often in daily life.
[m'ehn-taxl] states which are close to neutral\, where only one parameter
has a non-default value.
So that was description\, imitation\, [mihm'iyzihs]. But some other
things I showed were more complex. There we saw the power of ["aart].
I showed you some new cognitive states which you have never encountered
or experienced\, but which you recognize and understand completely\,
by means of a visceral ['ehmpaxthiy] which involves every cell of
So that is what I want to explore a little further\, in the last part
of this talk.
You see what happens now. Every human person knows [ehgz'ehktliy]\,
in what state another human person is\, when they make a face like
[dh"ihs]. [kahz] they know what state [dh"ey] would be in\, if [dh"ey]
would make a face like this.
And you also see now [qaxn] illustration of the possibilities of a
[n"uw qaart'ihstihk] genre. You see that by means of computer controlled\,
algorithmic choreography\, we can achieve [axn] ['axny`uwzhuwaxliy]
precise articulation\, of the existential anxiety and incoherent emotional
state\, of modern and post-modern humans.
But algorithmic choreography is not [ihnhx'ehraxntliy] sad. On the
contrary! Its most exciting possibility lies in its potential\, to
create [d"iyp] happiness in the hearts of human persons\, by resolving
the contradictions of their lives\, and the paradoxes of their thinking.
Seemingly ['ihn-kaxm`ehnzhaxraxbaxl] dimensions of life and reality\,
can be merged into a harmonious [`iykwihl'ihbriyaxm]. In one and the
same ['ihndihvihzihbaxl] moment\, we can experience the serene beauty
of mathematics\, and the [s'ehnzhuwaxs maxtiyriy'ehlihtiy] of the
So that is what I want to demonstrate now. I will show you a [s'iykwaxns]
of mathematically defined patterns on ['aarthahr 'ehlzaxnaars] face\,
accompanied by a [s'iykwaxns] of mathematically defined patterns on
[:nh :ra 150] First I show a [s'iykwaxns] of translations in the [yuwkl'ihdiyaxn]
And now we apply the transformations to a somewhat more complex pattern\,
and we make their succession a little [f'aestaxr]. [_]
O.K. [_<1000>] Then I would now like to demonstrate a rather
elegant symmetric pattern on the eyebrows.
And some patterns on the lips.
And some [m"aor].
And here are some interesting symmetry-transformations that you probably
have not seen [biyf'aor].
And some more\, this time a little [f'aestaxr].
And now some very fast vibrations!
[_<1500>] You see? We just saw the ultimate post-modern [tr'aens-rowm`aentihk
iym'owshaxnaxl] state\, where ["aol] the different [iym'owshaxnz]
that the human mind is capable of\, merge into one\, [q"aol-ehnk`aampaxsihnx]\,
[:nh :ra 130] So\, if humans are not afraid to wire themselves ["ahp-wihth]
computers\, the next step in computer ["aart]\, will be a new and
["axn-pr`ehsaxdehntihd] kind of collaboration between [hx"yuwmaxnz]
[:ra 120] We will have [ehlgowr"ihthmihk] choreography\, by [kaxmpy"uwtaxr-k`axntrowld]
human faces! [f"aynaxliy]\, the computers accuracy and abstract skills\,
will be married with the warmth\, the smoothness\, and [q"aol] the
other ['ehmpaxthiy] evoking properties of the [hx"yuwmaxn fl"ehsh]!
I have been very grateful for this opportunity to [priyz'ehnt] my
ideas\, to such [axn ax-t'ehn-tihv] audience. I would [ehsp'ehshaxliy]
like to thank my [ "aarthahr "ehlzahnaar]\, for his patient cooperation\,
and I want to thank you [q"aol]\, for your attention.
[ _<6000> th"aenk] you!