Theory, Practice, and Standardization of Eye-tracking Technology

Theory, Practice, and Standardization of Eye-tracking Technology

you very much, everybody. I feel a little as though I were
preaching to the choir here. You guys know an awful
lot about eye tracking. And I’m probably going
to say a bunch of things that you guys already know. But I’m going to put it together
in a way that sort of goes historically and builds
up how eye tracking got to where it is today,
because we stand on the shoulders
of a lot of people who’ve done a lot
of relevant research over the last 150 years. And understanding
what they went through makes it a lot easier
to understand where the technology sits today. LC Technologies,
it was basically founded on the hope
and the expectation that eye tracking has
a stunning future. And that assumption is based
upon two fundamental tenets. The first is that the
human eye is always looking at what’s most
important to us now. And if we’re going
to build devices out there in the world
that will help us, they need to know where we’re
looking because they need to know what’s
important to us now, and they need to be
able respond to that. So if you had a teacher in
broad areas of application or education and
entertainment, there are a whole bunch of them. I don’t want to get into a
lot of the details there. But if a teacher,
a reading teacher, knew what a kid was looking
at as that kid was going through the text and
knew where his gaze was, they’d give their teeth
to know that information. An eye tracker will give
them that information. And if we devise automated
adaptive pedagogies that can teach based upon
what the kid’s doing now, it can be a lot more
effective than it is without eye tracking. And similarly, if somebody
is doing a VR game, if the system doesn’t
know where you’re looking, how can it have a realistic
interaction with your psyche at the time? So that fundamental premise that
humans communicate eye to eye, we’ve got to give that same
characteristic to machines. And then machines can be
a lot more like humans. And that’s the
fundamental premise that LC Technologies was
founded with 30 years ago. So there ought to be an eye
tracker in every computer device. And that eye tracking
available ought to be available to
any application that’s running on that device. So looking to building eye
trackers, the concept of what an eye tracker looks
like, I’m going to review three
things in this talk. First, we’re going
to talk about several of the key anatomical and
physiological features of the eye, the beast
that we want to measure. And second, we’ll
talk about some of what the early pioneers
in the eye tracking industry have done, why they did
it, and what they learned. And finally, we’ll
review a whole bunch of eye tracking
performance metrics that we have to design to. So this picture you’re
all familiar with. It’s a standard
cross-section of the eye. You know all the parts. But there are two
particular features that are important that I want
to point out at this point. And one is the angle kappa. And the angle kappa is important
because our eyes do not look in the direction– the foveal vision of our
eyes is not perfectly aligned with the optic axis of our eye. And so when a measurement
comes in on our eye, we can see where the
optic axis is pointed, but we don’t know exactly
where the foveola is on the back of the
retina, so we have to make a correction to that. If we don’t make that
correction correctly, the accuracy of our eye
tracker is out the window. The second feature
that is important here is, if you
look very carefully at the corneal surface of
the eye, it’s not spherical. Most eye tracking software
makes the assumption that it’s fundamentally
spherical. But there are a couple of
things that aren’t, that don’t follow that model. First off, there’s
severe flattening of the corneal surfaces that
it gets out towards the edges. So that means in the pupal
center cornea reflection method, there’s a non-linearity
that happens out there that needs to be
accommodated in the software. And it’s different for
your eye than my eye. It’s different between my
right eye and left eye. And if we don’t calibrate
some of that information, it’s not going to
track me very well. And if we really want
accurate eye tracking, we’ve got to calibrate people. It’s a pain in the neck
to calibrate people. But we have to do it if we
want accurate eye tracking. AUDIENCE: Do you take questions? DIXON CLEVELAND: Sure. AUDIENCE: What
percentage of the cornea would you say would
fit accurately to an ellipisoidal model? DIXON CLEVELAND: It depends
upon how sophisticated your ellipsoidal model is,
because an ellipsoidal model can actually accommodate
some of that smoothing. If you make me an
elongated ellipse, it actually starts to smooth
out towards the edges, and that works. And another really nice thing
about the ellipsoidal model is that you can put
astigmatism into it. And so that accommodates several
of the really key features of the variability
of the human eye. So that’s a wonderful
thing to do. Terrific thing to do. There’s another
feature of the eye that is extremely important,
and it really allows eye tracking to
happen in the first place. And remember that eye tracking,
when we evolved our eyes, when nature evolved our eyes,
there are two competing mechanisms going on. One is, we want to have a
very, very wide field of view. And the second thing is, we
need to have high resolution. You can’t have both. And if you had both, if you
had 180 degree field of view and all of those resolution
that we currently have in our central
vision, our optic nerve would be about this big around. It would be about
nine inches across. We’d have a half a
cubic meter of brain function in the occipital
lobe to handle all that stuff. It’s not going to work. So nature found a really nice
way to compromise on that, so it gave us relatively low
res in the peripheral region, with very, very high res– 70 times the
intensity of cones– at the center of the eye. And across the pupil– so this diagram is kind of
neat because these green cones up there show how densely
packed the cones are. As a matter of fact, the cones
are so densely packed in there that they are just
the receptors. The cell bodies of the cones
in the central foveal part of the eye, exist outside the
central region of your retina. And all they do
is send the wires with the receptors, so
those little green things. And then as you get
further and further away from the center of the foveola,
all the little purple dots in here represent the
cones in your eye. And when you get way
outside the region, then the cones are fairly dense
and you all you’ve got’s rods. And rods and cones
do different things. This next slide is a more
scientific representation of what the rods and cones
distribution in the eye is. So here is this really sharp
spike at the central region. And the interesting thing
is, that when you go out to the peripheral vision, there
is almost as much rod density as there is cone density. And this is the
thing that people forget when they
design displays, and particularly the concept
of foveated rendering has to take into account. And that is, you can’t
make the assumption that just because
you’ve got high code density in the center
and low cone density elsewhere, that your eye
sees blurry stuff out there. Yes, it is blurry. You can’t make it out. If I hold my hand out here
and wiggle my fingers, I know because my own
proprioceptor feedback that I’m pointing up or down,
but you miss a lot of things. But the eye says, ooh,
there’s a hand out there, but it doesn’t necessarily
figure out at that point, there’s not enough information
to figure out which direction your hand is pointing. Well the fact that you
can figure out your hand, if the hand were absolutely
still you couldn’t make it out. But if it moves
any, those rods are terrific at figuring out
what’s out there and moving in the environment, because it
wants to figure out that there is this leopard out there
that’s about stalking you, and you need to find that thing. And you need to
make out what it is. So if it’s moving, you see
it, and then you swing around, your eyes swing around
and look at the thing. So this, then, brings us to the
next stage of what’s going on. If we have this
nice central vision with high resolution
and low cone resolution out in
the periphery, that means that the eye has to move. It can only look
one place at a time. If it wants high
resolution and I want to look at your
right eye, I can do that, but I can’t see
your eyes very well. I can barely make out
a face over there. So that means that nature
went to all the trouble to develop this concept
of fixations and saccades. We fixate our eyes
right on one place. Those muscles that we
have, that you see up here, the ocular muscles, hold my eye
extremely still for those 250 milliseconds takes to
get enough photons to get a good image of that foveal
area and I can make out and get the detail that I want. And then, snap,
those same muscles snap your eye over
to the next place and take a picture
someplace else. This happens roughly
four times a second. At a four hertz rate, our
eyes are darting around and they hold extremely still,
they snap extremely fast, saccad to the next place,
and they fixate again. So that behavior resulted
from the design of the eye to have to balance
this wide field of view versus a high
resolution trade off. So the output of the eyes
goes back, fundamentally, to the occipital lobe,
the visual cortex in the back of the brain
where it does all its magic and figures out what’s out there
and gives us all the feedback that we need to
interact with things. But for eye tracking, there’s
another really important element of the brain. And that’s the
superior colliculus. Hey, would you get me my pointer
out of the back of my briefcase there? The superior
colliculus is that part of the brain that in
one sense adjudicates– figures out for a whole brain– where the eye is going to
go on its next fixation. Thank you. So the SC sits
right there, right in the center of your brain. So the SC gets data from
all over your brain. And all your different
brain functions want some visual input. Everybody is starved
for visual input. Well who wins? Your eye can only look
one place at a time. It’s a winner take all
strategy, and the SC is the one that makes the
final resolution of that and tells your eye where
you’re going to go. So if you’re reading a book,
the next thing you want to do is move your gaze to
the next chunk of text. If the baby screams off
there on the right someplace, you say, oh. That part of your brain wants
to go pay attention to the baby. And then if you happen to
be walking down the street at the same time you’re
reading the book– this is a silly example, but
it demonstrates the point– and you trip over something,
your vestibular system says, you better
look at the ground because you’re going
to be breaking a fall. So it’s got these three
inputs all at the same time. And it’s got to decide
which one to do. Keep in the back of
your mind, while it’s making this
decision, it’s making the decision what’s the
most important thing for you to look at right
now, because it’s that fundamental
process of the brain, and the fact that we have
a central vision, that enables eye tracking. That’s the fundamental
reason that we can do it, is because that information. Nature poked a hole
in our skin to let the photons in to our brain that
allows all of this to happen. And we’ve got to take
advantage of that to build machines that have
the same kind of eye contact that we humans do. So the SC is a really kind
of a neat democratic thing. What it’s able to do
is get these inputs from all over the brain. And if you unfold the
thing, it’s like a map. And you can see areas poke up. If your eye is pointed
right over there and your peripheral
vision see something, it says, oh I want to saccad
that many degrees to go over and look at you
instead, if you happen to be waving your arms at me. You’re not, but that’s
what it would do. And something else, the
vestibular system says, something’s
happening over there. I want to saccad
to go over there. It’s got this map and it’s
building up these potentials of all these important areas. And it’s being very
clever about it, because each part of the brain
says, I want visual attention, but I want it with a level six. Your vestibular system comes
back and says, I want it, but I’m a nine. So that spike grows
faster on the surface of the superior colliculus’s
map of your retina. And finally one of these spikes
pokes through a threshold and bam, the superior
colliculus fires off, and that’s where the eye goes. And that’s that. That’s where the
next fixation is. So the key point here
is that an eye tracker can’t see what all the
alternative options were, when you made the decision to go. If you can figure
out that problem, that would be pretty cool. But one thing we can see
is the final decision that it did make,
and that decision is the most important
thing to your brain at that point in time. So the role of the
SC in eye tracking is a really big,
fundamental element of what allows us to achieve
these goals of getting systems that make eye contact with you. So what happened? How did these guys build this
equipment in the first place? What’s going on? Well– AUDIENCE: Excuse me. DIXON CLEVELAND: Sure. AUDIENCE: Could you
go back a few slides? DIXON CLEVELAND: Sure. AUDIENCE: What are
the two colors? Purple and– DIXON CLEVELAND: Oh, these
are the visual pathways. The purple is from the
left eye, and [INAUDIBLE]. So it goes through the optic
nerve, the optic chiasm, the optic tract, back here
to the pulvinar region. Some signals go back up into
your superior colliculus. Others head back into
your visual cortex. And then out from
the visual cortex, they go up to the frontal lobe. And from your
frontal lobe, that’s where a lot of the
decisions are made that feed back down to the visual cortex. AUDIENCE: And you said
that one [INAUDIBLE] the other side of the brain? DIXON CLEVELAND:
They all cross over. For some reason, your right
eye is connecting the left half of your brain, and vice versa. And one thing I don’t
quite understand, and it’s a fabulous
question, is what happens in the central region? In the foveola of your
brain, where is it connected? Is it divided down the middle? I don’t know. Doug Munoz at Queens
University did a lot of the research on that,
and found out the connectivity of all this sort of stuff. He did it with monkeys. They were nasty experiments. But he found out an
awful lot of the stuff. And that’s where it was
verified that these crossover things happened. It was speculated before
that, because somebody’d have a right brain problem,
or damaged right brain, and their right
eye wouldn’t work. And everybody thought,
well, that’s interesting. But it wasn’t until Munoz
did his work that we actually proved that that was the case. So, early people in the
researching in eye tracking, wanted to find out, how
is the brain doing this? What’s it doing? And so this guy, Delabarre,
back in 1989, built a device– this is not Delabarre’s
device by the way. But he built a system. He took a needle and
he stuck it in the eye. He took a little
piece of plaster and glued it to the eye. And this wire sticking out,
this needle sticking out with a string coming off of it. And the string dragged
along something on a strip chart recorder. And he recorded all
these eye motions. And you can see from the strip
chart recorder, there it was. Fixation, saccad, fixation,
saccad, fixation, saccad. So he demonstrated all that
with that crazy equipment 170, 80 years ago. So eye tracking’s not new. It’s been around
for a long time. Long time. And then this guy, Schachel– and there several other people
in this history– but Schachel then came along,
and he figured out that there is this
potential across the eyes. If you put a pair of electrodes
on right and left side of your eye, there’s a
potential across there. And as your eye rotates
from side to side, it presents a
variable resistivity that can be measured,
and a measured voltage across the eye. Well, that was a
hell of a lot easier than sticking a
needle in your eye, patching a needle to the
corneal surface of your eye. So EOG came around, and that
was really cool technology at the time. And then Yarbus. I’m sure all of you
guys know Yarbus. But the keen thing
that Yarbus did, he actually built some neat
eye tracking equipment. And pass this around. This is his device. So this device, somebody put
their head in the headrest here, and these two cameras
looked at their eyes. And they recorded
the data with video. So video eye tracking
finally came on the scene. And he was doing his work
in the ’50s and the ’60s. 1950 and ’60s. So even that work,
now, is 70 years old. And he was able to
collect the device. Those video cameras recorded
all the data, and then by hand, frame by frame,
they’d go through and calculate the corneal
reflection and the pupil center, calculate that data. They’d infer a calibration
for that person’s eyes and map out the gaze trace. It was beautiful research. And what he found was– well his chief purpose was
not to build an instrument. He did that, and he built
a wonderful instrument. But the key thing
that he found out was what people do
in visual search. And they do exactly what
you’d expect them to do, but he documented. What you’d expect them to
do is, when somebody walks into a room, you’d
expect them to take a general look all over,
pick up the big picture. Something will
attract their interest and then they’ll start zeroing
in on some particular detail. And he documented that
that’s what happened. So that was really
another fundamental piece that was putting together
this whole picture of how the eye behaves. So then there were these three
guys, Merchant and Mason– well, two guys, really. It was Merchant and Morrissette. These guys were
working for Honeywell. And Honeywell had a contract. They had this idea that they
took out to Wright-Patterson Air Force Base, that they wanted
to actually control something with their eyes. Not just see what they did,
but use their eyes as a control device. So what they did was come
up with this idea of– well, the basic application was,
they wanted to do targeting. So, a pilot was flying a plane. He’d look over at something. He’d say that’s what
he wanted to do. And he would indicate to a
target acquisition device by pressing the button once. If the target acquisition
device actually picked it up, he’d push the button
again, and the missile would go off and
kill the target. Really cool idea. But to do this, now
they’ve got to get eye tracking working in real time. But they didn’t. So they got a camera that could
capture the image in real time and digitize it. They took all the algorithms
for computing the pupil center and then projecting the gaze. And they put that stuff
into computer programs. And they got gaze point
calculation, programmed gaze point algorithms. And when this thing worked,
it generated eye tracking data automatically and in real time. In my opinion, these guys were
the Wright brothers equivalent of eye tracking to flight. Orville and Wilbur
got lift power and control all
solved simultaneously to get this plane of the ground,
fly 120 feet stably and land. And these guys,
Merchant and Morrissette built the first eye
tracker that was automatic, real time, and pretty
bloody good accuracy. They used the pupil center
corneal reflection method, which is the standard today. So back in 1969, these guys
published their results 50 years ago, and we’re still
using that concept today. So these guys laid down
a tremendous foundation for how eye tracking would work. They used the bright pupil
center method, as well. And LC Technologies
does that today. A lot of other companies
have abandoned that, because to get co-axial
lighting is very, very difficult to implement that in hardware
in a small piece of hardware. But it gives you the very
good image processing results, because it gets very good
contrast between the pupil and iris. And so you can get an accurate
pupil center measurement much better with a
bright pupil effect than you can with a
dark pupil effect. So what happened with
Merchant and Morrissette work? Well, there were three big
problems with this device. First off, they implemented
the image processing algorithms on a
PDP11 computer that sat in the corner of the room. They were never going to
get that in a fighter plane. Number two, the
room had to be dark. That wasn’t the
environment that a pilot would be with a canopy over his
head and the sun shining in. No way that was going to work. Let’s see, there
was a third problem. Oh, yeah, and it didn’t
allow any head motion. You had to hold your
head pretty still. You’re looking at
a Yarbus device, it was the same type of deal. To get a high enough
resolution image of the eye, they had to use a really high
telephoto lens that only had a very narrow field of view. You couldn’t move your eyes very
much, or your head very much, and have the eye tracker
continue to see you. So that project
just basically died and eye tracking sort of went
underground for about 20 years. But one thing really did
come out of those guys’ work. And that is a list of eye
tracker performance metrics. And this is a hodgepodge list. I mean, the list goes
on and on and on. It gets boring. And I’m sure you guys are
fairly familiar with it. But eye detection accuracy,
the image processing not only has to be able to measure the
corneal reflection of the pupil center, but if it doesn’t
recognize the thing that’s an eye in the first place– if it detects the
corner of my glasses– it can measure what it
thinks is the pupil center and corneal reflection
perfectly and you’re going to the garbage data. So you have to have
a good detection algorithm that differentiates
between eye and non eye objects. Then you have the gaze
prediction accuracy. You have to be able to measure
those things precisely. And then you have
to have some kind of freedom of head movement. And then you have
to have the ability to accommodate a whole bunch
of different eye variability. Then glasses and
contacts, tolerance to ambient infrared light,
ease of calibration, safety. There are certain maximum
permissible exposures that you can put on the eye. And it’s got to
work fast enough. It’s got to work really fast
in certain applications. And then size, weight,
power, and cost. All of those things,
the massive list of things that you have to do. So there are a lot of
problems in the industry, and that’s just a
summary of them. So I want to just give
a few examples of images that have some of these problems
that are difficult to process. Reflections cause a lot
of clutter in the images. This is an example
of a person sitting in a room with a
remote eye tracker, and they happen to
have glasses on. But you can see this
window in the background, reflecting off the
glasses, making a mess of that eye image. Most image processing
algorithms look at that thing, they throw up their
hands and they say, I don’t see an eye in there. You and I can see it, so
we know the information’s embedded in there. But we haven’t yet developed
enough really smart algorithms to dig that information out
of that cluttered image. And then over here is an
example of a reflection of the LED, the eye
tracker’s own illuminator. Well, you can’t see
it here, but there’s a big reflection off
the glasses itself, and there are some internal
reflections up here. And you can’t tell
which of those is the real corneal reflection. There are a bunch of glints
in there, but how does the image processing
know how to dig out the right corneal reflection? Or the right glint that
has the corneal reflection? Here’s a case where the
reflection off the glasses are getting right close
to the image of the eye, covering up the corneal
reflection completely. That’s not going to
give you any data. Over here on the left,
we have contact lenses. Contact lenses have
these little things stamped in the middle of them
that you and I can’t see, but since eye trackers work
in the infrared region, they reflect off this
thing and they come back with this little symbol. You can figure out who made
that damn contact lens. That’s of zero interest
to eye trackers, but they can get confused
by that data in an image. And we’ve got to design
algorithms that just say, uh-huh. I got it. Ignore that data and proceed on
with calculating the eye image. Hard line bifocals. The camera’s looking at your
eye from coming up, looking from below, and it typically
sees your eye right through that cut in the glasses. So the top half of the
pupil seems to be one thing. You see it through the top
lens, and the bottom half through the bottom lens. Where’s the center
of that pupil? We’ve got to design algorithms
that handle that stuff. The contrast on
this is fairly poor, but the real corneal inflection
is sitting right there. And that corneal reflection
straddles the perimeter between the pupil and the iris. So if we include this
thing in the pupil, are we going to compute
its pupil center right? It’s sitting right
on the gradient. The corneal reflection’s
sitting right on the gradient between
the pupil and the iris. It’s coming up, where is the
center of the real center of the corneal reflection? Image processing has got
to handle that problem. Another big problem
in eye tracking is measuring the
range to the eye. It’s extremely
important that we know the distance between the
lens and the corneal surface of the eye. Not just some place out in
the center of the eyeball, but right to the corneal surface
of the eye, the glint pupil vector. So here’s a standard
glint pupil vector when somebody is sitting in
a nominal location looking at a point on the screen. And then as the eye
were to move backwards, there’s a double barreled effect
of making an error in range. The first is, if the eye were
to move back and you didn’t know it– you being the eye tracking
instrument didn’t know it– the whole glint pupil vector
would get smaller first because the eye is
just further away. And the whole image got smaller. The second thing is that as
your eyes move backwards, if it’s still looking
at the same gaze point, it literally started
to point downward. So its orientation
was different. But if we didn’t know those
two things were happening, we would project a very
different gaze point. We’d project somebody looking
way down low on the screen. So we have to measure
gaze accurately. That’s one of the things that LC
Technology spent a lot of time, was being able to measure that. We use what we call
the triangular, or the asymmetric
aperture method. And it’s based upon the fact
that the corneal reflection looks like a point source
of light out in space. And it’s located right behind
the corneal surface itself. One half the radius behind
it, but that’s where it is. If the camera lens is
in the right place, all these rays coming
from my corneal reflection back to the lens and to the
sensor surface on the camera, will converge in a
perfect location. But if, for some
reason, the eye moved forward or back, or
conversely in a sense we’re playing the camera,
the lens lengths were too long or too short, the
gaze, the lines from that point wouldn’t converge
on either side. But there’s a really
cool property, and that is, if we make a
triangular aperture whose shape is different upside
down than right side up, we can look at that
corneal reflection. And it’ll be blurred
on either side, but we can tell
whether that thing is right side up or upside down,
because of the triangular aperture. So if we can design a control
loop that drives this thing back into focus, we’re good. We’ve got it. We can measure that. And we can use the
Gaussian lens equation to calculate exactly how
far the camera is focused. And so we can make very, very
precise measurements down to a tenth of an inch. And that tenth of
an inch is extremely important for getting accurate
gaze point calculations. Going the wrong
direction here, sorry. Large and small pupils. This is a problem with
the pupil center method, or with the bright pupil method,
because big pupils typically let a lot more light into the
pupil, and a lot more light back out. And so they end up
being very bright. And tiny little
smile pupils tend to just get very, very dim. And we have to design image
processing algorithms that can handle that wide dynamic
range of pupil amplitudes. AUDIENCE: Are these examples
using the triangular aperture [INAUDIBLE]? DIXON CLEVELAND: Yes. And you can see it back there. In the demo after the fact,
you’ll see all that stuff. We’ve got it demonstrated,
so you can actually see the mechanism
of how that works. This is a particularly
cool example because there’s
a tear down here. Light from the LED is going in,
reflecting off the tear glint at the bottom. There’s a tear glint at the top. There’s a tear glint on
there at the inner canthus. And there’s a real
corneal reflection. And there’s another
reflection that comes from an internal
thing in the lens. How do you find the
right corneal reflection? Which one’s the real one? If your image processing
doesn’t handle that, you’re going to
produce bogus data. Eyelashes come down. This is not a good
contrast here, you can see a
beautifully on my machine but you can’t see it
up here very well. But there are
eyelashes coming in. How do you map the
perimeter of that pupil? Does that clutter screw you up? If it does, your eye tracker
isn’t going to work very well. We got dry and
congealed corneas. I don’t see
technology that works with people with disabilities. And so we see all
the corner cases. A lot of people with
ALS, for example, have lost their blink reflex. Not only can’t
they speak and they can’t use their hands, their
eyes, the ocular muscles still work because connected
directly to the brain rather than going
down through the brain stem where there’s loss of
connection between the brain and the parts of the body that
the brain wants to control. So your ocular muscles work,
but your blink muscles don’t. Your eyelid muscles don’t. They are connected back
down through the brain stem. So the data you get there,
if somebody doesn’t blink, you get congealed tears all
over the front of your cornea. It’d be nice to handle that. In this case, you have dry eyes. So you don’t blink enough. So the corneal surface doesn’t
get lubricated properly. There’s no corneal
reflection there. There’s a nice tear glint, but
there’s no corneal reflection. What does your image
processing do with that? If you’re eye’s in motion
you get a very blurred image of the eye. So you have to take
high speed photography to get images that are
still enough that you can interpret correctly. One of the serious problems
in eye tracking is squinting. If you squint, you lose
the corneal reflection. If you don’t have both a corneal
reflection and a pupil center, the PCCR method doesn’t give
you the answer you need. So you need to start putting
other illuminators around at different places that
when you’re squinting, it’ll continue to work. Why is squinting a problem? Mostly because when
people are playing a game and they start to get
intense about something, the first reaction
you have is to squint. You just want to say, what in
the world is going on there? And you squint to look at it. And if you’re depending
upon the eye tracker to track you while you’re
doing that, and you squint, you’re in trouble. So we need to tackle
that problem too. LC Technologies has taken
a good step toward that. We did it for people
with disabilities because they squint
for a different reason. They’ve got ptosis
of the eyelid, or what we call droopy eyelids. Their eyelid control,
as we said before, isn’t working perfectly, so
the eyelids start to come down and they impinge particularly
on a large pupils like this one. So our image processing
algorithm looks at that pupil and it says, oh. The eyelid actually is occluding
the top edge of that thing. And so when it
fits the pupil, it doesn’t just find the center
of mass of that thing. It finds legitimate,
apparent pupil perimeter and fits only to that data. There’s a little
secret in eye tracking, or a concept in
eye tracking that’s not very well understood. And that is, the
pupil center does not open and close precisely
about a given point. You’ve got a sphincter
muscle that’s trying to constrict your
pupil, and radial muscles that open it up. And when your pupil
opens and closes, it doesn’t go around some
nice constant point there. Smaller pupil may drift over
here, a large pupil drift here. And if it starts
to drift around, the pupil center corneal
reflection method doesn’t know that the pupil center shifted. And so it’s using the
old calibration assuming the calibration of the
pupil center of the diameter when the calibration was taken. And then it starts to drift. A lot of eye trackers
had that problem. So we’re trying to
solve that problem, too. Calibration. You have to have the calibration
if you want accurate eye tracking, is really all
I’m going to say there. LC Technologies has
built this thing that we call an Eyefollower. And there’s a fundamental
trade off in eye tracking between accuracy and
tolerance to head motion. If you want a nice
tolerance to head motion, the typical solution
in the industry is to widen out the
camera field of view. Good, we can see you
wherever you are. We get your eyes. We can track them. But how good and
how well can you measure the glint pupil vector? As the resolution
goes down, you can’t. And there’s this
crazy trade off of, our eye trackers now have a
very narrow field of view. But you need a
wide field of view. So camera technology will get
you a factor two, maybe four in the field of view. But we want 10, 15
times the field of view. That’s not available
in camera technology. We can’t get that
high resolution. So we basically borrowed a
model from our own human eye, and that is we have a robotic
device where the eye tracking camera is actually on a
gimbal that moves around as your head moves around. And we have another
camera, a wide field camera that says, oh,
that’s where the head is. So if the eye tracking camera
is recording off here someplace, and your head’s over here,
the wide field of view camera, just like our peripheral
vision, will say, ooh, there’s the head. Go look there. Boom, our system will saccad
over and fixate on your eyes and start tracking your eyes. So we get, always, a
very high resolution image of the eye, no
matter where your head is. And at the same time,
we can track your head over a large range. So we’ve got our own little
robotic system to do that. And you’ll see a demo
of that in the back. You don’t need that
for all applications. If you’re doing VR glasses,
that’s not a problem. But for remote eye tracking,
it’s a very good solution. So where does the
technology stand today? Well, there are a lot
of pretty simple quote, good enough solutions. The systems that
are out there now, they track a fair
number of people. They track a fair
amount of the time. But they don’t get everybody
all the time with high accuracy. So the industry’s
got a long way to go to make that thing
work to be really robust across the
entire population. We need to make them
yet smaller and cheaper. Today there’s these
little bars coming out. Cool. You made it smaller. But you gave up
all that accuracy. You gave up the
robustness in tracking. That’s not a good
enough trade off today. We’ve got to do
better than that. So we’ve got to have
all kinds of fanciness. I was going through
all those images, showing about how you process
different kinds of images, problem images, to be
able to cover a broader range of the population. And accuracy really
remains a key problem. So the next steps. There are, obviously,
two different categories of product design. There are remote head trackers
and head mounted eye trackers. So we have to
design both of those for different applications. Hardware miniaturization
is really a key. It’s a tremendously
important thing. We’ve got to build
much smaller cameras. We’ve got to build
cameras that are adapted to the specific purpose
of eye tracking, not taking off the shelf cameras
and just throwing them in there and making it work. Same for lenses. There is more glass and stuff
in those off the shelf lenses than we ever need
in eye tracking. We need to throw
most of that out. But there are certain
characteristics that don’t exist in the
lenses that we want, that should be built in
to eye tracking lenses. That optimization has really
never been addressed adequately in the rest of the industry. But the key thing, as I
was mentioning before, we still need to do
this image processing. We’ve come a long
way in that area, but there’s a lot more work. That is the tougher problem
of all of these things. The hardware is really
pretty straightforward. You get a bunch of good
engineers in a room and they’ll make it smaller. We can solve those problems. But the thing that
handles the diversity of the human factor in our
eyes lies in the software. And so we have to do that. And I believe that
LC Technologies has delved into
that problem better than anybody else
in the industry. Well, thank you
very much you guys. It’s been a fun talk,
and good questions.

You May Also Like

About the Author: Oren Garnes


  1. Great talk. Important to Education, Entertainment indeed, but possibly vitally important to advertising, politics and controlling people as well. I have no wish to critisise the positive direction of this technology but do think that we should inform ourselves about some of the possible negative uses to which it might be put.

  2. Eye-tracking is a very important topic. I expect it to become ubiquitous, just as nearly all UI software today expects the user to have a pointing device.

  3. It's extremely sad that a video about technology as important as this, able to offer a communication option for ALS, MS etc. sufferers when they are at their most vulnerable, receives a mere 3,000 views and 4 comments in just over a month, while someone posting a video about their cat playing with the parrot goes viral overnight. Something is wrong with our society! Are we so self-absorbed that we ignore the most needy, focusing instead on nonsense.

  4. It's too bad your settings block pinning to Pinterest. That limits the number of people who would know about this.

  5. another intrusive horrific method to know me whether I want it or not. For those with locked in or other paralyzing diseases such as Stephen Hawking, GO -yes. But now with facial recognition tracking, what if you have to defend yourself but your attacker knows you are looking at his balls?

Leave a Reply

Your email address will not be published. Required fields are marked *