Sign Language Recognition

Question

I am a final year undergraduate student of Information Technology. My team and I have taken up "Sign Language Recognition" as our Final Year Project. We have just started with it and we are in the phase of information gathering (gathering data). We plan to use Instrumented Gloves as the input device. But we do not have much knowledge in the area.

Also, we came across the following methods for training the system for actual recognition of gestures and hence sign language.

1. Neural Networks
2. Symbolic Learning Algorithms
3. Hidden Markov Models
4. Instance Based Learning
5. Grammar based techniques

Please tell me which of them I should use for sign recognition.

Also, tell me about Instrumented Gloves and is there any specific variety which we should choose for our project?

Answer 1

Thad Starner (now at GA Tech) was able to get this to work with HMMs and a cap-mounted video camera back in 1995. I imagine you'll have some good luck along those lines with the massive increase in processing power and sensors since then.

Thad Starner and Alex Pentland. Real-time american sign language recognition from video using hidden markov models. In Proceedings of International Symposium on Computer Vision, Coral Gables, FL, USA, 1995. IEEE Computer Society Press.

You're going to have to do a lot of data cleaning/filtering before it gets to the HMM, however. This is a signal-processing task and it may be suitable to use fuzzy logic to categorize the inputs (look at how mouse-gesture recognition is done) and something like a NN to downsample the data.

Answer 2

Borrowing heavily from the comments: I think you should look into existing image recognition algorithms. Personally, I would simply implement an existing open source one. Once you have some understanding of and comfort with one, train it on a single sign.

Use the output from your gloves to generate an image. To that end I would look for a glove that provides output easily translatable to an image, such as a table of bit representing the location of the digits. Then pass the images into the image recognition software.

Once those two are working together, your gloves and image recognition software, you should move on to whatever User Interface you want, such as speech or text output.

Finally keep adding signs, testing after each addition, till your semester is up, or the system performance degrades too much.

Answer 3

There is another approach available, recently I've worked on a project that used gabor wavelets in face recognition Wikipedia entry for Gabor wavelets. And I stumbled on a interesting project that fits your need.

The project revolves around a paper, rock & scissors game and the recognition of symbols used inside a game. It tracks the player trough the camera and recognized the hand gesture.

The project can be found on the following link (Ropesi project). There are some videos inside the svn trunk that demonstrate the use and application of the project.

Answer 4

I know some ASL (?) has been captured and recognised by computers in the past, and that was a while ago. I remember watching videos of this on VHS. 1994 at the latest.

Sign Languages (there's more than one) are typically multi-channel media — it's not just the handshapes, so the gloves may not be quite enough (depending on your aims). My ASL is almost non-existent, but British Sign Language uses something like this (pinch of salt required, I'm very rusty):

• ‘Phonology’:
  • 26 hand-shapes (configurations of the fingers).
  • Various hand orientations
  • Various hand starting positions
  • Various types of hand movements
  • Shoulder shapes.
  • Head position and tilt. (in BSL, a slight ‘nod’ indicates first person)
  • Eyebrows and eyes.
  • Mouth shapes.
  • Overall expressions which are context-sensitive. Something bad may have happened, but your expression is expected to change briefly to the expression required by the sign.
• Grammar:
  • Signing space (where, in relation to the body, signs are allowed)
  • Very different from English grammar. E.g. BSL generally follows an Object-Subject-Verb word order, where sentences like CAT DOG CHASED (‘the dog chased the cat’) are the norm.
  • Spatial pronouns: hand-spell a name, point to a location in your signing space, and you can then point back there rather than hand-spelling or signing the name.
  • Verb aspect: I don't know about ASL, but BSL has some seriously complex aspect. You can stress a sign by lengthening the ‘hold time’ (starting state of the sign) and making the movement abrupt. More aspect I can think of right now indicates whether something takes place slowly, quickly (and how slowly or quickly), direction of travel, the signer's attitude towards what's being signed, etc. ‘They stared at me for a long time and I was put off’ is essentially the single verb ‘SEE’ with lots of aspect.
• Syntax: again, pretty different to what you be used to. BSL is somewhere between Japanese and Spanish, but that's probably an oversimplification.
• Complications: fingerspelling follows slightly rules in BSL (not so much in ASL). There's a staggering amount of community-local slang (sign languages aren't written, so they're a lot like spoken languages before the invention of writing — in some cases, boys finishing boys' schools and girls finishing girls' schools graduated to find they had very divergent languages), and quite a few loanwords. My favourite one is (probably only Scottish) slang for ‘I love you’, which is an amalgam of the ASL letters I, L, Y (think standard Spiderman gesture) turned into a transitive verb (the sign moves from close to the signer's torso to the intended person): a completely invalid handshape in BSL, but turned into a BSL verb.

I think you'll be seriously hard-pressed to capture and parse successfully all of these channels. Perhaps reading a suitably limited subset may be a good idea: basic handshapes (ASL fingerspelling is very clear, and should be very easy today) and some basic motions. You may have to limit the complexity of the sentences you parse since you probably won't be able to capture all aspect reliably.

The software will feel to native signers a bit like a hearing learner of sign language, missing out on a lot. I suspect signers will adapt very quickly. After all, native English speakers can still understand a sentence even if some of the articles are missing.