← Back to blog

What I learned building GestureKey

Training data Real signers
Datasets cluster tightly around one kind of signer. Real signers don't.

I built the first version of GestureKey because of my niece. I went into it thinking the hard part would be the model. I was wrong. The hard part was the data.

Sign language and gesture datasets are small, inconsistent, and overwhelmingly non-diverse. They tend to capture a narrow range of signers: similar lighting, similar backgrounds, similar hand shapes. A model trained on that data learns one particular version of a gesture, and then fails the moment someone signs slightly differently: a different hand size, a different skin tone, a different angle, a different room.

That's not a minor technical inconvenience. For the people GestureKey is meant to serve, a model that works 80% of the time isn't 80% useful. It's another system that almost works for them, which in some ways is more frustrating than one that plainly doesn't.

80% accuracy For the same person, that missing 20% never goes away
Not random misses spread across everyone. The same 20%, for the same person, every time.

So the question shifted. It stopped being "can I build a gesture classifier" and became "how do I build one that actually generalises, across people, contexts, and signing styles, when the data available to train it was never representative in the first place." That question is part of why I'm looking at pursuing a PhD. But I didn't want to just sit and wait for that before building something. So I started Japa Technologies, and GestureKey is what's come out of it so far.

The more I built, the bigger the picture got, too. When I started, I was thinking about messaging: the simple, human act of sending a friend a note. But people with motor disabilities are also underrepresented in formal employment, not because they lack capability, but because workplaces are built around input methods they can't always use. Keyboards. Touchscreens. Voice interfaces in an open office. The tools of a modern workplace assume a physical standard that can exclude someone before anyone's even looked at their CV.

If a gesture-based interface can drive text input, it can drive anything text input drives: email, documents, forms, the software people actually use to do their jobs. That's the version of GestureKey I'm building toward. Not just a way to send a message to a friend, a genuine way for people with disabilities to participate fully in professional environments, on the same terms as everyone else.

Text input Email Docs Forms Software Every app that takes a keystroke
Not a messaging feature. A way into everything a keyboard already opens.