My team SacHacks 2018 built this hack, which made a first step toward allowing the deaf, hard-of-hearing, and noverbal communicators to take advantage of smart home technology like Amazon Alexa products.
To-date, over 100 million Alexa units have been sold globally, leaving the millions in the world's deaf and nonverbal communities in the dust as products that rely on verbal commands.
I wanted to address this gap this exciting innovation in home technology created at SacHacks 2018 by utilizing open-source gesture recognition software built by EvilPort, Amazon Alexa Skills, and a computer webcam to allow an Amazon Alexa to respond to ASL gestures processed from the webcam by the recognition software. The hack would read off the gestures to the Amazon Echo from the computer's speakers, and Alexa responded accordingly. In 24 hours, my team was able to register a few commands using ASL letters, like turning on lights or closing a door. To run the training files on new gestures we wanted to add, we took advantage of Google Cloud Computing Engine.
Though much of the functionality of smart home products - playing music, reading off facts, etc. - could never be made entirely accessible to the deaf community, we showed that these smart home manufacturers could still allow more people to take advatage of the technology's ability to perform other practical tasks, like turning lights on/off, locking doors, opening a garage, turning off a faucet, etc.
To see the code, visit the Github repo!
Created by Yassin Oulad Daoud for SackHacks 2018.
The SacHacks organizers, me, and one of my teammates from UC Davis, Jeremey Liu. Our teammate, James Jung (UC Berkeley), is not pictured.