Skip to content

sisaha9/smartboard

Repository files navigation

Smartboard

Code for using a Smart Letterboard for communication

Smartboard Demo

Table of contents

What does it do

The Smart Letterboard enables communicating with others without the use of voice or hand movement. A laser pointer is attached to one's ear or glasses. The laser pointer than points at the letter on the letter board. A camera sends the image to a computer (in our case it was the Jetson Xavier NX SBC). The computer processes the image to figure out which letter is being pointed at. This uses the laser tracker code developed here. Credit to @bradmontgomery for the initial implementation. Once we figure out the letter, we use the Happy Transformers library to predict the top 3 words (either completion if we are in the process of communicating a word or prediction of new words if we have completed). This is then voiced using the gTTS (Google Text To Speech) library. The letter and predictions are also sent to the Arduino so they can be communicated to the users via LED strips on each letter and an LCD screen that displays the sentence being built and predictions

Installation Instructions

We need to install the following packages

sudo apt install libopenblas-dev
sudo apt install ffmpeg
sudo apt install mpg321

Then we need to install Torch. If this is on Desktop use https://pytorch.org/ for installation instructions. If on a Jetson use https://docs.nvidia.com/deeplearning/frameworks/install-pytorch-jetson-platform/index.html

Finally we need to install the following pip libraries

pip3 install git+https://github.com/sisaha9/transformers.git@add_prefix
pip3 install git+https://github.com/sisaha9/happy-transformer.git@add_prefix
pip3 install pyserial
pip3 install gTTS
pip3 install opencv-python

Note the source installations of the transformers and happy-transformer library. These are needed as the forks have some additions that support word completion capabilities

Usage Instructions

  1. Clone this repository
git clone --recurse-submodules https://github.com/sisaha9/smartboard.git

Once cloned, switch directories to inside this repository

  1. Follow the installation instructions

  2. Flash the Arduino code in smartboard_arduino

  3. Run python3 laser_tracker/laser_tracker/laser_tracker.py

  4. Position your USB camera so that the letter board is centered in view

Smartboard Position

  1. Stop running the laser tracker. Now run ./run.sh (Note that you still need to be connected to internet for the gTTS library to operate)

  2. It takes around 10-30 seconds to initialize (depending on hardware) but once it does the letter tracking works and you will be able to use the Smartboard as intended

References

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published