Wearable Computer for the Visually Impaired

2019 COE Engineering Design Project (NMK02)

---

Faculty Lab Coordinator

Naimul Mefraz Khan

Topic Category

Software Systems

Preamble

Advances in computer vision and machine learning has made detecting objects, faces, texts from images easier. With the power of services like Google cloud vision, it is possible to provide a "virtual eye" to a visually impaired person, where through continuous analysis of video stream from a camera, a wearable computer can provide spoken feedback on anything that is in front of it. By providing some rudimentary user control over the analysis process, this type of wearable computer can become an indispensable companion for the visually impaired.

Objective

The objective of this project is to build a wearable computer that can provide automated cloud-based and user-specified analysis of images through a video stream obtained from a camera, and provide spoken feedback on the analysis results through a speaker. The wearable camera can be controlled through a voice-controlled interface, where audio captured through a microphone can be analyzed through cloud-based speech processing (e.h. Google Assistant/Amazon Alexa).

Partial Specifications

1. The computer should have a camera, a speaker, and a microphone.
2. Images obtained through the camera should be streamed to a server for cloud-based analysis.
3. The user can specify what to analyze from the image through the microphone. Example: recognize face, recognize object, extract text. The users' instruction should be interpreted through integration with Amazon Alexa/Google Assistant.
4. The analysis results are provided as a spoken feedback through a speaker.
5. The computer, camera, speaker, and microphone should be packaged into a wearable rig.

Suggested Approach

1. Raspberry Pi/NVIDIA Jetson Nano with camera, speaker and Microphone.
2. Connect to some vision API (e.g. Google cloud vision) and speech API (e.g. Google assistant, Amazon Alexa).
3. Stream images to cloud.
4. Play analysis results back through speaker.
5. Analyzed audio instruction should be able to control the nature of analysis (e.g. recognize face, recognize object, extract text).

Group Responsibilities

1. Survey hardware and software (cloud API) to figure out the best approach.
2. System design.
3. Implementation.
4. Testing under varying conditions (low lighting, fast motion, difficult angles, noisy environment etc.)

Student A Responsibilities

Hardware + wearable rig

Student B Responsibilities

Programming for image and audio streaming

Student C Responsibilities

Programming for cloud-based vision API (e.g. Google cloud vision)

Student D Responsibilities

Programming for cloud-based speech API (e.g. Google Assistant, Amazon Alexa)

Course Co-requisites