A Wearable navigation assistant for the visually impaired

2021 COE Engineering Design Project (NMK02)

---

Faculty Lab Coordinator

Naimul Mefraz Khan

Topic Category

Software Systems

Preamble

Advances in computer vision, machine learning, and depth interception has made it possible to provide a "virtual eye" to a visually impaired person. Through continuous analysis of an image captured through an RGB and depth camera, the "virtual eye" can provide spoken feedback on anything that is in front of it.

Objective

The objective of this project is to build a wearable computer (ideally head-mounted like a VR headset) that can provide automated analysis of objects and obstacles in front of a person while they navigate their surroundings.

Partial Specifications

1. The wearable computer should have a depth camera, an RGB camera, and a speaker (to play back audio feedback).
2. Images captured through the depth and RGB camera should be analyzed through a machine learning algorithm for object detection and localization (e.g. how far is the table from me?)
3. The feedback should be played back through the speaker in real-time to help the person in navigation.

Suggested Approach

1. Raspberry Pi/NVIDIA Jetson Nano with depth and RGB camera and speaker.
2. Analyze RGB image to detect objects through on-device machine learning (e.g. YOLOV4).
3. Map object bounding box to depth map to identify the distance of the object from the person.
4. Play analysis results back through speaker.

Group Responsibilities

1. Survey hardware and software to figure out the best approach.
2. System design.
3. Implementation.
4. Testing under varying conditions (low lighting, fast motion, difficult angles, noisy environment etc.)

Student A Responsibilities

Hardware + wearable rig

Student B Responsibilities

Programming for image capture and audio playback

Student C Responsibilities

Programming for object detection through machine learning

Student D Responsibilities

Programming for mapping depth map to object detection output

Course Co-requisites

To ALL EDP Students

Due to COVID-19 pandemic, in the event University is not open for in-class/in-lab activities during the Winter term, your EDP topic specifications, requirements, implementations, and assessment methods will be adjusted by your FLCs at their discretion.