Wearable Computer for Assisting the Hearing Impaired

2021 COE Engineering Design Project (NMK06)

---

Faculty Lab Coordinator

Naimul Mefraz Khan

Topic Category

Software Systems

Preamble

Real-time categorization (what type of sound is it) and localization (where the sound is coming from) of audio can improve the quality of life for people who are deaf or experience severe hearing loss. The maturity of wearable devices, IoT and machine learning, it is possible to create an intelligent device that can provide robust categorization and localization of sound with minimal cost and simple hardware.

Objective

To create a wearable computer that can categorize and localize different types of sound to aid the hearing impaired, and display the result on a display device (e.g. "dog barking, on the left side").

Partial Specifications

1. The wearable computer should have an LCD display and a microphone array. Can be worn anywhere on the body, as long the display is visible.

2. The device should actively listen to the surrounding environment through the microphone array.

3. The audio captured through the microphone should be categorized using a machine learning model that will be developed in this project.

4. The audio captured through the microphone should further be localized using a machine learning model that will be developed in this project.

5. A dataset should be built for the machine learning training if required. Existing datasets such as urban sound 8k can be used.

6. The categorization and localization result should be shown on the device for easy readability.

Suggested Approach

1. Raspberry Pi/ Nvidia Jetson based wearable rig with an LCD display and a microphone array.

2. Experiment with existing machine learning models for sound categorization and localization.

3. Decide whether the ML model deployment should be local or on a cloud (e.g. Microsoft Azure).

4. Integrate the two separate ML models into a standalone module for easy visualization of results.

Group Responsibilities

1. Build the hardware rig, consider multiple options for an ergonomic design.

2. Literature review on existing ML models for categorization and localization.

3. Find codebase to build upon/ develop codebase.

4. Test the device for limits (distance from microphone, noise etc.).

Student A Responsibilities

Build hardware rig

Student B Responsibilities

ML model for audio categorization

Student C Responsibilities

ML model for audio localization

Student D Responsibilities

Overall integration of hardware and software

Course Co-requisites

To ALL EDP Students

Due to COVID-19 pandemic, in the event University is not open for in-class/in-lab activities during the Winter term, your EDP topic specifications, requirements, implementations, and assessment methods will be adjusted by your FLCs at their discretion.