Naimul Mefraz Khan
Signal Processing / Communication
Real-time categorization (what type of sound is it) and localization (where the sound is coming from) of audio can improve the quality of life for people who are deaf or experience severe hearing loss. The maturity of wearable devices, IoT and machine learning, it is possible to create an intelligent device that can provide robust categorization and localization of sound with minimal cost and simple hardware.
To create a wearable computer that can categorize and localize different types of sound to aid the hearing impaired, and display the result on a display device (e.g. "dog barking, on the left side").
1. The wearable computer should have an LCD display and a microphone array. Can be worn anywhere on the body, as long the display is visible.
2. The device should actively listen to the surrounding environment through the microphone array.
3. The audio captured through the microphone should be categorized using a machine learning model that will be developed in this project.
4. The audio captured through the microphone should further be localized using a machine learning model that will be developed in this project.
5. A dataset should be built for the machine learning training if required. Existing datasets such as urban sound 8k can be used.
6. The categorization and localization result should be shown on the device for easy readability.
1. Raspberry Pi/ Nvidia Jetson based wearable rig with an LCD display and a microphone array.
2. Experiment with existing machine learning models for sound categorization and localization.
3. Decide whether the ML model deployment should be local or on a cloud (e.g. Microsoft Azure).
4. Integrate the two separate ML models into a standalone module for easy visualization of results.
1. Build the hardware rig, consider multiple options for an ergonomic design.
2. Literature review on existing ML models for categorization and localization.
3. Find codebase to build upon/ develop codebase.
4. Test the device for limits (distance from microphone, noise etc.).
Build hardware rig
ML model for audio categorization
ML model for audio localization
Overall integration of hardware and software
NMK06: Wearable Computer for Assisting the Hearing Impaired | Naimul Mefraz Khan | Sunday September 1st 2019 at 03:13 PM