11th International Conference on ICT for Sustainable Development

Authors - Aaron Mendonca, Arya Gawde, Nikki Mehta, Nilay Koul, Rohit Parmar, Nikita Raichada
Abstract - Communication barriers have a major influence on the deaf and mute society in India, resulting in social isolation and restricted access to education, employment, and everyday interactions. Indian Sign Language (ISL) is the primary mode of communication, but its lack of widespread understanding restricts integration with the larger society. This study presents a real-time ISL recognition and translation system that integrates deep learning, spatio-temporal analysis, and natural language processing (NLP) to overcome this communication barrier. This study proposes a real-time ISL gesture recognition and translation system utilizing Long Short-Term Memory (LSTM) networks, which are ideal for sequential gesture recognition so that accurate mapping of static and dynamic ISL gestures into text and speech can be done. A spatiotemporal feature extraction pipeline is incorporated using MediaPipe-based skeletal keypoint detection to guarantee strong recognition through capturing hand, facial, and body landmarks. The dataset, created with deaf and mute people’s inputs, provides regional gesture diversity and sign diversity. It has been engineered to operate effectively in real-world environments, with adaptations to lighting changes, background noise, and the complexity of gestures. This work contributes to assistive technology, accessibility, and human computer interaction, fostering social inclusion through facilitating effective communication between the hearing and non-hearing populations. This paper is a step towards a more inclusive digital communication environment, empowering the deaf community in various aspects of life.