Sign Pulse: a hybrid AI-based learning framework for enhancing education among deaf and hard-of-hearing students.

Authors

Fidaa Khandaqji, Huthaifa I Ashqar, Abdelrahem Atawnih

Published in

Disability and rehabilitation. Assistive technology. Pages 1-18. Aug 17, 2026. Epub Aug 17, 2026.

Abstract

Sign Pulse is a hybrid AI-driven educational framework designed to enhance STEM learning for deaf and hard-of-hearing (DHH) students in grades 1-4 (approximately 6-10 years of age) through native sign language interaction. Addressing the limited availability for low-resource sign languages, the system is among the first to focus on Palestinian Sign Language (PSL) within a classroom-ready learning environment.
Sign Pulse integrates real-time sign language recognition with intelligent pedagogical feedback by combining a Siamese Vision Transformer and a β-Variational Autoencoder (β-VAE) for robust spatial-temporal gesture modelling, together with Large Language Models (LLMs) augmented through Retrieval-Augmented Generation (RAG) to deliver curriculum-aligned explanations, questions, and feedback. The system was evaluated on a newly developed PSL dataset consisting of 434 annotated classroom video samples covering foundational STEM concepts.
Experimental results demonstrate strong recognition and verification performance, achieving a Top-1 accuracy of 90%, Top-5 accuracy of 99.65%, an AUC of 92%, and an equal error rate (EER) of 7.8%. Beyond technical performance, the integration of RAG-enhanced LLMs significantly improved the quality of educational feedback, yielding a BERT Score F1 of 88% and a ROUGE-Recall of 55%, substantially outperforming baseline fixed-response systems. These results demonstrate that Sign Pulse effectively transforms sign language recognition from a standalone accessibility tool into an interactive learning system that supports personalised, concept-driven STEM education. By unifying low-resource sign language data, advanced vision-based recognition, and generative pedagogical reasoning within a single framework, Sign Pulse offers a scalable and replicable model for inclusive AI-powered education for DHH learners in underrepresented linguistic communities.

PMID:
42606369
Bibliographic data and abstract were imported from PubMed on 17 Aug 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.