Towards handshape identification for automatic gesture recognition using sign notation systems
Key: AGK+21-1
Author: Philipp Achenbach, Yasmin Göksu, Timo Kullmann, Thomas Tregel, Stefan Göbel
Date: June 2021
Kind: In proceedings
Book title: Proceedings of the 8th European Conference on Social Media ECSM 2021
Keywords: handshape identification, gesture recognition, sign language, sign notation systems, sign learning app
Abstract: Today, about 72 million people worldwide are speaking sign language. Since many deaf people are also dumb, they cannot communicate with hearing people through spoken language, even if they can lip-read. But sign language is difficult to learn, and more than 300 different sign languages in the world make things even more challenging. Therefore, to support the learning of sign language, we want to develop a gamified learning app for sign language that includes automatic sign recognition. The application should provide constructive feedback to the user about the quality of the executed sign. Each sign could be parameterised in terms of its characteristic handshape and its orientation and position: the more parameters are available, the more accurate and detailed feedback can be provided for the user. However, the parameters must also be distinguishable from a technical point of view. In linguistics, different notation systems exist to translate signs into written form. For this, the systems decompose signs into their characteristic properties. We want to utilise these notation systems to reduce signs to parameters that are easy to measure, e.g., the hand's shape, orientation, or position. Since the sign notation systems originate from different fields and have different backgrounds, they also differ in their objectives and thus in numbers and extents of parameters and respective features, further called symbols. Therefore, there are systems whose notations have just enough detail to identify the meant sign and those with so much detail that the reader can reproduce the sign. This higher number of details is reflected in a higher number of parameters and symbols. Hence, we present eleven sign notation systems, starting by examining the handshape as the most concise parameter of sign language. We compare it in the context of notation systems for its suitability for our gamified learning app for sign language. A clear differentiation of the handshapes needed for American Sign Language is essential for qualitative feedback for the user. At the same time, a small number of handshapes should reduce the technical effort required for reliable recognition.

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, not withstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.