RoboComp Logo

A simple robotics framework.

GSoC’21 RoboComp project: Sign language recognition

14th Aug 2021


This component requires the same environment as HandPose and ImageBasedRecognition component, please check this link.


Pose based Sign language recognition:

The pose detector component is in robocomp-robolab/components/detection/BodyHandJointsDetector/ . The recognized component is in robocomp-robolab/components/detection/poseBasedGestureRecognition . and its client is in robocomp-robolab/components/detection/test/poseBasedGestureRecognitionClient/

Copy pretrained model from this link to src/_model/ in the detector component folder.

Steps: 1) Follow the introduction in the to cmake the folder, and run the component. 2) Run the detector and recognizer before the client.

This client will get image from camera and send to detector and get back the body pose + hand joints. Then these poses are collected and send to recognizer to get the class of gesture.

Currently, this component is still debugging.

WLASL dataset:

As mentioned, we use WLASL dataset to recognize sign language. For demonstration, we just use the 100 classes WLASL models. Therefore, if users want to apply for larger range of classes in WLASL (1000 or 2000). Please follow these links:

Please clone this git for testing and converting pretrained pytorch model. The convert code can be found here:


HandBodyPoseDetector component/client:

handbody demo

ImageBasedRecognition component/client:

image based recognition demo