RoboComp Logo

A simple robotics framework.

Final Project Submission

DATE : August 24, 2020

This is the final post of DNN’s for precise manipulation of household objects project. Here, I show the work done and the results of the project. Also, I discuss some future improvements and link to all my contributions during the project.

Brief Description

The ability of a robot to detect, grasp and manipulate objects is one of the key challenges in building an intelligent collaborative humanoid robot. In this project, I, Mohamed Shawky, worked with RoboComp and my mentor, Pablo Bustos, to develop fast and robust pose estimation and grasping pipeline using the power of RoboComp framework and recent advances in Deep Learning. I used two recent DNN architctures, which are Segmentation-driven 6D Object Pose Estimation and PVN3D to build pose estimation component that estimates objects poses from RGBD images. Also, I was able to integrate the work done with Kinova Gen3 arm with PyRep API for a fast and robust path planning and grasping workflow. Finally, I worked on the integration of pose estimation and grasping into CORTEX archiecture (based on Deep State Representation (DSR) and implemented using Real-time Pub/Sub (RTPS) and Conflict-Free Replicated Data Type (CRDT)).

Completed Work

Here is a summary of all the work done through each period of the project.

Community Bonding Period

  • Communicated with my mentor, Pablo Bustos, to discuss the previous work on the grasping problem and form detailed project workflow.

  • Familiarized myself more with RoboComp framework and 6D object pose estimation literature.

  • Started working on Segmentation-driven 6D Object Pose Estimation network implementation based on open-source inference code.

First Coding Period

  • Completed a full implementation of Segmentation-driven 6D Object Pose Estimation network.

  • Trained the network on YCB-Videos dataset, which is one of the most diverse open-source datasets.

  • Tested the trained network on YCB-Videos dataset and data collected from CoppeliaSim environments.

  • Implemented a Python data collector using PyRep API, Open3D and CoppeliaSim to collect more data for augmentation.

  • Retrained the network on augmented dataset.

Second Coding Period

  • Improved the implementation of Segmentation-driven 6D Object Pose Estimation network through using focal loss and fixing some training issues.

  • Retrained the network on the final settings.

  • Investigated further improvements of pose estimation pipeline, which led me to PVN3D network.

  • Studied PVN3D network and integrated its open-source implementation to a new RoboComp component.

  • Completed two pose estimation components. One for RGB inference using Segmentation-driven 6D Object Pose Estimation network and another for RGBD inference using PVN3D network.

  • Adapted the previous work with Kinova Gen3 arm to PyRep API, where I added embedded Lua scripts to the arm’s model and called them, remotely, through PyRep API. Thus, we have a fast and robust workflow for path planning and grasping.

  • Implemented viriatoGraspingPyrep component, which integrates and tests the whole pose estimation and grasping pipeline.

Third Coding Period

  • Created the first complete grasping demo of Kinova Gen3 arm.

  • Studied the workflow of the new DSR architecture through installation, testing and code review.

  • Discussed the process of pose estimation and grasping integration into DSR with my mentor, Pablo Bustos.

  • Merged both RGB and RGBD pose estimation components into one component, named objectPoseEstimation, for DSR integration.

  • Implemented an object detection component that uses RetinaNet to improve object tracking in DSR.

  • Implemented graspDSR C++ agent, which is an interface between objectPoseEstimation component and the shared graph (a.k.a. G). Also, it plans the dummy targets for the robot arm to reach the required object.

  • Tested the validity of the arm’s embedded Lua scripts using DSR.

  • Helped debugging and improving some features of DSR Core API.

  • Wrote a detailed documentation about DSR integration and its problems.

  • Improved visualization and memory allocation in pose estimation components.

  • Revisited all the previous work and improved some documentation.

Results and Demonstrations

Figure(1): A sample of the collected output from the Python data collector.


Figure(2): Visualizations of the estimated poses by Segmentation-driven 6D Object Pose Estimation on YCB-Videos dataset.


Figure(3): Visualizations of the estimated poses by objectPoseEstimation component in CoppeliaSim environment.


IMAGE ALT TEXT
Figure(4): Demo of path planning using the DNN-estimated poses and the embedded Lua scripts.


IMAGE ALT TEXT
Figure(5): Complete Demo of the arm grasping and manipulation using DNN-estimated poses without DSR.


IMAGE ALT TEXT
Figure(6): Test demo of the embedded Lua scripts in DSR using the simulator poses.


Contributions Summary

Repo Commits PRs Issues
robocomp/grasping link - -
robocomp/dsr-graph link link link
robocomp/robocomp link link link
robocomp/DNN-Services link - -
robocomp/web link link -

Future Work and Improvements

The work with this project has been completed successfully with full documentation and a decent set of demos. However, there is still some work to be done with DSR framework to be able to completely use the implemented pose estimation and grasping pipeline through DSR. This work can be summarized as follows :

  • Solve the related open issues on robocomp/robocomp repo.

  • Update viriatoDSR to pass the arm target poses to viriatoPyrep from G.

  • Add grasping methods to viriatoPyrep, similar to viriatoGraspingPyrep.

  • Test the pose estimation and grasping pipeline with tracking and social navigation, once done.

Mohamed Shawky