RoboComp Logo

A simple robotics framework.

Final Grasping Results and Integration with DSR

DATE : August 7, 2020

In this post, I show the final results of object grasping, using the entire pipeline. Also, I share our integration strategy with DSR graph and the new architecture.

Final Grasping Results (Second Demo)

In the previous post, I showed the complete system integration and the results of path planning using the estimated poses. However, the system should be tested on object grasping and manipulation, as well. Consequently, I managed to extend the embedded Lua scripts with gripper control functions. Then, viriatoGraspingPyrep component was updated to call these scripts using PyRep API. This way, we have a complete grasping pipeline in viriatoGraspingPyrep component, which goes as follows :

  • viriatoGraspingPyrep component opens the arm gripper through embedded Lua scripts.
  • viriatoGraspingPyrep component captures the RGBD signal from the shoulder camera and passes it to pose estimation components.
  • Pose estimation components estimate objects’ poses and send them back to viriatoGraspingPyrep component.
  • viriatoGraspingPyrep component, then, creates a dummy with the estimated pose and calls the embedded Lua scripts to perform path planning.
  • The arm gripper is, then, closed and thus the object is grasped.
  • From there, we can create other dummies for the arm to manipulate or move the object to another position.

Following this procedure, I managed to integrate the full grasping pipeline and create a full demo for grasping using DNN-estimated poses. I used an ensemble of both RGB and RGBD estimated poses to perform pose estimation. Also, I included multiple objects in the scene to provide more challenge for both pose estimation and grasping.

Figure(1): Video of grasping second demo.

Figure(2): Visualization of the DNN-estimated pose in second demo.

DSR Integration Strategy

Figure(3) : Simplified schema for grasping and pose estimation integration with DSR.

As shown in the figure, the components workflow goes as follows :

  • viriatoPyrep component streams the RGBD signal from CoppeliaSim simulator using PyRep API and publishes it to the shared graph through viriatoDSR component.

  • graspDSR component reads the RGBD signal from shared graph and passes it objectPoseEstimation component.

  • objectPoseEstimation component, then, performs pose estimation using DNN and returns the estimated poses.

  • graspDSR component injects the estimated poses into the shared graph and progressively plans dummy targets for the arm to reach the target object.

  • viriatoDSR component, then, reads the dummy target poses from the shared graph and passes it to viriatoPyrep component.

  • Finally, viriatoPyrep component uses the generated poses by graspDSR to progressively plan a successful grasp on the object.

Important Dates

  • August 2, 2020 :

Finish the whole grasping pipeline and create a complete demo.

Commit :

  • August 3, 2020 :

Merge the two pose estimation components into objectPoseEstimation component for integration with DSR.

Commit :

Upcoming Work

  • Start working on graspDSR component.

  • Test the grasping pipeline through the new architecture.

  • Write a full documentation on integration with DSR.

  • Final evaluation and code submission.

Mohamed Shawky