Final Project Submission

This is the final post of Monocular Depth Estimation using RBG signals project.I will present all the work done and results of the project with link to all my contributions and issues that needs to resolved along with future extensions.

Brief Description

Depth Estimation is one of the attractive research areas in computer vision. The motivation of this project is to estimate depth without the use of LIDAR sensors or any other related costly setups. And it also makes the robot more robust and efficient to carry out any task using the crucial depth information.

In this project, Myself, Vaibhaw Khemka, worked with RoboComp and my mentor, Mohamed Shawky,to develop real-time depth estimation pipeline using latest research and advances in Deep Learning. I have build a Deep Neural Network(DNN) architecture (named MobDepth) which is integrated to depthEstimation component built on RoboComp Framework. It gives high quality depth estimation in both real and simulated world and works well with complex indoor scenes.Finally, I worked in integration of depth estimation component into CORTEX architecture(based on Deep State Representation) by building depthDSR agent.

Project Details

Summary of all the tasks done in each period :

Community Bonding Period

• Communicated with my mentor, Mohamed Shawky, about the project plan and workflow.

• Set up my development environment and started familiarizing more with RoboComp Frameworks.

• Identified different challenges that exist in the project like: * Trade off between Accuracy and Speed of the model. * Model should be Generalizable and robust.

• Started working on implementing different literatures for building depth estimation neural network.

  First Coding Period

• Results obtained on existing literatures for depth estimation were not good as it didn’t satisfy both real-time and high quality estimation of depth.

• Started building my own architecture(MobDepth) using Transfer Learning on MobileNet architecture to meet the requirement of real-time and high quality depths.

• Trained on NYU dataset, which consist of complex indoor scenes to make it more generalizable.

• Tested on real world scenarios and added skip connections in the model which gave a very good result.

• Prepared the dataset using python data collector from coppeliaSim for model to also work in a simulated world and augmented it with NYU dataset.

• Retrained/Fine-Tuned the network on augmented dataset.

Second Coding Period

• Build depthEstimation component and interfaces in consistent with RoboComp Framework.

• Integrated and tested the trained depth estimation model with the component.

• Started familiarizing with CORTEX architecture(based on Deep State Representation) and discussed with my mentor, Mohamed Shawky to gain more exposure on this.

• Studied DSR architecture through installation and tested different agents like - graspDSR, yolov4-detector etc.

• Implemented depthDSR C++ agent which acts as an interface between depthEstimation component and shared graph(G). It takes RGB image from Graph and estimate depth using depthEstimation component.

• Wrote a detailed documentation about DSR integration and its problems.

• Faced multiple issues and problems while integration With DSR.

• Resolved most of the issues with help of my mentor, Mohamed Shawky. But there are still some issues while connecting with depthEstimation component.

• Updated and improved the documentation of all components and agents.

Results and Demonstrations

Fig-1: Depth Map for Real world Scenes

Fig-2: Depth Map before and after Fine-Tuning

Fig-3: Demo of depthEstimation component in simulated world

https://user-images.githubusercontent.com/46538042/129046230-2305ee52-9609-4ef5-b81c-8d21989ff2da.mp4

Contributions Summary

| Repo | Commits | PRs | Issues | | ————- | ————- | ————– | ————- | | robocomp/DNN-Services | link | link | - | | robocomp/robocomp | link | link | link | | robocomp/dsr-graph | link | link | link | | robocomp/web | link | link | - |

Future Works and Improvements

The work with the project has been completed successfully with demos and documentation except the issue of depthDSR while connecting with depthEstimation component.So,Summary of future improvements are as follows:

• Resolve the issue of depthDSR while connecting with depthEstimation component.

• Extend the depth estimation model to work better even for outdoor scenes as well.

• Build Collision-detector component with help of depth estimated from the agent.

• Update and maintain the components in consistent with RoboComp regularly.

• Explore more applications of depth estimation which can be integrated with RoboComp.

Vaibhaw Khemka