Final Project Submission
This is the final post of
Monocular Depth Estimation using RBG signals project.I will present all the work done and results of the project with link to all my contributions and issues that needs to resolved along with future extensions.
Depth Estimation is one of the attractive research areas in computer vision. The motivation of this project is to estimate depth without the use of LIDAR sensors or any other related costly setups. And it also makes the robot more robust and efficient to carry out any task using the crucial depth information.
In this project, Myself, Vaibhaw Khemka, worked with RoboComp and my mentor, Mohamed Shawky,to develop real-time depth estimation pipeline using latest research and advances in Deep Learning. I have build a Deep Neural Network(DNN) architecture (named
MobDepth) which is integrated to
depthEstimation component built on RoboComp Framework. It gives high quality depth estimation in both real and simulated world and works well with complex indoor scenes.Finally, I worked in integration of depth estimation component into CORTEX architecture(based on Deep State Representation) by building
Summary of all the tasks done in each period :
Community Bonding Period
Communicated with my mentor, Mohamed Shawky, about the project plan and workflow.
Set up my development environment and started familiarizing more with RoboComp Frameworks.
Identified different challenges that exist in the project like: * Trade off between Accuracy and Speed of the model. * Model should be Generalizable and robust.
- Started working on implementing different literatures for building depth estimation neural network.
First Coding Period
Results obtained on existing literatures for depth estimation were not good as it didn’t satisfy both real-time and high quality estimation of depth.
Started building my own architecture(
MobDepth) using Transfer Learning on MobileNet architecture to meet the requirement of real-time and high quality depths.
NYU dataset, which consist of complex indoor scenes to make it more generalizable.
Tested on real world scenarios and added skip connections in the model which gave a very good result.
Prepared the dataset using python data collector from coppeliaSim for model to also work in a simulated world and augmented it with
- Retrained/Fine-Tuned the network on augmented dataset.
Second Coding Period
depthEstimationcomponent and interfaces in consistent with RoboComp Framework.
Integrated and tested the trained depth estimation model with the component.
Started familiarizing with CORTEX architecture(based on Deep State Representation) and discussed with my mentor, Mohamed Shawky to gain more exposure on this.
depthDSRC++ agent which acts as an interface between
depthEstimationcomponent and shared graph(G). It takes RGB image from Graph and estimate depth using
Wrote a detailed documentation about DSR integration and its problems.
Faced multiple issues and problems while integration With DSR.
Resolved most of the issues with help of my mentor, Mohamed Shawky. But there are still some issues while connecting with
Updated and improved the documentation of all components and agents.
Results and Demonstrations
Fig-1: Depth Map for Real world Scenes
Fig-2: Depth Map before and after Fine-Tuning
Fig-3: Demo of depthEstimation component in simulated world
| Repo | Commits | PRs | Issues | | ————- | ————- | ————– | ————- | | robocomp/DNN-Services | link | link | - | | robocomp/robocomp | link | link | link | | robocomp/dsr-graph | link | link | link | | robocomp/web | link | link | - |
Future Works and Improvements
The work with the project has been completed successfully with demos and documentation except the issue of
depthDSR while connecting with
depthEstimation component.So,Summary of future improvements are as follows:
Resolve the issue of
depthDSRwhile connecting with
Extend the depth estimation model to work better even for outdoor scenes as well.
Collision-detectorcomponent with help of depth estimated from the agent.
Update and maintain the components in consistent with RoboComp regularly.
Explore more applications of depth estimation which can be integrated with RoboComp.