Register now After registration you will be able to apply for this opportunity online.
Image Based Robust Pose Estimation for General Excavator Buckets
The efficient operation of excavators in construction environments necessitates precise pose estimation of their buckets. Current methods rely on IMUs placed on the excavator arm which require tedious calibration and can be damaged during construction operations. This project aims to leverage computer vision and machine learning to enhance pose estimation, thereby enabling VR overlays for teleoperation and facilitating automation tasks.
The primary objective of this project is to develop a robust computer vision pipeline capable of accurately estimating the pose of excavator buckets using an off-the-shelf stereo camera setup. The project seeks to generalize the pose estimation model across various bucket geometries, including scenarios with occlusions and challenging lighting conditions. The student will be able to build on an existing pipeline which uses RGB images as well as LiDAR point clouds, and will explore synthetic data generation using the Isaac Sim simulator.
The primary objective of this project is to develop a robust computer vision pipeline capable of accurately estimating the pose of excavator buckets using an off-the-shelf stereo camera setup. The project seeks to generalize the pose estimation model across various bucket geometries, including scenarios with occlusions and challenging lighting conditions. The student will be able to build on an existing pipeline which uses RGB images as well as LiDAR point clouds, and will explore synthetic data generation using the Isaac Sim simulator.
- Literature Review: Conduct an extensive review of existing methods for pose estimation in industrial settings, with a focus on computer vision techniques and stereo camera setups. Identify key challenges and promising approaches in the field.
- Data Synthesis: Gather real-world RGB data capturing excavator operations, including various bucket shapes and environmental conditions. Augment this dataset with synthetic data generated using state-of-the-art simulation environments, simulating diverse scenarios encountered in construction sites.
- Pipeline Development: Design and implement a computer vision pipeline for pose estimation, integrating image processing algorithms, feature extraction, and machine learning techniques. Explore deep learning architectures suitable for handling stereo image inputs and capable of generalizing across different bucket geometries.
- Model Training and Evaluation: Train the pose estimation model using the collected dataset, employing transfer learning and data augmentation techniques to enhance generalization. Evaluate the performance of the trained model using metrics such as accuracy, robustness to occlusions, and computational efficiency.
- Literature Review: Conduct an extensive review of existing methods for pose estimation in industrial settings, with a focus on computer vision techniques and stereo camera setups. Identify key challenges and promising approaches in the field. - Data Synthesis: Gather real-world RGB data capturing excavator operations, including various bucket shapes and environmental conditions. Augment this dataset with synthetic data generated using state-of-the-art simulation environments, simulating diverse scenarios encountered in construction sites. - Pipeline Development: Design and implement a computer vision pipeline for pose estimation, integrating image processing algorithms, feature extraction, and machine learning techniques. Explore deep learning architectures suitable for handling stereo image inputs and capable of generalizing across different bucket geometries. - Model Training and Evaluation: Train the pose estimation model using the collected dataset, employing transfer learning and data augmentation techniques to enhance generalization. Evaluate the performance of the trained model using metrics such as accuracy, robustness to occlusions, and computational efficiency.
- Experience in neural network training frameworks such as PyTorch or TensorFlow.
- Strong background in computer vision, image processing, and machine learning.
- Familiarity with Python and ROS (Robot Operating System).
- Experience in simulation environments for generating synthetic data is desirable.
- Experience in neural network training frameworks such as PyTorch or TensorFlow. - Strong background in computer vision, image processing, and machine learning. - Familiarity with Python and ROS (Robot Operating System). - Experience in simulation environments for generating synthetic data is desirable.