Register now After registration you will be able to apply for this opportunity online.
3D Hand Forecasting (HoloAssist: Interactive AI Assistants)
3D hand pose forecasting is a new benchmark introduced by HoloAssist [1]. Existing action forecasting work mostly focuses on providing semantic labels of future actions and does not provide explicit 3D guidance on hand poses. Predicting 3D hand poses can be useful for various applications, and it can augment instructions and spatially guide users in different tasks. In this benchmark, we take 3 seconds inputs similar to other 3D body location forecasting literature and forecast the continuous 3D hand poses for the next 0.5, 1.0, and 1.5 seconds. The evaluation metric is the average of mean per joint position error over time in centimeters compared to ground truth. To have a proper evaluation metric that can help 3D action guidance, we remove the mistakes from the action sequences and only forecast 3D hand pose for the correct labels.
[1] Wang, X., Kwon, T., Rad, M., Pan, B., Chakraborty, I., Andrist, S., ... & Pollefeys, M. (2023). Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 20270-20281).
Keywords: Action Recognition, Video Understanding, Hand forecasting, Hand pose, AI, AI Agent.
Qualifications:
- Experience in Python.
- Interest in Mixed Reality.
- Interest in Machine Learning and Computer Vision.
In this project, the students will
- Develop algorithms that will achieve state-of-the-art results in 3D hand forecasting.
Qualifications:
- Experience in Python.
- Interest in Mixed Reality.
- Interest in Machine Learning and Computer Vision.
In this project, the students will - Develop algorithms that will achieve state-of-the-art results in 3D hand forecasting.
Implement an algorithm that can forecast 3D hand poses
Implement an algorithm that can forecast 3D hand poses
Please send an email with your CV and transcript to apply for this opportunity.
Taein Kwon taein.kwon@inf.ethz.ch
Please send an email with your CV and transcript to apply for this opportunity.