Towards More Likely Models for AI Planning
Turguy Caglar, Sirine Belhaj, et al.
IJCAI 2023
In this paper, we explore self-supervised audio-visual models that learn from instructional videos. Prior work has shown that these models can relate spoken words and sounds to visual content after training on a large-scale dataset of videos, but they were only trained and evaluated on videos in English. To learn multilingual audio-visual representations, we propose a cascaded approach that leverages a model trained on English videos and applies it to audio-visual data in other languages, such as Japanese videos. With our cascaded approach, we show an improvement in retrieval performance of nearly 10x compared to training on the Japanese videos solely. We also apply the model trained on English videos to Japanese and Hindi spoken captions of images, achieving state-of-the-art performance.
Turguy Caglar, Sirine Belhaj, et al.
IJCAI 2023
Eduardo Almeida Soares, Dmitry Zubarev, et al.
ICLR 2025
Srikanth Tamilselvam, Dinesh Khandelwal, et al.
ACML 2022
Eduardo Almeida Soares, Victor Shirasuna, et al.
ACS Fall 2024