Course details

8.3530

Grounding Language into the External Visual World ( Part 1 )

 S
WS 2021 Dr. Elia Bruni ONLINE
4h/wk
8 ECTS 		B.Sc modules:
CS-BWP-CL - (Computational) Linguistics
KOGW-WPM-CL - Computational Linguistics
M.Sc modules:
CC-MWP-CL - Computational Linguistics
CS-MWP-CL - (Computational) Linguistics

CS-BW - Bachelor elective course 		Thu: 8-12

Everyday interactions require a common understanding of language, i.e. for people to communicate effectively, words (for example ‘cat’) should invoke similar beliefs over physical concepts (what cats look like, the sounds they make, how they behave, what their skin feels like etc.). However, how this ‘common understanding’ emerges is still unclear. One appealing hypothesis is that language is tied to how we interact with the environment. As a result, meaning emerges by ‘grounding’ language in modalities in our environment (images, sounds, actions, etc.). This course will review recent works in machine learning which bridges visual and natural language understanding through visually-grounded language learning tasks, e.g. through natural images (Visual Question Answering, Visual Dialog), or through interactions with virtual physical environments. As the grounding problem requires an interdisciplinary attitude, this course aims to gather students with broad expertise in various fields -- machine learning, computer vision, natural language, neuroscience, and psychology -- and who are excited about this space of grounding and interactions. In the second part of this course, we will provide a brisk introduction to the basic tools that make the life of a computational linguist easier. In particular, we will talk about the command-line, powerful text editors, version control systems such as git and GitHub, bash, and many other useful features.