AMORE: A distributional MOdel of Reference to Entities

5-year European Research Council (ERC) Starting Grant (nr. 715154; February 2017 - January 2022) funded within the H2020 Programme

News!

  • October 2017: Laura Aina and Ionut-Teodor Sorodoc have joined the team. Welcome!
  • September 2017: Invited talk on AMORE in the GLiF seminar of U. Pompeu Fabra, Barcelona, Spain (slides).
  • September 2017: Invited talk on AMORE at TbiLLC 2017 in Lagodekhi, Georgia (slides).
  • September 2017: Invited talk on AMORE at INLG 2017 in Santiago de Compostela, Spain (slides). Appears in the news here and here.
  • July, August 2017: Matthijs Westera and Carina Silberer have joined the team. Welcome!
  • July 2017: We have been awarded an artist in residency  through the VERTIGO STARTS program! Theo Rhyn (Kate Aspinall) will create a work of art related to AMORE during a 3-month residency.
  • July 2017: Post-doc position awarded to Kristina Gulordava, who will join early 2018. Again, super happy and proud that she accepted!
  • May 2017: PhD positions awarded to Laura Aina and Ionut-Teodor Sorodoc, joining in the fall. We're very lucky to have gotten such great students!
  • March 2017: AMORE is featured in the Spanish newspaper El Periódico (Spanish, Catalan).
  • March 2017: Post-doc positions awarded to Carina Silberer and Matthijs Westera, who will join in the summer. We're very happy and proud to have them onboard!
  • February 2017: The project has started!

Overview

Imagine your GPS could see. To answer the question Do I turn there where that big tree is?, a camera is not enough; the GPS needs to connect what you say to the portion of reality that surrounds your car. AMORE enables machines to connect language to reality, and seeks an understanding of how people make this connection when they talk. The project thus explores the phenomenon of reference in natural language via computational modeling experiments, and we are particularly interested in the interaction of language with conceptual knowledge, on the one hand, and the extralinguistic context, on the other. 

The main challenges are: 1) identifying which entities ("that big tree") are being talked about, both in the visual and in the linguistic camps; 2) tracking the entities as they are mentioned again, retrieving and adding new information about them as needed; 3) crucially, having the machine learn these two abilities directly from examples of how people use language. We face the machine with different tasks that require using language to talk about the world, and it progressively learns to represent both the entities and the language that we use to refer to them. Specifically, we test our computational model in referential tasks that require matching noun phrases (such as the examined boy in the example on the right) with entity representations extracted from text and images. 

This interdisciplinary project builds on two complementary semantic traditions: 1) Formal semantics, a symbolic approach that can delimit and track linguistic referents, but does not adequately match them with the descriptive content of linguistic expressions; 2) Continuous approaches to language such as deep learning models and distributional semantics, which can handle descriptive content but do not associate it to individuated referents. AMORE synthesizes the two approaches into a unified, scalable model of reference that operates with individuated referents and links them to referential expressions characterized by rich descriptive content. The model is a distributed (neural network) version of a formal semantic framework that is furthermore able to integrate perceptual (visual) and linguistic information about entities. 

AMORE advances our scientific understanding of language and its computational modeling, and contributes to the far-reaching debate between symbolic and continuous approaches to cognition with a proposal that falls clearly on the continuous camp, but integrates key insights from the symbolic camp.

More background:

Team


Senior researcher: Louise McNally
Post-docs: Carina Silberer, Matthijs WesteraKristina Gulordava (the latter starting early 2018)
PhD students
: Laura Aina, Ionut-Teodor Sorodoc

Advisory Board

Contact

Write to gemma DOT boleda AT upf DOT edu.

(Image credits: Hagerty Ryan, USFWS)

This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 715154).

https://erc.europa.eu/about-erc http://ec.europa.eu/programmes/horizon2020/