Towards Further Comprehension on Referring Expression with Rationale

Rengang Li, Baoyu Fan, Xiaochuan Li, Runze Zhang, Zhenhua Guo, Kun Zhao, Yaqian Zhao, Weifeng Gong, Endong Wang

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Referring Expression Comprehension (REC) is one important research branch in visual grounding, where the goal of REC is to localize a relevant object in the image, given an expression in the form of text to exactly describe a specific object. However, existing REC tasks aim at text content filtering and image object locating, which are evaluated based on the precision of the detection boxes. This may lead models to skip the learning process of multimodal comprehension directly and achieve good performance. In this paper, we work on how to enable an artificial agent to understand RE further and propose a more comprehensive task, called Further Comprehension on Referring Expression (FREC). In this task, we mainly focus on three sub-tasks: 1) correcting the erroneous text expression based on visual information; 2) generating the rationale of this input expression; 3) localizing the proper object based on the corrected expression. Accordingly, we make a new dataset named Further-RefCOCOs based on the RefCOCO, RefCOCO+, RefCOCOg benchmark datasets for this new task and make it publicly available. After that, we design a novel end-to-end pipeline to achieve these sub-tasks simultaneously. The experimental results demonstrate the validity of the proposed pipeline. We believe this work will motivate more researchers to explore along with this direction, and promote the development of visual grounding.

Original languageEnglish
Title of host publicationMM 2022 - Proceedings of the 30th ACM International Conference on Multimedia
PublisherAssociation for Computing Machinery, Inc
Pages4336-4344
Number of pages9
ISBN (Electronic)9781450392037
DOIs
Publication statusPublished - 10 Oct 2022
Externally publishedYes
Event30th ACM International Conference on Multimedia, MM 2022 - Lisboa, Portugal
Duration: 10 Oct 202214 Oct 2022

Publication series

NameMM 2022 - Proceedings of the 30th ACM International Conference on Multimedia

Conference

Conference30th ACM International Conference on Multimedia, MM 2022
Country/TerritoryPortugal
CityLisboa
Period10/10/2214/10/22

Keywords

  • computational linguistics
  • multimodal learning
  • referring expression comprehension
  • visual grounding

Fingerprint

Dive into the research topics of 'Towards Further Comprehension on Referring Expression with Rationale'. Together they form a unique fingerprint.

Cite this