Logo ÚFAL Logo MFF

Hindi Visual Genome


Hindi-English Multimodal Dataset

ÚFAL
Hindi Visual Genome

WAT2020 Multi-Modal Translation Task

After a positive response from the participants for the “WAT 2019 Multimodal Translation Task”, the Workshop on Asian Translation 2020 (WAT2020) will continue the task of multimodal English-to-Hindi translation which is the first multimodal translation task for any Indian language. The task relies on our “Hindi Visual Genome”, a multimodal dataset consisting of text and images suitable for English-Hindi multimodal machine translation task and multimodal research.

Timeline

  • August 15, 2020: Release of Data set
  • September 18, 2020: Translations need to be submitted to the organizers
  • October 23, 2020: System description paper submission deadline
  • October 30, 2020: Review feedback for system description
  • November 6, 2020: Camera-ready
  • December 4-7, 2020: WAT2020 takes place

Task Description

The setup of the WAT2020 task is as follows:

  • Inputs:
    • An image,
    • A rectangular region in that image
    • A short English caption of the rectangular region.
  • Output:
    • The caption translated to Hindi.

Types of Submissions Expected

The setup of the WAT2020 task is as follows:

  • Text-only translation
  • Hindi-only image captioning
  • Multi-modal translation (uses both the image and the text)

Training Data

The Hindi Visual Genome consists of:

  • 29k training examples
  • 1k dev set
  • 1.6k evaluation set

Evaluation

WAT2020 Multi-Modal Task will be evaluated on:

  • 1.6k evaluation set of Hindi Visual Genome
  • 1.4k challenge set of Hindi Visual Genome

Means of evaluation:

  • Automatic metrics: BLEU, CHRF3, and others
  • Manual evaluation, subject to the availability of Hindi speakers

Participants of the task need to indicate which track their translations belong to:

  • Text-only / Image-only / Multi-modal
    • see above
  • Domain-Aware / Domain-Unaware
    • Whether the full (English) Visual Genome was used in the training or not.
  • Constrained / Non-Constrained
    • 29k training segments from the Hindi Visual Genome
    • HindEnCorp 0.5
    • (English-only) Visual Genome [making the submission a domain-aware run]
  • Non-constrained submission may use other data, but need to specify what data was used.
       

Download Link

 

Submission Requirement

The system description should be a short report (4 to 6 pages) submitted to WAT 2020 describing the method(s).

Each participating team can submit at most 2 systems for each of the task (e.g. Text-only, Hindi-only image captioning, multimodal translation using text and image). Please submit through the submission link available in the WAT2020 website and select the task for submission.   

Paper and References

Please refer to the below papers:

[paper] : https://www.cys.cic.ipn.mx/ojs/index.php/CyS/article/view/3294

[arxiv] : https://arxiv.org/abs/1907.08948

[WAT 2019 Proceedings] : https://www.aclweb.org/anthology/D19-5200/

[Reference Papers]

Idiap NMT System for WAT 2019 Multimodal Translation Task

English to Hindi Multi-modal Neural Machine Translation and Hindi Image Captioning

WAT2019: English-Hindi Translation on Hindi Visual Genome Dataset

Presentation

The overview presentation includes the task, participants, results, and analysis for the WAT2019 Multimodal task. Presented at EMNLP 2019 Hongkong (WAT Workshop) on 4th Nov 2019. [ppt] 

Organizers

  • Shantipriya Parida (Idiap Research Institute, Switzerland)
  • Ondřej Bojar (Charles University, Czech Republic)

Contact

email: wat-multimodal-task@ufal.mff.cuni.cz

License

The data is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International.

Acknowledgement

This shared task is supported by the below projects/grants from Idiap Research Institute (Switzerland) and Charles University (Czech Republic).

  • European Union, Project code: EC/H2020/833635, Project name: ROXANNE - Real time network, text, and speaker analytics for combating organized crime
  • InnoSuisse, Project code: 29814.1 IP-ICT, Project name: SM2: Extracting Semantic Meaning from Spoken Material” funding application no. 29814.1 IP-ICT
  • Grantová agentura České republiky, Project code: 19-26934X, Project name: Neural Representations in Multi-modal and Multi-lingual Modelling