Organizers will provide training data for all language pairs. Participants can use these data and are also free to use out-of-domain data.
Participants will use their systems to translate a test set of unseen sentences in the source language.
The translation quality is measured by various automatic evaluation metrics.
You may participate in any or all of the language pairs. Organizers provide a common framework to upload submissions where results can be compared. Multilingual systems, parsers, or morphological analyzers are allowed.
Participants are provided with a single repository for all the tasks they take part in.
The repository contains the runs, resources, code, and report of each participant.
The repository is organised as follows:
submission: this folder contains the runs submitted for the different tasks in the different evaluation rounds.
score: this folder contains the performance scores of the submitted runs.
code: this folder contains the source code of the developed system.
resource: this folder contains (language) resources created during the participation.
report: this folder contains the rolling technical report describing the techniques applied and insights gained during participation, round after round.
Covid-19 MLIA Eval consists of three tasks run in three rounds.
score folders are organized into sub-folders for each task and round as follows:
submission/task1/round1: for the runs submitted to the first round of the first taks. Similar structure for the other tasks and rounds.
score/task1/round1: for the performance scores of the runs submitted to the first round of the first taks. Similar structure for the other tasks and rounds.
Participants which do not take part in a given task or round can simply delete the corresponding sub-folders.
The goal of Covid-19 MLIA Eval is to speed up the creation of multilingual information access systems and (language) resources for Covid-19 as well as openly share these systems and resources as much as possible. Therefore, participants are more than encouraged to share their code and any additional (language) resources they have used or created.
All the contents of these repositories are realeased under the Creative Commons Attribution-ShareAlike 4.0 International License.
Rolling Technical Report:
The rolling technical report should be formatted according to the Springer LNCS format, using either the LaTeX or the Word template. LaTeX is the preferred format.
The training data contains over 1 million examples. The dataset is available here.
The test dataset will be available soon.