Information Extraction

Covid-19 MLIA Eval

Task Description

The goal of the Information Extraction task is...

To participate in the Information Extraction task, the groups need to register at the following link:

Register

Important Dates - Round 1

Round starts: June xx, 2020

Corpora and topics released: June xx, 2020

Runs due from participants: June xx, 2020

Ground-truth released and runs scored: June xx, 2020

Synchronization meeting (remote): July xx, 2020

Round ends: July xx, 2020

Participation Guidelines

Some description here...

Participant Repository:

Participants are provided with a single repository for all the tasks they take part in. The repository contains the runs, resources, code, and report of each participant.

The repository is organised as follows:

Covid-19 MLIA Eval consists of three tasks run in three rounds. Therefore, the submission and score folders are organized into sub-folders for each task and round as follows:

Participants which do not take part in a given task or round can simply delete the corresponding sub-folders.

The goal of Covid-19 MLIA Eval is to speed up the creation of multilingual information acces systems and (language) resources for Covid-19 as well as openly share these systems and resources as much as possible. Therefore, participants are more than encouraged to share their code and any additional (language) resources they have used or created.

All the contents of these repositories are realeased under the Creative Commons Attribution-ShareAlike 4.0 International License.

Task Repository:

Organizers share contents common to all participants through the Information Extraction task repository.

The repository is organised as follows:

Covid-19 MLIA Eval runs in three rounds. Therefore, the topics and ground-truth folders are organized into sub-folders for each round, i.e. round1, round2, and round3.

All the contents of this repository are realeased under the Creative Commons Attribution-ShareAlike 4.0 International License.

Rolling Technical Report:

The rolling technical report should be formatted according to the Springer LNCS format, using either the LaTeX or the Word template. LaTeX is the preferred format.

Corpora:

The Beautiful corpus contains over 1.8 million articles... The dataset is available here.

Topics and Ground-truth:

Submission Guidelines

Participating teams should satisfy the following guidelines:

Trec Format:

Runs should be submitted with the following format:


30 Q0 ZF08-175-870  0 4238 prise1
30 Q0 ZF08-306-044  1 4223 prise1
30 Q0 ZF09-477-757  2 4207 prise1
30 Q0 ZF08-312-422  3 4194 prise1
30 Q0 ZF08-013-262  4 4189 prise1
...
   						
where: It is important to include all the columns and have a white space delimiter between the columns.

Submission Upload:

Runs should be uploaded in the repository provided by the organizers. Following the repository structure discussed above, for example, a run submitted for the first round of the Information Extraction task should be included in submission/task1/round1.

Runs should be uploaded with the following name convention: <teamname>_task1_<freefield> where teamname is the name of the participating team, task1 is the identifier of the Information Extraction task, and freefield is a free field that participants can use as they prefer.

Performance scores for the submitted runs will be returned by the organizers in the score folder, which follows the same structure as the submission folder.

The rolling technical report has to be uploaded and kept update in the report folder.

Here, you can find a sample participant repository to get a better idea of its layout.

Evaluation:

The quality of the submitted runs will be evaluated with the following measures:

Organizers

Thierry Declerck, DFKI, Germany
declerckdfki.de

Pierre Zweigenbaum, LMSI, France
pzlimsi.fr