NTCIR-10 Math Converter – Converts NTCIR-10 Math datasets, and judgements into NTCIR-11 and NTCIR-12 format

The retrieval unit in the NTCIR-10 Math task dataset is an arXiv document and the judgement unit in the relevance judgements is an XML element. On the other hand, the retrieval and judgement units in the NTCIR-11 Math-2, and NTCIR-12 MathIR task dataset, and relevance judgements is an arXiv document paragraph. This makes it difficult to use both datasets together in a single evaluation.

NTCIR Math converter is a Python 3 command-line utility that converts the NTCIR-10 Math XHTML5 dataset and relevance judgements to the NTCIR-11 Math-2, and NTCIR-12 MathIR XHTML5 format by splitting the dataset into paragraphs and redirecting the relevance judgements from elements to their ancestral paragraphs. As a result, the NTCIR-10 Math dataset, and relevance judgements can be easily used together with the NTCIR-11 Math-2, and NTCIR-12 MathIR dataset, and relevance judgements in a single workflow.

Usage

Installing

The package can be installed by executing the following command: Installing:

$ pip install ntcir10-math-converter

Displaying the usage

Usage information for the package can be displayed by executing the following command:

$ ntcir10-math-converter --help
usage: ntcir10-math-converter [-h] --dataset DATASET [DATASET ...]
                              [--judgements JUDGEMENTS [JUDGEMENTS ...]]
                              [--num-workers NUM_WORKERS]

Convert NTCIR-10 Math XHTML5 dataset and relevance judgements to the NTCIR-11
Math-2, and NTCIR-12 MathIR XHTML5 format.

optional arguments:
  -h, --help            show this help message and exit
  --dataset DATASET [DATASET ...]
                        A path to a directory containing the NTCIR-10 Math
                        XHTML5 dataset, and a path to a non-existent directory
                        that will contain resulting dataset in the NTCIR-11
                        Math-2, and NTCIR-12 MathIR XHTML5 format. If only the
                        path to the NTCIR-10 Math dataset is specified, the
                        dataset will be read to find out the mapping between
                        element identifiers, and paragraph identifiers. This
                        is required for converting the relevance judgements.
  --judgements JUDGEMENTS [JUDGEMENTS ...]
                        Paths to the files containing NTCIR-10 Math relevance
                        judgements (odd arguments), followed by paths to the
                        files that will contain resulting relevance judgements
                        in the NTCIR-11 Math-2, and NTCIR-12 MathIR format
                        (even arguments).
  --num-workers NUM_WORKERS
                        The number of processes that will be used for
                        processing the NTCIR-10 Math dataset. Defaults to 1.

Converting a dataset, and relevance judgements

The following command converts both a dataset, and relevance judgements using 64 worker processes:

$ ntcir10-math-converter --num-workers 64 \
>     --dataset ntcir-10 ntcir-10-converted \
>     --judgements \
>         NTCIR_10_Math-qrels_ft.dat NTCIR_10_Math-qrels_ft-converted.dat \
>         NTCIR_10_Math-qrels_fs.dat NTCIR_10_Math-qrels_fs-converted.dat
Retrieving judged document names, and element identifiers from NTCIR_10_Math-qrels_ft.dat
100%|███████████████████████████████████████████████████████| 1425/1425 [00:00<00:00, 9634.03it/s]
Retrieving judged document names, and element identifiers from NTCIR_10_Math-qrels_fs.dat
100%|███████████████████████████████████████████████████████| 2129/2129 [00:00<00:00, 9671.33it/s]
Processing dataset ntcir-10
Converting dataset ntcir-10 -> ntcir-10-converted/xhtml5
Building a mapping between element identifiers, and paragraph identifiers
100%|████████████████████████████████████████████████████| 100000/100000 [06:45<00:00, 246.50it/s]
Converting relevance judgements NTCIR_10_Math-qrels_ft.dat -> NTCIR_10_Math-qrels_ft-converted.dat
Skipping identifier f080935#idp57072, as it appears outside a paragraph
Skipping identifier f039264#id60072, as it appears outside a paragraph
Skipping identifier f059698#id58538, as it appears outside a paragraph
...
Skipping identifier f023353#idp65840, as it appears outside a paragraph
Skipping identifier f048268#id53551, as it appears outside a paragraph
100%|█████████████████████████████████████████████████████| 1425/1425 [00:00<00:00, 252199.81it/s]
1425 / 1394 input / output relevance judgements
Converting relevance judgements NTCIR_10_Math-qrels_fs.dat -> NTCIR_10_Math-qrels_fs-converted.dat
Skipping identifier f095981#id72919, as it appears outside a paragraph
Skipping identifier f061190#id56357, as it appears outside a paragraph
Skipping identifier f033738#id116089, as it appears outside a paragraph
...
Skipping identifier f019052#id54515, as it appears outside a paragraph
Skipping identifier f021845#id53581, as it appears outside a paragraph
100%|█████████████████████████████████████████████████████| 2129/2129 [00:00<00:00, 291048.96it/s]
2129 / 2076 input / output relevance judgements

Converting only a dataset using 64 worker processes:

$ ntcir10-math-converter --num-workers 64 \
>     --dataset ntcir-10 ntcir-10-converted
Processing dataset ntcir-10
Converting dataset ntcir-10 -> ntcir-10-converted/xhtml5
100%|████████████████████████████████████████████████████| 100000/100000 [07:34<00:00, 220.10it/s]

The following command converts only relevance judgements using 64 worker processes:

$ ntcir10-math-converter --num-workers 64 \
>     --dataset ntcir-10 \
>     --judgements \
>         NTCIR_10_Math-qrels_ft.dat NTCIR_10_Math-qrels_ft-converted.dat \
>         NTCIR_10_Math-qrels_fs.dat NTCIR_10_Math-qrels_fs-converted.dat
Retrieving judged document names, and element identifiers from NTCIR_10_Math-qrels_ft.dat
100%|███████████████████████████████████████████████████████| 1425/1425 [00:00<00:00, 9539.55it/s]
Retrieving judged document names, and element identifiers from NTCIR_10_Math-qrels_fs.dat
100%|███████████████████████████████████████████████████████| 2129/2129 [00:00<00:00, 9332.81it/s]
Processing dataset ntcir-10
Building a mapping between element identifiers, and paragraph identifiers
100%|████████████████████████████████████████████████████████| 2405/2405 [00:16<00:00, 144.41it/s]
Converting relevance judgements NTCIR_10_Math-qrels_ft.dat -> NTCIR_10_Math-qrels_ft-converted.dat
Skipping identifier f080935#idp57072, as it appears outside a paragraph
Skipping identifier f039264#id60072, as it appears outside a paragraph
Skipping identifier f059698#id58538, as it appears outside a paragraph
...
Skipping identifier f023353#idp65840, as it appears outside a paragraph
Skipping identifier f048268#id53551, as it appears outside a paragraph
100%|█████████████████████████████████████████████████████| 1425/1425 [00:00<00:00, 252199.81it/s]
1425 / 1394 input / output relevance judgements
Converting relevance judgements NTCIR_10_Math-qrels_fs.dat -> NTCIR_10_Math-qrels_fs-converted.dat
Skipping identifier f095981#id72919, as it appears outside a paragraph
Skipping identifier f061190#id56357, as it appears outside a paragraph
Skipping identifier f033738#id116089, as it appears outside a paragraph
...
Skipping identifier f019052#id54515, as it appears outside a paragraph
Skipping identifier f021845#id53581, as it appears outside a paragraph
100%|█████████████████████████████████████████████████████| 2129/2129 [00:00<00:00, 291048.96it/s]
2129 / 2076 input / output relevance judgements