The code in these files is concerned with computing the Dense Captioning metric. The metric is inspired by the meap Average Precision in object detection literature in that it uses a set of thresholds, for each threshold it computes average precision, and the final number is the mean. In ordinary object detection one would sweep through IoU (Intersection over Union) overlaps at thresholds of 0.3, 0.4, 0.5, 0.6, 0.7 (the latest recommendation is to use all of these thresholds instead of only using 0.5, as done previously), and consider a candidate detection to be correct if it satisfied the desired overlap. In Dense Captioning we don't only evaluate localization but also captioning. Therefore, we also compute the METEOR score for every candidate detection and consider a candidate detection correct if it is both above a localization threshold and a METEOR score threshold. This is then interpreted as both a correctly localized and correctly described prediction.
The METEOR thresholds we use are 0, 0.05, 0.1, 0.15, 0.2, 0.25, so the total number of pairwise thresholds we have to compute the score over is 5 x 6 = 30 (5 for IoU overlap thresholds 6 for METEOR thresholds). Note that the 0 threshold isn't a mistake or a "catchall" because we want the score to be strictly greater.
The most important entry point to evaluation is inside eval_utils.lua, and it's the DenseCaptioningEvaluator class. You'll notice that it has an addResult() function, and most of the heavy lifting of the evaluation is in the evaluate() function.
One important and perhaps non-trivial point to mention is that we use merge_boxes() to merge ground truth boxes in the VG data that heavily overlap (>=0.7 IoU) into a single ground truth box with multiple reference captions.
Unfortunately the METEOR code is in Java and it's not easy to communicate with this process directly from Lua. Therefore, we adapted the Python wrapper written by Xinlei Chen from the MSCOCO challenge that communicates with the Java process using subprocess, and the Lua code communicates with this Python script (meteor_bridge.py) through json files. As a result, this ends up being unfortunately quite hairy at the moment.
- Java. METEOR requires Java installed. On Ubuntu you can do
sudo apt-get install default-jre. You can dojava -versionto confirm that you have it installed. - You'll need to download the METEOR binary
meteor-1.5.jarand place it inevalfolder and alsoparaphrase-en.gzand place it it ineval/datafolder. You can do this automatically by running the scriptscripts/setup_eval.shfrom the project's root directory.