Dense Image Captioning

Dense captioning is a task that requires a computer vision system to both localize and describe salient regions in images in natural language. It generalizes object detection (when each description is a single word) and image captioning (when one predicted region covers the whole image). "DenseCap: Fully Convolutional Localization Networks for Dense Captioning" by Johnson et al. proposed a Fully Convolutional Localization Network (FCLN) that processes an image in a single efficient forward pass, needs no external region proposals, and trains end to end in one round of optimization. The architecture combines a convolutional network, a novel dense localization layer, and a recurrent neural network language model that generates the label sequences.

In this project we did two things. First, we reproduced the authors' results to get familiar with the codebase. Second, we replaced the test-time Non-Maximal Suppression step with a Tyrolean Network, as described in "A convnet for non-maximum suppression" by Hosang et al. This gave a slight increase in the Mean Average Precision of DenseCap compared to our run of the original code.