Dense video caption is a task that aims to help computers analyze the content of a video by generating abstract captions for a sequence of video frames. However, most of the existing methods only use visual features in the video and ignore the audio features that are also essential for understanding the video. In this paper, we propose a fusion model that combines the Transformer framework to integrate both visual and audio features in the video for captioning. We use multi-head attention to deal with the variations in sequence lengths between the models involved in our approach. We also introduce a Common Pool to store the generated features and align them with the time steps, thus filtering the information and eliminating redundancy based on the confidence scores. Moreover, we use LSTM as a decoder to generate the description sentences, which reduces the memory size of the entire network. Experiments show that our method is competitive on the ActivityNet Captions dataset.
- dense video caption
- feature extraction
- multi-modal feature fusion
- neural network
- video captioning
FingerprintDive into the research topics of 'Fusion of Multi-Modal Features to Enhance Dense Video Caption'. Together they form a unique fingerprint.
Reports from Faculty of Applied Sciences Highlight Recent Research in Sensor Research (Fusion of Multi-Modal Features to Enhance Dense Video Caption)
1 item of Media coverage