Imperial College London
Portrait Picture

DR BERNHARD KAINZ

Faculty of Engineering, Department of Computing

Reader in Medical Image Computing
 
 
 
//

Contact

 

+44 (0)20 7594 8349b.kainz Website CV

 
 
//

Location

 

372Huxley BuildingSouth Kensington Campus

//

Summary

 

Publications

Citation

BibTex format

@article{Liu:2022:10.1109/TIP.2022.3143699,
author = {Liu, T and Meng, Q and Huang, J-J and Vlontzos, A and Rueckert, D and Kainz, B},
doi = {10.1109/TIP.2022.3143699},
journal = {IEEE Transactions on Image Processing},
pages = {1573--1586},
title = {Video summarization through reinforcement learning with a 3D spatio-temporal U-Net},
url = {http://dx.doi.org/10.1109/TIP.2022.3143699},
volume = {31},
year = {2022}
}
Download

RIS format (EndNote, RefMan)

TY  - JOUR
AB  - Intelligent video summarization algorithms allow to quickly convey the most relevant information in videos through the identification of the most essential and explanatory content while removing redundant video frames. In this paper, we introduce the 3DST-UNet-RL framework for video summarization. A 3D spatio-temporal U-Net is used to efficiently encode spatio-temporal information of the input videos for downstream reinforcement learning (RL). An RL agent learns from spatio-temporal latent scores and predicts actions for keeping or rejecting a video frame in a video summary. We investigate if real/inflated 3D spatio-temporal CNN features are better suited to learn representations from videos than commonly used 2D image features. Our framework can operate in both, a fully unsupervised mode and a supervised training mode. We analyse the impact of prescribed summary lengths and show experimental evidence for the effectiveness of 3DST-UNet-RL on two commonly used general video summarization benchmarks. We also applied our method on a medical video summarization task. The proposed video summarization method has the potential to save storage costs of ultrasound screening videos as well as to increase efficiency when browsing patient video data during retrospective analysis or audit without loosing essential information.
AU  - Liu,T
AU  - Meng,Q
AU  - Huang,J-J
AU  - Vlontzos,A
AU  - Rueckert,D
AU  - Kainz,B
DO  - 10.1109/TIP.2022.3143699
EP  - 1586
PY  - 2022///
SN  - 1057-7149
SP  - 1573
TI  - Video summarization through reinforcement learning with a 3D spatio-temporal U-Net
T2  - IEEE Transactions on Image Processing
UR  - http://dx.doi.org/10.1109/TIP.2022.3143699
UR  - http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcApp=PARTNER_APP&SrcAuth=LinksAMR&KeyUT=WOS:000750373700004&DestLinkType=FullRecord&DestApp=ALL_WOS&UsrCustomerID=1ba7043ffcc86c417c072aa74d649202
UR  - http://hdl.handle.net/10044/1/96851
VL  - 31
ER  -
Download