Imperial College London
Alternate

ProfessorChristos-SavvasBouganis

Faculty of Engineering, Department of Electrical and Electronic Engineering

Professor of Intelligent Digital Systems
 
 
 
//

Contact

 

+44 (0)20 7594 6144christos-savvas.bouganis Website

 
 
//

Location

 

904Electrical EngineeringSouth Kensington Campus

//

Summary

 

Publications

Citation

BibTex format

@unpublished{Rajagopal:2020,
author = {Rajagopal, A and Vink, DA and Venieris, SI and Bouganis, C-S},
publisher = {arXiv},
title = {Multi-Precision Policy Enforced Training (MuPPET): A precision-switching  strategy for quantised fixed-point training of CNNs},
url = {http://arxiv.org/abs/2006.09049v1},
year = {2020}
}
Download

RIS format (EndNote, RefMan)

TY  - UNPB
AB  - Large-scale convolutional neural networks (CNNs) suffer from very longtraining times, spanning from hours to weeks, limiting the productivity andexperimentation of deep learning practitioners. As networks grow in size andcomplexity, training time can be reduced through low-precision datarepresentations and computations. However, in doing so the final accuracysuffers due to the problem of vanishing gradients. Existing state-of-the-artmethods combat this issue by means of a mixed-precision approach utilising twodifferent precision levels, FP32 (32-bit floating-point) and FP16/FP8(16-/8-bit floating-point), leveraging the hardware support of recent GPUarchitectures for FP16 operations to obtain performance gains. This work pushesthe boundary of quantised training by employing a multilevel optimisationapproach that utilises multiple precisions including low-precision fixed-pointrepresentations. The novel training strategy, MuPPET, combines the use ofmultiple number representation regimes together with a precision-switchingmechanism that decides at run time the transition point between precisionregimes. Overall, the proposed strategy tailors the training process to thehardware-level capabilities of the target hardware architecture and yieldsimprovements in training time and energy efficiency compared tostate-of-the-art approaches. Applying MuPPET on the training of AlexNet,ResNet18 and GoogLeNet on ImageNet (ILSVRC12) and targeting an NVIDIA TuringGPU, MuPPET achieves the same accuracy as standard full-precision training withtraining-time speedup of up to 1.84$\times$ and an average speedup of1.58$\times$ across the networks.
AU  - Rajagopal,A
AU  - Vink,DA
AU  - Venieris,SI
AU  - Bouganis,C-S
PB  - arXiv
PY  - 2020///
TI  - Multi-Precision Policy Enforced Training (MuPPET): A precision-switching  strategy for quantised fixed-point training of CNNs
UR  - http://arxiv.org/abs/2006.09049v1
UR  - http://hdl.handle.net/10044/1/80080
ER  -
Download