Imperial College London
Alternate

ProfessorMurrayShanahan

Faculty of Engineering, Department of Computing

Professor in Cognitive Robotics
 
 
 
//

Contact

 

+44 (0)20 7594 8262m.shanahan Website

 
 
//

Location

 

407BHuxley BuildingSouth Kensington Campus

//

Summary

 

Publications

Citation

BibTex format

@article{Tax:2017:10.3390/e19090474,
author = {Tax, T and Martinez, Mediano PA and Shanahan, M},
doi = {10.3390/e19090474},
journal = {Entropy},
title = {The Partial Information Decomposition of GenerativeNeural Network Models},
url = {http://dx.doi.org/10.3390/e19090474},
volume = {19},
year = {2017}
}
Download

RIS format (EndNote, RefMan)

TY  - JOUR
AB  - In this work we study the distributed representations learnt by generative neural network models. In particular, we investigate the properties of redundant and synergistic information that groups of hidden neurons contain about the target variable. To this end, we use an emerging branch of information theory called partial information decomposition (PID) and track the informational properties of the neurons through training. We find two differentiated phases during the training process: a first short phase in which the neurons learn redundant information about the target, and a second phase in which neurons start specialising and each of them learns unique information about the target. We also find that in smaller networks individual neurons learn more specific information about certain features of the input, suggesting that learning pressure can encourage disentangled representations.
AU  - Tax,T
AU  - Martinez,Mediano PA
AU  - Shanahan,M
DO  - 10.3390/e19090474
PY  - 2017///
SN  - 1099-4300
TI  - The Partial Information Decomposition of GenerativeNeural Network Models
T2  - Entropy
UR  - http://dx.doi.org/10.3390/e19090474
UR  - http://hdl.handle.net/10044/1/50586
VL  - 19
ER  -
Download