Imperial College London
Portrait Picture

DrAnsVercammen

Faculty of Natural Sciences, Centre for Environmental Policy

Honorary Research Fellow
 
 
 
//

Contact

 

ans.vercammen15

 
 
//

Location

 

503Weeks BuildingSouth Kensington Campus

//

Summary

 

Publications

Citation

BibTex format

@article{Marcoci:2019:10.3389/fpsyg.2018.02634,
author = {Marcoci, A and Burgman, M and Kruger, A and Silver, E and McBride, M and Thorn, F and Fraser, H and Wintle, B and Fidler, F and Vercammen, A},
doi = {10.3389/fpsyg.2018.02634},
journal = {Frontiers in Psychology},
title = {Better together: reliable application of the post-9/11 and post-Iraq US intelligence tradecraft standards requires collective analysis},
url = {http://dx.doi.org/10.3389/fpsyg.2018.02634},
volume = {9},
year = {2019}
}
Download

RIS format (EndNote, RefMan)

TY  - JOUR
AB  - Background: The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq’s Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the United States Intelligence Community. As part of the reform, analytic tradecraft standards were revised and codified into a policy document – Intelligence Community Directive (ICD) 203 – and an analytic ombudsman was appointed in the newly created Office for the Director of National Intelligence to ensure compliance across the intelligence community. In this paper we investigate the untested assumption that the ICD203 criteria can facilitate reliable evaluations of analytic products.Methods: Fifteen independent raters used a rubric based on the ICD203 criteria to assess the quality of reasoning of 64 analytical reports generated in response to hypothetical intelligence problems. We calculated the intra-class correlation coefficients for single and group-aggregated assessments.Results: Despite general training and rater calibration, the reliability of individual assessments was poor. However, aggregate ratings showed good to excellent reliability.Conclusion: Given that real problems will be more difficult and complex than our hypothetical case studies, we advise that groups of at least three raters are required to obtain reliable quality control procedures for intelligence products. Our study sets limits on assessment reliability and provides a basis for further evaluation of the predictive validity of intelligence reports generated in compliance with the tradecraft standards.
AU  - Marcoci,A
AU  - Burgman,M
AU  - Kruger,A
AU  - Silver,E
AU  - McBride,M
AU  - Thorn,F
AU  - Fraser,H
AU  - Wintle,B
AU  - Fidler,F
AU  - Vercammen,A
DO  - 10.3389/fpsyg.2018.02634
PY  - 2019///
SN  - 1664-1078
TI  - Better together: reliable application of the post-9/11 and post-Iraq US intelligence tradecraft standards requires collective analysis
T2  - Frontiers in Psychology
UR  - http://dx.doi.org/10.3389/fpsyg.2018.02634
UR  - http://hdl.handle.net/10044/1/66719
VL  - 9
ER  -
Download