National hospital performance surveillance system
Start and end dates
2007 to present
Since 2007, the Dr Foster Unit has been notifying NHS trusts of possible high mortality outcomes for a range of diagnosis and procedures. Each month, we write to trusts with clear signals and share our findings with the Care Quality Commission. Partly as a result of our alerts, the CQC instituted an inquiry which resulted in a damning report on the delivery of emergency care at Mid Staffordshire NHS Trust.
The origins of the monthly mortality alerting system lie in analyses commissioned by the Bristol Royal Infirmary Inquiry in 1999 examining paediatric cardiac surgical outcomes at the Bristol Royal Infirmary. Our group confirmed serious concerns around the surgical outcomes at Bristol and established the usefulness of routine administrative data (Hospital Episode Statistics) in helping to identify quality of care issues. In further research commissioned by the Shipman Inquiry and published in 2003, our group established the role that statistical process control charts (specifically log likelihood CUSUM charts), and other routinely collected data (from death certificates) could play in the continuous surveillance of healthcare outcomes, and in this specific case, the detection of unusual patterns of patient mortality within General Practices. Since then we have demonstrated that coverage and completeness of routinely collected administrative data is comparable (or better) compared with clinical audit data. We have used mortality rates (because these are well recorded) as a potential reflection of quality of care, with the inference that possibly more less serious complications underlie them. We have subsequently published other indicators on safety stroke care and returns to theatre based on administrative data.
In 2014 we received NIHR funding for a two-year project 'National Surveillance System for Mortality Alerts' to improve the understanding of mortality alerts and evaluate the impact of the alerts as an intervention.
Dr Foster Unit Imperial College Mid Staffordshire Inquiry
How it works
Our work on appropriate statistical techniques in mortality surveillance in general practice, established the utility of statistical process control charts in surveillance of healthcare performance. This led to the application of these methods to secondary care administrative data in the development of a surveillance system to detect high mortality rates, emergency readmission and long length of stay in 256 diagnosis groups and 120 surgical procedure groups in collaboration with a commercial company, Dr Foster. The system is now used in 70% of acute hospital trusts in England.
Following the development of the surveillance system, it became apparent that we were identifying a number of alerts each month, some of which were generated in trusts that did not have access to Dr Foster analytical tools. In 2007, we created a system, whereby each month, we notified trusts by letter (regardless of whether they were Dr Foster customers or not) if they had alerted for a subset of diagnoses and conditions at a high alarm threshold. In the letter, we are explicit in the range of possible reasons for a signal, including random variation, poor data quality or coding problems and case-mix issues, and we emphasise that we draw no conclusions as to what lies behind the figures. We worked closely with the Healthcare Commission (which was to be taken over by the Care Quality Commission (CQC)) in setting up this Mortality Alerting and Surveillance System (MASS), and agreed to notify them about the letters we sent. The CQC subsequently set up their own alerting system, based on similar methods, but their data were originally less current and based around Healthcare Resource Groups (HRGs). Within the first 8 months of running our monthly alerts, one hospital had a series of 6 mortality alerts in a range of conditions and procedures. This helped trigger an inspection of care at Stafford Hospital, which was within Mid Staffordshire NHS Foundation Trust. The initial inspection team found “appalling” failures of emergency care within the trust, and a number of inquiries, culminating in the public inquiry led by Robert Francis, confirmed a litany of failings. The Inquiry report recognised the role that our surveillance system of mortality alerts had to play in identifying Mid Staffs as an outlier. Key recommendations reflecting our unit’s work, were that all healthcare provider organisations should develop and maintain systems which give effective real-time information on the performance of each of their services, specialist teams and consultants in relation to mortality, patient safety and minimum quality standard.
Mid Staffordshire Public Inquiry
The Dr Foster Unit's work in patient safety was noted in the Mid Staffordshire Public Inquiry:
- 5.237 "There is no doubt that, without the work of the Dr Foster Unit and Dr Foster Intelligence, comparative mortality statistics would not have been published as quickly, or as fully, as they now are.”
- 5.192 “There is now a consensus that significantly high HSMR/SHMI results should trigger a serious consideration of whether poor care is an explanation for them, whatever other steps are also taken. In any event, the explanation of poor coding offers little reassurance that there is no matter of concern.”
Learn more about the Mid Staffordshire Public Inquiry.
- Bottle A, Aylin P. Predicting the false alarm rate in multi-institution mortality monitoring. J Oper Res Soc 2011;62(9):1711-1718. SpringerLink
- Gandy A, Kvaloy JT, Bottle A, Zhou F. Risk-adjusted monitoring of time to event. Biometrika 2010; 97(2): 375-388. PubMed
- Bottle A, Aylin P. Intelligent information: A national system for monitoring clinical performance. Health Serv Res 2008; 43: 10-31. PubMed