Home » Mortality
The war over Jarman’s data
Prof Brian Jarman
1. Introduction
The fiasco over the HSMR is now of totemic proportions.
In many ways, the battle over the Channel 4 headline story of Wednesday saying “for the first time” that hospital mortality rates in the UK were considerably higher than other jurisdictions has become more totemic than the notorious “War over Jennifer’s Ear“. To give him credit, Prof Brian Jarman (@Jarmann) is always very helpful on Twitter. There can be a temptation to play the man not the ball, but Jarman has a very distinguished professional record as a member of the medical profession as part of the St. Mary’s Hospital/Imperial College centres in London. By his own admission, his first degree was in physics, and his PhD on Fourier analysis of seismic wave propagation. He changed to medicine age 31. There is absolutely no doubt that Prof Brian Jarman is an incredibly bright man. On the basis of presumption of innocence under English law, it is critical to assume that Prof Jarman did not intend to mislead the general public through the reporting of his claims over hospital mortality statistics. This is indeed consistent with the positioning of excellent patient safety campaigning groups who overall feel that a discussion over the statistics is a genuine offense to the misery of relatives (and families) who have suffered. There is perhaps an issue of whether Jarman was particularly reckless in how such data might be presented before a fearful public, and possibly how Jarman’s data are presented in the media, following the Telegraph and similar reports and this week’s Channel 4 presentation, in the future might be the best surrogate guide to Jarman’s true intentions.
The Channel 4 website report, “NHS hospital death rates among worst, new study finds“, is here.
“NHS chief Sir Bruce Keogh says he is taking very seriously figures revealed by Channel 4 News which show that health service patients are 45 per cent more likely to die in hospital than in the US.”
According to this Channel 4 website report:
“The figures prompted Sir Bruce Keogh, medical director of the NHS, to say he will hold top-level discussions in a bid to tackle the problems. I want our NHS to be based on evidence. I don’t want to disregard stuff that might be inconvenient or embarrassing…I want to use this kind of data to help inform how we can improve our NHS,” he told Channel 4 News.I will be the first to bring this data to the attention of clinical leaders in this country to see how we can tackle this problem.”
2. The methodology of the “hospital standardised mortality ratios” (HSMRs)
The methodology of the ‘hospital standardised mortality ratios‘ is well known to Sir Bruce Keogh. The wider criticisms of the HSMR have been extensively discussed in the medical press, see for example “Using hospital mortality rates to judge hospital performance: a bad idea that just won’t go away” (in the BMJ, April 2010).
In the widely publicised “Review into the quality of care and treatment provided by 14 hospital trusts in England: overview report” published from earlier this year, it is stated:
Jarman’s ‘discovery’, in a flourish worthy of ‘Gone with the Wind’ perhaps, is described thus:
“What he found so shocked him, he did not release the results. Instead, he searched – in vain – for a flaw in his methodology and he asked other academics to see if they could find where he might have gone wrong. They, too, could not find fault.”
The question remains: what happened in the intervening nine years which prevented any activity culminating in professional peer review?
3. These data were indeed presented as far back as 2004
The Channel 4 reporting was sensational as illustrated in this excerpt from the Channel 4 report by Victoria Macdonald:
“So now he is releasing the findings. And they are shocking. The 2004 figures show that NHS had the worst figures of all seven countries. Once the death rate was adjusted, England was 22 per cent higher than the average of all seven countries and it was 58 per cent higher than the best country.”
The data are even known to the Department of Health as evidenced by Jarman himself here:
A tweet shared by Brian Jarman provides a link to the presentation of the data in 2004.
The starting point of the report was to go to the Mayo Clinic:
“Because of confidentiality issues we are not allowed to name the other countries. But America stands out in the data for its lower mortality rates. So we went to find out why. At the Mayo Clinic Hospital in Phoenix, Arizona, they are in the best two per cent in the country. It is an impressive hospital, with piano music playing in the lobby and sunshine streaming into the rooms.”
4. Concerns about the data
Any (reasonable) reviewer of these data, particularly given the bold and significant nature of the claims therein, will be concerned about the ‘quality’ of the data, in much the same way Jarman is concerned about the ‘quality’ of healthcare. This is particularly so if one adopts the approach of ‘treating the data’ rather than ‘treating the clinical patient’, which many clinicians would not advocate anyway in isolation.
This ‘confidentiality issue’ about the manner in which these data were provided is confirmed in a recent tweet by Jarman:
In the absence of clarity of what these confidentiality issues precisely are, it is hard to deduce fully what it is exactly that is so prohibitive for Jarman publishing his data. Many senior academics indeed converge on the notion that the publication of these data would be a useful contribution to the field, provided that the publication were properly refereed from two perspectives. These perspectives are that the statistical techniques used are sound. The second perspective is from a clinician’s perspective that the correct public health policy issues have been identified, and analysed correctly using available global evidence, and the citation of relevant background assumptions and confounding factors. Whilst it is hard to conceive that Governments have been withholding publication of data on public policy grounds (and arguably there is no more important a policy issue than a country’s mortality), it is possible that individual private companies may not wish to disclose fully confidential data. In the private sector even, such lack of disclosure of confidential data has led to accusations of fraud and discussion of mitigation, because of the sensitivity of such data to the markets in the dividend signalling theory. Academics have suggested to Jarman on Twitter that it might be possible to publish these data using the techniques of anonymisation or pseudo-anonymisation, as is prevalent in contemporaneous scientific research, but again no answer has been readily provided.
5. The utility (and futility) of cross-jurisdictional comparisons
The financial situation of the Mayo Clinic itself is known to be strong, with a colossal amount of income per patient at the Mayo Clinic compared to a patient in the English National Health Service:
“But a copy of the clinic’s consolidated financial report obtained by the Pioneer Press shows total revenue of $8.48 billion in 2011. That was an increase of $533 million, or nearly 7 percent over revenue of $7.94 billion during the previous year. Income from operations increased by 18 percent, growing from $515.3 million in 2010 to $610.2 million last year, according to the financial report. “This is very strong,” said Steve Parente, a professor of finance at the University of Minnesota’s Carlson School of Management. “In terms of pure operations, they’re doing quite well….Their grants and contracts are going up, too,” said Parente, who reviewed the financial report.””
A pertinent issue is whether the hospital episode statistics are themselves reliable. Prof Jarman quoting other reports feels that they are reliable (see tweet). However, this is in contradistinction from other reports sourced at the Royal College of Physicians of London (as described in a previous Guardian article):
“Currently the public can use the NHS Choices website to help them choose a hospital for treatment. NHS Choices, and the information used by Dr Foster, is based on “Hospital Episode Statistics” (HES) data, which the NHS says is “authoritative and essential”. However, NHS insiders say the information, usually collected by administrative staff from patient records, is unreliable.Professor John Williams, director of the health information unit at the Royal College of Physicians, carried out a study into HES data and found a significant number of operations were recorded inaccurately. He has called for a change in the way data is collected, saying flaws in the HES database were exposed as long ago as 1982.”
Hospital death rates, particularly if followed over time, can give useful warning of problems, as Sir Bruce Keogh has stated. Dr Jacky Davis in a subsequent Channel 4 interview (see below) was asked about the ‘smoke detector’ use of HSMRs as being pivotal in warning about problems.
It is, for example, argued that the issues in Mid Staffs would not have been exposed, but for the HSMRs. This argument is relatively convincing, but it is also argued that any reader of the local papers in Mid Staffs would have been aware of the problem long before the official HSMR figures emerged. Dr Jacky Davis in her C4 interview on Thursday, in reply, admitted that this ‘smoke detector argument’ might be true, but explained further ‘if there’s smoke going up you have to make sure – is there a fire there?‘. This is intuitively true, as well as the notion that the mortality rate in any given hospital will depend on the numbers of people who are actually dying ‘on site’. In general of course variation of data in other jurisdictions are likely to relate to regional variation of the care of palliative care patients, for example, and also where the funding for medical care is coming from (for example, the location of death may be an artifact of the conditions of private insurance funding.)
However, arguably reliable comparison of such data internationally is fraught with difficulties, as Prof Walter Holland and colleagues found out when they published the European Atlas of Avoidable Deaths in 1988. Prof Holland is an eminent expert in public health and a visiting Chair at the London School of Economics. Since total mortality rates in the UK are similar, or lower, than in the United States, as a large proportion of the population in both countries die in a hospital such a difference seems unlikely, according to Holland. To compare hospital mortality rates between hospitals, whether in one or more countries, arguably it is necessary to take into account such factors as availability of discharge facilities, e.g. hospices, hospital admission criteria, length of time in hospital, and many other procedural and cultural factors, according to the eminent Holland.
Comparison of outcomes for individual conditions is even more difficult because of differences in diagnostic and coding procedures which have been illustrated many times. In a famous paper “Evidence of methodological bias in hospital standardised mortality ratios: retrospective database study of English hospitals” published in the BMJ in 1999, this coding problem was identified thus:
“Our findings suggest that the current Dr Foster Unit method is prone to bias and that any claims that variations in standardised mortality ratios for hospitals reflect differences in quality of care are less than credible.8 12 Indeed, our study may provide a partial explanation for understanding why the relation between case mix adjusted outcomes and quality of care has been questioned.24 Nevertheless, despite such evidence, assertions that variations in standardised mortality ratios reflect quality of care are widespread,25 resulting, unsurprisingly, in institutional stigma by creating enormous pressure on hospitals with high standardised mortality ratios and provoking regulators such as the Healthcare Commission to react.20“
In our jurisdiction, for example, the proportion of surgery done as elective day cases is reported to be quite high. Unfortunately the data presented by Jarman in his original 2004 presentation and the Wednesday Channel 4 package are inadequate alone for anyone to determine the validity of the conclusions or the methods used. This is a striking media story but unfortunately has not been subjected to proper scrutiny. This will be essential for policy in English health policy, particularly in the critical issue of patient safety, to progress; and progress it must not on the basis of emotion or soundbites.
6. A wider issue
Prof Jarman has consistently stated that he is in favour of a NHS funded by taxation, which is different from the attitude of libertarians and the Mayo Clinic, as shown here:
Critics of Jarman have felt that his criticisms of some foci of the delivery of care undermine the daily work of clinicians across the country, and may even undermine the doctor-patient relationship so pivotal for the work of clinicians in real life (see for example here).
Jarman and supporters feel, however, that for ages criticisms of poor quality of care in the NHS have been suppressed, and that there is a professional duty to speak up about such poor care to improve the NHS. However, critics of Jarman feel that such an approach ‘in the wrong hands’ is merely providing ammunition for further marketisation and privatisation of the NHS, and that Jarman is carrying out the ‘handy work’ for politicians to see this ideological goal come into fruition.
This debate is intense with accusation and counter-accusations, but the volume of criticism regarding the NHS article headlining Channel 4 news on Wednesday is a testament to how the public will not accept unqualified comparisons of the NHS with other jurisdictions either. The ensuing analysis has further put the spotlight on Jarman’s data, but this can be no bad thing either. The medical and nursing professions, and their regulatory bodies, will wish for all involved to maintain the reputation, trust in and confidence in their work, but this is only possible if the work is ‘fit for purpose’. Politicians can slag off the NHS to an extent to which clinicians would likely become disciplined for, and it is this contradiction which raises eyebrows.
7. Conclusion
Notwithstanding, it is clear that the fundamental mistake is that it was perceived by some that the Channel 4 report represented irresponsible journalism in that the assumptions were not accurately stated, it suffered from bias by omission by lack of explanation about the surrounding issues (not least presenting Keogh as somebody who tacitly endorsed the uncritical issue of the HSMR, which is false), and basically left a very ugly taste in many people’s mouths. It did no favours for the reputation of, trust in, and confidence in Channel 4 news reporting either that night (Wednesday), but in fairness to Channel 4 news they hosted an interview with @DrJackyDavis – Co-Chair of the NHS Consultants Association – the following evening (Thursday).
However, it is impossible to deny the value of the discussion which has ensued after this dreadful report. The policy situation is precarious, aggravated by certain Ministers of the Crown lying blatantly in the whole saga over a period of months in the current Government.
The Health and Social Care (2012) was not about patient safety: implications for the Keogh report
Julie Bailey
The Health and the Social Care Act (2012) is a massive tome. It actually reads, for lawyers who are well acquainted with such statutory instruments, like a huge patchwork quilt of commercial and corporate law strands. While voluminous, at 473 pages, it has two critical clauses. The first is section 75, and its concomitant now famous Regulations, which provides the statutory basis for procurement contracts in the NHS to be put up for price competitive tendering as the default option, thus fixing the NHS in a competitive market of an economic activity. This is of course the mechanism for outsourcing NHS services into the private sector, and indeed the vast majority of contracts have now been won by the private sector. This was widely predicted, as the private sector have skills and resources to make slicker bids, irrespective of the bid they ultimately deliver, to transfer a much higher proportion of “NHS services” into the profit-making private sector. All of this costs the NHS more money sadly, as while it may not matter to you ‘who provides your services’, you’re in trouble if the private provider goes bust, and you’re not paying for anything at anywhere near cost-price because of the mark-up for profit. This section 75 clause acts in tandem with section 164(1)(2A) which allows any NHS hospital to receive up to 50% of its income from private sources. Thus the Act, and the £2.4 NHS “reforms”, have been a bonanza for the private sector, and disastrous from the perspective of a state-provider of universal, comprehensive healthcare.
Patient safety is in fact only mentioned once in the Act, in clause 281. That is in reference to the abolition of the National Patient Safety Agency. The National Reporting and Learning System which was hosted by NPSA has a two year stint at Imperial College Hospital NHS Trust, while a tender process is scoped and developed by the Board. NPSA’s responsibilities concerning patient safety will transfer to the NHS England.
The Health and Social Care Act 2012 (c. 7) is an Act of the Parliament of the United Kingdom. It is the most extensive reorganisation of the structure of the National Health Service in England to date. It proposes to abolish NHS primary care trusts (PCTs) and Strategic Health Authorities (SHAs). The Act’s proposals were not discussed during the 2010 general election campaign and were not contained in the 20 May 2010 Conservative – Liberal Democrat coalition agreement, which declared an intention to “stop the top-down reorganisations of the NHS that have got in the way of patient care”. However, within two months a white paper outlined what the Daily Telegraph called the “biggest revolution in the NHS since its foundation”. The white paper, Equity and Excellence: Liberating the NHS, was followed in December 2010 by an implementation plan in the form of Liberating the NHS: legislative framework and next steps. The bill was introduced into the House of Commons on 19 January 2011, and received its second reading, a vote to approve the general principles of the Bill, by 321-235, a majority of 86, on 31 January 2011.
The British Medical Association opposed the bill, and held its first emergency meeting in 19 years, which asked the government to withdraw the bill and reconsider the reforms. A later motion of no confidence in Lansley by attendees at the Royal College of Nursing Conference in 2011, however, succeeded, with 96% voting in favour of the motion. Nurses have consistently been opposed to the the “efficiency savings” measures being undertaken across the NHS, with many raising concerns of their material impact on frontline medical services. “People will die”, Richard Horton, editor of The Lancet, warned in March 2012, as he predicted “unprecedented chaos” as a result of the reforms, with a leaked draft risk-assessment showing that emergencies would be less well managed and the increased use of the private sector would drive up costs.
The Bill is now Law, and where are the measures to deal with this longrunning problem of patient safety, particularly in the acute setting? There are none. The media was sent into overdrive in portraying the NHS has a “death machine”, despite the best attempts of nurses and Doctors to run the service under difficult conditions. The publication of the damning Keogh Report (“Report”), which spelt out the failings of 14 hospital trusts which did not quote “13,000 “needless deaths” since 2005″, is despite exhaustive pre-briefing to the media. The Report depicts a situation in certain trusts where patient safety is poor, with no reference to what action has been taken by the Government and their civil service to remedy this since the General Election in May 2010, which the Conservatives lost. Sir Bruce Keogh, the NHS’s Medical Director, will describe how each hospital let its patients down through poor care, medical errors and failures in management, but the Report is as if the clinical regulatory bodies do not exist, the General Medical Council, the Nursing and Midwifery Council and the Care Quality Commission. How they have escaped blame for this reported ‘scandal’ is incredible, although one suspects the media will catch up with them eventually. It might be that the media for whatever reason known to them do not feel the clinical regulators are in “the firing line”, despite being supposed to be responsible for patient safety, in the same way that lawyers are not responsible for the global financial crisis despite being supposed to regulated on the safety of financial instruments.
From a management point of view, the Keogh Report serves a function for convincing the public of a need to take patient safety extremely seriously. However, to sell the Keogh Report as “Do you now see the need for the NHS reforms?” maybe hitting a target but missing a crucial point. The NHS reforms are all to do with outsourcing and eventual privatisation of the NHS. They are nothing to do with patient safety, as even right-wing think tanks and their spokesmen have previously conceded in public. In fact, it is worse than that. The £2.4 reorganisation which nobody voted for, but which private healthcare companies extensively lobbied for, was a reckless missed opportunity to put resources into something other than frontline care, and the opportunity cost of this piece of legislation will continue to haunt the general public for many years. Unfortunately, the media and the members of the Establishment, some members of which have tenuous links with the institutional shareholders in private healthcare companies, will be more than aware of this hard fact. The Conservatives are desperate to pin every conceivable woe of the NHS on Andy Burnham, and every interview which Burnham now does must feel like “Groundhog Day” for him. He has nothing much more in his defense. Meanwhile, the Conservatives are exasperated that they have been unable to get the Burnham scalp, but there are as yet unresolved issues about what Government departments have done about NHS complaints in the last three years since May 2010. The bottom line is that the Health and Social Care Act is nothing to do with patient safety: even safety campaigners in the NHS know this, and they know of the even worse battle now facing them, of a fragmented privatised NHS which is even harder to regulate from that point of view. The NHS reforms, and more specifically the Health and Social Care Act which underpins them, have nothing to do with patient safety. More disturbingly, the Keogh report, when eventually published, will not stop ‘another Francis’, and it is entirely the Government’s fault we are in this stupid ridiculous position.
“Lessons learned” – If every unemployment statistic is a tragedy, what was every ‘excess death’ at Mid Staffs?
It never fails to amaze me how certain policy strands run in parallel along a disastrous course, but silos in journalism mean that you’ll never get people joining the dots.
One example of this is the competitive tendering in legal services which Chris Grayling MP is currently shoehorning through, despite overwhelming opposition from lawyers including QCs. Everytime the unemployment figures up, or we have another revival in youth employment, Chris Grayling used to be the guy on TV saying that ‘every statistic is of course a personal tragedy’. Curiously you never get this phrase said about any excess death from the NHS which happened out of the ordinary. The concept that it is impossible to measure excess deaths at all will be alien to any professional in clinical negligence, who will be able to follow through the well-worn logic of duty-of-care of a clinician, failure of that duty causing breach, and that breach causing damage provided that there is not remoteness. We all know that the media is prone to hysteria, and indeed John Prescott once advised me not to believe everything written about ‘one’ in the papers. And an issue undoubtedly is that some are using what happened at Mid Staffs for their own agendas. You’d be forgiven for thinking some reports have the sole intention of shutting down the entire NHS as a national health service, blow all its credibility to smithereens, and to prepare its purchase price for the lowest bidder in a Government which has relish in outsourcing and privatisating the State infrastructure.
However, the sensationalism which was embraced whether there were any ‘excess deaths’ or not is perhaps distasteful at best, and frankly rude at worst. Mortality ratios are supposed to be the ‘smoke alarm’, but now that the inferno has happened, it is not time to remove the batteries from the smoke detector. The public inquiries at Mid Staffs I feel were essential. I don’t feel that this is an issue which could have been discussed behind closed doors ‘in camera’. It might be feasible to hold no-one accountable as the ‘culture’ is so widespread, but that has not led professionals to escape liability ever before for fundamental breaches in care, such as poor note-keeping, unprompt investigations, poor conduct and communication, from the professional regulators. The frustration has been there appears to have been very little accountability, and this is significant whether one feels the role of the justice system should be fundamentally restorative, retributive or rehabilitative. A certain amount of hysteria has instead engulfed proceedings at Mid Staffs, with the recently reported hostile behaviour towards Julie Bailey, remarkable campaigner and founder of ‘Cure the NHS’.
However, Julie has never wanted to ‘Kill the NHS’, but is deeply hurt about what happened to her Mum. Deb Hazeldine is very hurt about what happened to her mum. Any reasonable daughter would. These are times for reports of personal tragedies. Whilst we all have to move on, it is important to acknowledge accurately the distress of what happened, and this is precisely what we achieved in the Francis Inquiries. The accounts in those Inquiries are not figments of anyone’s imagination. It is even possible that we may have to learn from what happened there for other NHS Trusts. There is a trail of logic which goes that ‘efficiency savings’ were in fact cuts which included relative staff shortages, despite more being spent on the NHS budget overall including for salaries for certain personnel; this meant that key critical frontline staff were overstretched, there were genuine clinical events in patient safety which went beyond ‘near misses’, but they were not adequately dealt with. The Francis Inquiries should not be used to draw closure on the matters for the Labour administration, which I broadly supported. The reaction to the situation, a real one of personal tragedy, should not in my view a retweet of a blog which says that standard mortality ratios are unreliable, however correct that blog might be. This for me is not in any way personal – I like and respect very much people on all sides of what has been a highly charged discussion. I have known some of them for ages, and I will continue to support them publicly and in private.
We are not at the end of the solution of what happened in Mid Staffs, and for the time-being we should honestly recognise that.
