Home » Posts tagged 'bias'
Tag Archives: bias
The “Friends and Family Test” – lessons from the “Do you masturbate often?” question
In whatever way you wish to ask the question from an expert statistician, a reasonable statistician will spit bullets at the methodology of “The Friends and Family Test”. Like Trip Advisor, it is susceptible to a phenomenon called ‘shilling’, where fake respondents bias the sample with their fake appraisals. The Friends and Family Test (FFT) is a single question survey which asks patients whether they would recommend the NHS service they have received to friends and family who need similar treatment or care. Conceptually, it is of course a terrific idea to ask the patient what they think of the NHS, but it is susceptible to too many uncontrolled variables for the result to be particularly meaningful; for example, how long after the clinical event should you ask the patient to ‘rate’ the episode? The responses to the FFT question are used to produce a score that can be aggregated to ward, site, specialty and trust level. The scores can also be aggregated to national level.
Most people in the public are very slightly interested in the geeky way in which statisticians produce the results, The scores are calculated by analysing responses and categorising them into promoters, detractors and neutral responses. The proportion of responses that are promoters and the proportion that are detractors are calculated and the proportion of detractors is then subtracted from the proportion of promoters to provide an overall ‘net promoter’ score. NHS England has not prescribed a specific method of collection and decisions on how to collect data have been taken locally. Each trust has been able to choose a data collection method that works best for its staff and people who use services. The guidance suggests a range of methods that can be adopted including tablet devices, paper based questionnaires and sms/text messages, amongst others. How you collect the data adds a further level of complexity to the meaningless nature of these data.
There is a phenomenon in statistics called the “social desirability responding” (“SDR”), a tendency of respondents to answer questions in a manner that will be viewed favorably by others. It can take the form of over-reporting “good behaviour” or under-reporting “bad,” or undesirable behaviour. The tendency poses a serious problem with conducting research with self-reports, especially questionnaires. This bias interferes with the interpretation of average tendencies as well as individual differences. There might be an age-related effect; older patients have tended to hold the NHS with greater reverence, whatever their political loyalties might be, compared to younger patients who believe in a market and/or believe they are ‘entitled’ to the NHS.
Topics where socially desirable responding (SDR) is of special concern are self-reports of abilities, personality, sexual behaviour, and drug use. When confronted with the question “How often do you masturbate?”, for example, respondents may be pressured by the societal taboo against masturbation, and either under-report the frequency or avoid answering the question. Therefore the mean rates of masturbation derived from self-report surveys are likely to be severe underestimates. Social desirability bias tends to be highest for telephone surveys and lowest for web surveys. This makes web surveys particularly well suited for studies of sexual behaviour, illicit activities, bigotry, or particularly threatening topics. Fundamentally, respondents do not answer questions the same way in person, on the phone, on paper or via the web. Different survey modes produce different results. Robert Groves, in his 1989 book “Survey Errors and Survey Costs“, argues that each survey mode puts respondent into a different frame of mind (a mental “script”). Face-to-face surveys prompt a “guest” script. Respondents are more likely to treat face-to-face interviewers graciously and hospitably, leading them to be more agreeable. Phone interviews prompt a “solicitor” script. Respondents are more likely to treat phone interviews the way they treat calls from telemarketers, making them more likely to go through the motions of answering questions in order to get the interviewer off the phone.
That is why reasonable statisticians take care when comparing the results of surveys conducted by different modes. Humans process language differently when reading, when listening to someone over the phone, or when listening to someone in the same room (when visual cues and body language kick in). It is no surprise that these different modes lead to different behavior by respondents. The lack of a standardised methodology in the FFT means that there are likely to be, what are known as, mode effects. Mode effect is a term used to describe the phenomenon of different methods of administering a survey leading to differences in the data returned. For example, we may expect to see differences in responses at a population level when comparing paper based questionnaires to tablet devices. On a positive note, mode effects do not prevent trusts from comparing their own data over time periods when they have conducted the test in the same way, as any biases inherent in the individual approaches are constant over the period.
Much money has been pumped into the FFT, and one wonders whether the FFT would stop another Harold Shipman or Mid Staffs. Even Trusts with excellent ratings, such as Lewisham, are in the “firing line” for being shut down. So one really is left wondering what on earth is the point of flogging this dead policy horse?
George Osborne and the perception of apparent incompetence
There’s only one person who is genuinely funnier than David Mitchell. I think that person is Robert Jay QC, lead Counsel for the Leveson Inquiry. Jay can deliver gags with more precision than an exocet missile, for example referring to the bias of Jeremy Hunt as ‘equal and opposite’ to that of Vince Cable.
Jay has been clearly worrying about the issue of the difference between apparent and actual bias in his line of questioning. The jurisprudence of this will indeed be very well known to all law students, including Porter v Magill and Pinochet [No 2]. On the whole, apparent bias, so believes (or so knows) Jay, is harder to prove than actual bias.
So by what standard should we judge George Osborne? Should Osborne be judged by the standards of a reasonable Minister, or one who has a specialist expertise in economics and finance? Arguably he should not be judged against the latter as he has a II.1 in the Final Honour School of Modern History from University of Oxford.
Victoria Coren, who is better known as Victoria Coren than Mrs David Mitchell, was forced to respond to a question last night on BBC Question Time (available here) as to whether Osborne was incompetent. Coren conceded that this was a difficult question, but explained that – if it came to looking after her pets or pot-plant when she went on holiday – the answer was no.
Labour has a massive problem in the perception of its economic competence. That is despite the overriding consensus of available data proving that Osborne’s economic experiment has wholeheartedly failed. Osborne managed to reverse a growing economy into a failing economy, way before any Eurozone crisis, with falling tax receipts, an overall decrease in full-time employment, greater spend on welfare benefit, and of course negative growth. That is actual incompetence.
It is going to be easier to prove the apparent incompetence of George Osborne. Members of Labour will remember clearly how he used to treat Alistair Darling with contempt, but there is now a never-ending stream of debacles for Osborne to admit. You can choose between the caravan tax, the pasty tax, or the charity tax, in addition to an imploding economy – it doesn’t matter. Whether you throw into the mixer his collusion with Jeremy Hunt in dealing with the BSkyB, as evidenced in Leveson yesterday, quite frankly neither here or there.
I have never bought into the agenda of Osborne as a great strategist. In fact, I think he is also abysmal at economics. He epitomises for me what the wonderful John Maynard Keynes, from King’ s College Cambridge, described as the type of person who does not wish to invest in the future, for various reasons:
(vii) To bequeath a fortune.
(viii) To satisfy pure miserliness, i.e. unreasonable but insistent inhibitions against acts or expenditure as such
“The general theory of employment, interest and money” John Maynard Keynes
It’s no accident that the UK tax system treats capital and income separately. In an ideologically-driven austerity driven agenda ‘to pay off the credit card debt’, the deficit doubled in February 2012, as the nation’s income fell, as a result of its disgraceful treatment of human capital, in what promises to be one of the most spectacular failings of the UK government. The return on such high risk strategy means that Osborne will not be able to pay any electoral dividends in 2015 as he will have no distributable profits by then; in fact, the Coalition deserves to go into receivership.
[This post does not represent the views of BPP, or of the BPP Legal Awareness Society, a society to promote law in the business strategy.]
Time for the BBC to give up on the pretence of responsible journalism
Today, I loved reading the Times on my iPad. Indeed, parts of the British media are world-class, and worthy of our reputation abroad. The Times and Financial Times are probably my most favourite media publications of all.
Unfortunately, in the run-up to the General Election, the BBC were without shadow of a doubt gunning for Gordon Brown – to lose. Many of my friends were appalled about the highly personal comments made towards him in both style and manner, and this includes so-called respectable people in respectable institutions (for example, Nick Clegg’s conduct in the Lower House in Prime Minister’s Questions). For the BBC and people like Adam Boulton, ‘Bigotgate’ was possibly a gift.
Some have said that senior presenters of the BBC, Laura Kuenssberg and Nick Robinson, put the most unbelievable gloss on the Tory Party, that a large number of my 2400 friends on Facebook were talking about not renewing their TV licence as class action protest. Maybe, taken as a whole, the BBC does not suffer from lack of impartiality, and indeed some of the output of the BBC is first-rate (for example, the Today programme). Some items on BBC online news would be more fitting for a tabloid on a bad day.
Right-wingers tend to claim the BBC has enormous left-wing bias, therefore providing evidence that it produces balanced coverage. My parents, who have lived in this country since 1961, used to have enormous respect for the BBC, and indeed the brand of the BBC used to be superb internationally, but now that they have zero respect for it. Whilst there used to be goodwill for ‘Beeb’, the illusion has nearly become shattered to an irreparable state. Now that its standards have declined so much, it is vital that an external entity should look at the functioning of the BBC as a professional media operation. The BBC investigates complaints internally mainly, leaving little recourse for complaints, because OFCOM’s terms-of-reference are so narrow.
The journalists are supposed to obey the Editorial guidelines of the BBC which are widely publicized, but within a single day it is ‘dead easy’ to find examples of problems in accuracy, balance and impartiality. However, one has to wonder whether journalists should declare a ‘conflict of interest’ in the same way that directors of companies in England have to declare a financial interest under the Companies Act (2006)? Does it matter that a highly influential person within the BBC News machine, Nick Robinson, was a prominent Tory at University? His argument will be that his professional manner can be divorced from his political views, in that a doctor with severe depression can be a psychiatrist, but might it be worth the while of the BBC to publish once-and-for-all some statistics on the volume of complaints for a definable and measurable period, such as the 2010 General Election? Throughout the election campaign, the coverage towards David Cameron and Nick Clegg was much more lenient than towards Gordon Brown.
The BBC has for some time been producing inaccurate coverage of news stories, some of which are clearly not in the overall public interest but constitute a ‘witch-hunt’ at best. The BBC regularly contravenes rules of responsible journalism as explained in Reynolds v Times Newspaper case from the House of Lords. The recent debacle has been that Question Time has been accused of demonstrating left-wing bias, when David Dimbleby was virtually shouting down answers given by Hillary Benn. Even when it comes to defamation, it is not a problem as they have a well-funded legal team, paid for by millions of tax-payers. Protecting the identity of ‘Stig’ in the public interest did not come particularly cheap, ‘reliable sources claim’.
Apparently, a Conservative source said:
Now, more than ever, is the time for the BBC to be careful and frame the debate responsibly so that the facts are properly heard. The spending review is a serious topic for all of us, it needs to be treated as such.’
Surely 150 days is a bit early for right-wing political paranoia to start setting in?
Today, we have a main news item concerning Wikileaks suggesting that all we see in the media may not be what is happening in real life.
How transparent is the BBC machinery? Sure, they can publish the salaries of Directors who are earning £500,000 a year, or more, but is this what is really ‘getting the goat’ of ordinary licence payers? Was it correct that the BBC refused to play the DEC humanitarian appeal? The Glasgow Media Group repeatedly has shown the BBC is more right wing in coverage; a genuine public interest point is that, with the BBC attacking pensions of BBC workers and now to make 16% cuts, we can expect even more right wing bias.
Take specifically what happened last Wednesday. An individual has written to me the following:
“My part of my union (Revenue & Customs, PCS) had a small demonstration outside our HQ @ 100 Parliament Street (opposite the HOP). I was offered a spot on the Radio 5 Question Time being held on the Green after the cuts were made. There was some confusion and I was advised that the BBC didn’t want any trade union representatives on air!!! However, a few of us hung around whilst the political heavyweights were being interviewed. No one from any UK news outlet paid us (or any other protesters) any notice at all.
However, my colleague was filmed by Al Jazeera – who seemed more interested in what the protesters were saying than the politicians. She also did a longish interview for a Danish TV station and an interview for the Portuguese press. I was intervied by Helsinki Sanomat in some depth. The European press were interested in the lack of action by the TUC. I was asked if I would rather be French. The day before we were followed by Japanese TV for a documentary there and today we were interviewed in London by the French TV.”
On the wars in Iraq and Afghanistan, it is mooted that BBC still broadcasts much more pro-war views, even when 76% think troops should be returned. The most sinister development in their editorial policy is that they appear parrot ‘we have got to cut the deficit’ views without even providing the evidence from the Nobel Laureate, Paul Krugman, and David Blanchflower CBE, that the cuts will be a disaster. The BBC then creates editorial imbalance by not presenting half of the argument, thus making the entire argument grossly inaccurate. It is then easy for the BBC’s Director General Mark Thompson to satisfy the Conservative PR machine to present the coalition’s cuts in a favourable light, and for George Osborne to claim that Labour has no alternative.
The spin that has been propagated on this is truly mortifying. No mention is made by the BBC that the Conservatives supported the Labour borrowing plan between 2001-2007, the UK had the lowest debt of G20 countries on entering the recession, the recession was truly worldwide (as they might be forced to admit when we go into a double-dip), and that the reason Labour does not wish to specific which would it cut first is (a) because Labour with the Fawcett Society think the budget contravenes the Equality Act (b) Labour does not agree with the macroeconomic policy in the first place. Labour has made it perfectly clear in the public record for a long time that it does not support the rate or depth of cuts. It is especially nauseating that the Coalition does not command any authority on narrowing the ‘tax gap’.
The BBC could do a lot for public confidence in its reputation by reporting on tax avoidance by millionaires, or reporting on the alternative funding of the public sector services, rather than what it seems to spend most of its time in: gutter, trashy witch-hunts to grab headlines, so-called “breaking news”.
The real reason that people appear to hate the cuts is actually – shock horror – because real people (not millionaires) hate the cuts. The Coalition will be hard pushed to find a city sympathetic to their cause – maybe Middlesborough was a bad choice, but I look forward to Question Time from the BBC, in my home city of Glasgow next Thursday.
It’s all getting a bit serious isn’t it?
Here’s a video of Adam Boulton ‘losing it’ with Alastair Campbell
and Nick Robinson potentially contravening the Criminal Damage Act (1971)
Your journalism is safe in their hands? I’m saying nothing..
Dr Shibley Rahman is a research physician and research lawyer by training.
Queen’s Scholar, BA (1st.), MA, MB, BChir, PhD, MRCP(UK), LLB(Hons.), FRSA
Director of Law and Medicine Limited
Member of the Fabian Society and Associate of the Institute of Directors
