Home » Posts tagged 'BCAT'
Tag Archives: BCAT
The potential use of the BCAT to teach critical reasoning skills in law schools
Diversity on barrister courses and in the legal profession was a pervasive theme in the responses to the Bar Course Aptitude Test (BCAT) consultation completed earlier this year. The Bar Standards Board (BSB) remains committed to diversity at the Bar, and it has been noted, for example, that both high and low scorers on this proposed BCAT test exist in various ethnic groups.
The Legal Services Board (LSB) has previously identified that the proposed BCAT has only been piloted thus far in limited circumstances, and this absolute uncertainty about how the test will perform in practice throws doubt in how the LSB can perform its regulatory duties in the public interest, according to a statement by LSB Chief Executive Chris Kenny. On account of this uncertainty, the LSB has asked the BSB to commit to undertake a five-year data gathering, analysis and evaluation period. Following the conclusion of this, a formal review will take place with a decision to “continue, revise, suspend or cease the BCAT”.
The early version of the BCAT published yesterday contains sample questions unsurprisingly exactly the same as known example questions of the Watson-Glaser test. This test has large samples of normative data, including from law schools around the world such as Thurgood Marshall School of Law in Texas Southern University. This school of law, under the auspices of its own Centre for Legal Pedagogy chaired by Professor Dannye Holley, Professor Anthony Palasota and Professor Edieth Wu, has conducted a number of studies using the test on existing students at their law school.
In this regard, it is illuminating to consider how the Watson-Glaser Test came about in the first place, as a learning exercise rather than an assessment exercise. The instrument is used already by a handful of City firms in their applications procedure. The language in the Bar Standards Board BCAT consultation report is one of ‘cut-offs’, especially in regard to the diversity impact monitoring. However, Goodwin Watson (1899-1976) and Edward Glaser (1911-1993) in their seminal work on critical thinking, focused on the link between rational thought and the educational process.
Their view of “critical thinking” is therefore as a set of teachable skills underlying successful learning, identified by five aspects of rational thought when effectively solving a problem: the ability to define a problem, the ability to elect pertinent information for the solution of a problem, the ability to recognise stated and unstated assumptions, the ability to formulate and select relevant and promising hypotheses, and the ability to draw conclusions validly and to judge the validity of inferences.
Alternative forms of this test allow the Thurgood Marshall School of Law to measure the development of these critical thinking skills, either as a consequence of specific instruction or over a long period of time. The ability to think critically is considered to be a desirable outcome of legal courses anyway. It is thought that law students should have an ability to extract relevant facts, and to distinguish them from legal or factual ambiguities, for example. This could of course be validly developed through other routes, such as the Masters degree or the Ph.D. degree, perfectly rationally.
All of this tends to beg the question: why didn’t the Bar Standards Board wish to use the test in charting any improvement of skills in critical thinking in barristers-to-be? This exercise would have been more fruitful, and would have allowed a more holistic view of the educational development of a barrister in training. For example, the interest in the development of ‘emotional intelligence’ in the psychological, organisational and leadership fields is substantial to thus of us who know these fields well. Taking a view of regulation as representative of a culture of ongoing obligations throughout one’s career, which all professional regulatory bodies undoubtedly do, it would have made more sense to use the application of the Watson-Glaser Test to encourage social mobility, rather than to limit it potentially unfairly. The fact that you can be instructed to perform well at the Watson-Glaser Test may put some students at an unfair advantage, dependent on their schooling, for instance.
Whatever then the outcome of the five-year data collection exercise, I believe it would be most valuable to consider the test performance of students as they progress through their career, from the positive perspective that any glaring critical thinking shortfalls can be ameliorated. This is of course assuming that the BCAT gets implemented in the first place by the Bar Standards Board, and that students are not overloaded through it.
Will the new proposed Bar Course Aptitude Test be fair to aspiring barristers?
The completion of the training of barristers is a genuine regulatory concern of the Bar Standards Board (BSB). According to recent statistics, the number of students who want to become barristers shows little sign of diminishing with 3,100 applicants to the Bar Professional Training Course (BTPC) in 2010/2011 and 3,016 in 2011/2012. There is a growing level of concern at the Bar and amongst law students that the rising demand for the BPTC is not reflected by rising availability of pupillages, coupled with increasing fees for the Bar Professional Training Course (BPTC). BPP Law School, one of the main providers of the BPTC, recently announced an increase in fees by five per cent for September 2012 to £16,540. It is hoped that introduction of the Bar Course Aptitude Test (BCAT) will introduce fairness, by decreasing the number of law students who fail the BPTC, and the public consultation until 29 February 2012 is encouraged to see if this will be the case.
The BSB proposes that, in addition to existing entry requirements as specified in the Bar Training Regulations, all BPTC students should attain a minimum pass threshold on the BCAT, which has been carefully developed and piloted specifically for this purpose. It is proposed that this implementation of the BCAT by Pearson Vue should commence with the cohort of candidates applying from November 2012 to start the course in September 2013. Whatever is proposed by the Bar Standards Board, it will be for the Legal Services Board to determine whether the proposal may be implemented.
The BCAT is based on the established and recognised Watson Glaser Critical Thinking Test which is used by some law firms in recruitment assessment days and by the Graduate Management Admissions Council. The Wood Review of the BPTC’s predecessor, the BVC, had been commissioned by the BSB with a Working Group chaired by Derek Wood QC, and published in July 2008. Learners will now be able to take the BCAT at any stage of their education or career, including after applying for the BPTC. The requirement will be that applicants must have scored the threshold pass before enrolling on the course, similar to the current English language rule and other entry requirements. It is staggering that, despite the fact that no socioeconomic data in the pilot were collected, the Bar Standards Board find that the test will not any effect, adverse or otherwise, on socioeconomic factors of enrolment on the BPTC.
Students in the UK are already prone to be most over-assessed in Europe, and it is an embarrassment to our educational system that academic competences will not have been identified by other means by the time a learner sits the test. The cost of taking the test will be a consideration for some learners at a time when some may in future be coping with university course fees. Unfortunately there is a slightly higher cost of taking the test for international students due to higher cost to Pearson Vue of testing and processing results overseas. The BSB has apparently explored the chance of a reduction with Pearson Vue but it has been confirmed that this cost is non-movable.
Certainly the equality and diversity impact assessment will have to be evaluated critically. For example, while no significant differences were found for age, or disability, statistically significant but small differences were found for gender and primary language. Indeed, Pearson Vue intend to mitigate against the effects of disability, indeed as they are obliged to under current equality legislation, through “reasonable adjustments” at test centres. The BSB will need to ensure that these are enforced rigorously, as anecdotal reports on the success of implementation by legal recruiters have been unimpressive for training contract applications.
A legal secretary commented on Twitter today that diversity would be ensured according to ‘how easy is it to pay for coaching to pass’. Formal education is currently expensive, and it is unlikely that educational providers might draw attention to the finding in the report that ‘coaching can have a small effect’, and indeed It is the view of the Bar Standards Board that it is more important to ensure fairness by allowing an unlimited number of re-sits, as the risk of applicants being coached sufficiently to achieve a pass is limited.
Views of current BPTC towards the proposed BCAT are mixed, but few current students are enthusiastic about what it will achieve. One current BPTC student ‘tweeted’, ‘I did LNAT and failed. According to that I shouldn’t have done law at uni, or anything further. So sceptical at aptitude tests.’
It is hugely impressive that the BSB have put produced such an excellent report into the development of the BCAT which is now open to consultation until 29 February 2012. It will be very interesting to see how the profession responds. The Law Society has already commenced investigations into a similar LPC aptitude test, and will undoubtedly follow the progress of the BSB BCAT with enormous interest.
Does the proposed Bar Course Aptitude Test examine the right skills?
On 24 February 2011, Catherine Baksi from the Law Gazette reported that the Chair of the Bar’s Regulator, Baroness Ruth Deech, had said that too many people on the Bar Professional Training Course (“BPTC”) are ‘wasting their money’ because they are ‘not up to it’.
Baroness Deech said the Bar Standards Board (BSB) would press ahead with its plans to introduce aptitude tests for students before they can undertake the BPTC.
This week they indeed proposed a new aptitude test, the Bar Course Aptitude Test (“BCAT”), and are welcoming feedback on their public consultation before 29 February 2012. This test is designed to measure an aptitude to complete the BPTC, though we are specifically not told whether this is the same for aptitude for a successful career at the Bar. The BSB perceives the main problem to be the current high failure rates of the BPTC.
Currently the admissions requirement for the BPTC at both the College of Law and BPP, as well as an arbitrary good standard of English, is a II.2, whereas the usual minimum requirement for a corporate training contract is a II.1. Currently, at Cambridge, approximately 80% of students achieve in Part II of the Tripos (Finals) a I, II.1 or II.2 in some subjects. The most parsiminious explanation therefore for a high failure rate on the BPTC is actually that its academic requirements are simply not high enough. However, many teachers feel that a simplistic judgement based on academic qualifications is not that fair in determining success in this profession.
Most curiously, the latest initiative from the Bar Standards Board seems to be in a ‘parallel universe’ to the excellent initiative, including eminent academics, teachers and lawyers, called the ‘Legal Education and Training Board‘. Part of the problem for the high failure rate of the BPTC, according to the BSB, may be students not succeeding in small, interactive sessions such as involving role-play.
It is now consistently acknowledged that people have different skills in general cognitive (thinking) intelligence and social/emotional intelligence. In the current management and leadership literature, as well as in experimental psychology, the role-plays found in the BPTC would in fact be considered to tapping domains of emotional intelligence, as framed latterly by Daniel Goleman, not traditional cognitive intelligence. The BCAT is based on the Watson-Glaser Test, and this test does not test emotional intelligence at all. For anybody to ignore knowingly the recent decades of intelligence in the last decades is rather perplexing. The BCAT does not test either how law students would react in well-validated ‘real life’ scenarios they might experience as a law student, or ultimately a trainee. This would be a ‘situational judgement test’.
The main potential fault with the solution proposed within the proposed BCAT is that there exist no published data about the reliability of the Watson-Glaser Test in predicting success on a BPTC course. To my knowledge, one magic circle firm is thought to use the Watson-Glaser Test in selection of future solicitors, and the most of the rest use the SHL online verbal reasoning test. It has previously been argued that psychometric testing should not a sole criterion for job suitability. There is no compulsory regulatory requirement of these psychometric tests, but the British Psychological Society has been calling for best practice standards in this field for a longtime along with their European counterparts. There are few published studies about the correlation between psychometric tests and performance in a any professional legal service environment, let alone formal studies comprising barristers or solicitors in training.
The BSB consultation provides a description of the features of the ideal psychometric aptitude test, and this indeed useful; it is a list of characteristics which few people in this field would disagree with. However, even if you are testing cognitive intelligence rather than emotional intelligence, there are many different cognitive skills you can choose. For example, it is possible to perform poorly at the SHL online verbal reasoning test due to cognitive deficits in learning, memory, attention, strategy and language, so therefore most reasonable cognitive neuropsychologists would find drawing conclusions from such tests difficult. However, these tests determine the future of intelligent students wishing with all the best will to enter the learned profession. Morale of my student colleagues is poor – managing strategic change like this new test will need the goodwill of the profession, including all its stakeholders such as students. The Bar Standards Board will then have to listen carefully to the expert views of academics, not just psychological test suppliers.
However, the Bar Standards Board have to be congratulated for producing and recommending such a comprehensive proposal for a practical aptitude test for the BPTC, the success of which is pivotal for regulating overall standards at the Bar from an early stage. The proposal, if students can afford it, seems eminently practical with an industry-respected test supplier. The views of the open public consultation are bound to be interesting.
