David Free believes that Literary Panels are a Terrible Idea
By Craig Lawn
In response to David Free’s article about literary awards in the SMH 9/7/21
As a commentator, David’s role is to engage the reader. He has done this by writing a polemic piece. He has been one-eyed and controversial, but he has succeeded – it stirred me up enough to bother to research and write a response.
His challenge resonated with me. I am married to a literary judge and critic, who over just the last five years has chaired or been on the panel of about 20 major literary awards (including repeat years on the same panel). I trust and respect her judgement and she has deep personal experience. Over that time I have observed the professionalism and commitment that she and her fellow panellists have devoted to the process.
Is it relevant and worthwhile for us to ask – are literary panels the best way to choose, select and celebrate the best books? A huge amount of effort is thrown at literary panels by very experienced people – is there a better way?
David has raised some good challenges that, through debate, may lead us to a better process. Unfortunately for David, he confesses that he poses his issues from a partially informed view. He has never had real experience of a panel. He was invited to be on a major literary panel but chose not to be involved. If he had been more courageous and taken on the role he may have had more valuable insights and credibility. It is always difficult being a credible commentator if you have “never played the game”.
From my research I have found little academic research on the veracity of his claims – are the benefits from literary panels for the authors, readers and society superior to other approaches? I have therefore summarised some personal comments on David’s challenges and used insights from my wife’s extensive experience of judging both adult and children’s books awards.
A précis of his major arguments that literary panels are fundamentally flawed is discussed below.
DF: There is a huge volume of books and it is barely feasible for the judges to read these over about two months, being the period that the panels typically operate – he estimates reading four books every day for two months. (His maths broadly equates to 160 books to be read if you only work Monday to Friday.)
Craig: The approach of using literary experience to be able to spot the weaker books after 20 pages is a flawed process. He implies that it does not do justice to the entrants and may result in the best books not being selected.
Joy: The workload is significant and typically requires reviewing 40 to 140 books, narrowed down to a long list around 10-12, and then to a shortlist around 5 or 6, before a winner. However, this does not mean that the workload is unrealistic or results in a flawed process. Digesting the volume of books is feasible and has been achieved time and time again. Often the books are initially allocated between pairs of judges to share the workload before books of likely potential are judged by all the panel. Also, the judges are chosen because of their relevant experience and realistically and reasonably can use their judgement to cull books without reading every word on a first read. If a book has not captured you by 20- 50 pages then surely it should not pass the grade of a great book worthy of the short list.
DK: The pay is minimal.
Craig: Apart from concluding that he would choose higher paying roles he does not indicate why this would result in the literary panels being flawed – perhaps if it impacts on the quality of the reviewers or their commitment?
Joy: Yes, the pay is normally poor, if not insulting or non-existent, but this is quite typical for the book world. There is also prestige and an opportunity to make a real difference through championing the very best books and giving them the best chance through financial rewards to the author and publisher and opportunities to publicise the book. It is a privilege to be involved and I see it is part of a vocation as much as a role. No panel or process is perfect but it is my experience that there is a consistently high quality of judges doing quality jobs.
DK: The panels result in random and subjective results dependant on the mix of the individual judges’ bias.
DK: Humans conform when in a group resulting in predictable and obedient winners.
Joy: This is categorically not the case. The best quality books are consistently selected because of the rigorous panel review process. This requires every book to be read initially by at least two panel members, and a series of three to four panel debates (as well as ongoing feedback or status reports) to winnow the books down to, say, 40 to12 to 5 to one. There is often a formal scoring matrix adopted with a secretarial role to capture the scoring and ensure due process.
This process is not random, and in my experience results in the best quality books being selected. However, clearly the process relies on the opinion of the group of judges on the panel assessing very subjective elements of the literary merit and other qualities of a book. I would estimate that 95% of the time I would personally agree with the panel’s decision of best book, and in only 5% of cases I personally think that they may have made the wrong choice, but I trust the process and the experience and opinions of the other judges.
DK: The conclusion that panels make better judgements than individuals is a furphy.
Craig: This is easy for David to say, but his conclusion is completely without facts or reasoning. I could not find specific academic research on literary panels. However, common sense and my experience demonstrates that any well run and chaired team will result in a superior outcome compared to the best that an individual can produce. This is supported by academic studies, for example, Synergetic effects in working teams by Guido Hertel University of Munster, 2011. This paper outlines the systematic research on performance gains in teams (often called “synergetic effects”). The paper clarified the concept of process gains (or synergy) in teams, and introduced recent findings from basic psychology that can be valuable for the management of high performing teams.
David indicates that he thinks differently and seems to believe that if he was on a panel his choice would not be accepted by the group. I think that he sees himself as a maverick and not suited to group work. This is a shame as my personal experience is that being part of a well-run team is a wonderful experience that provides a richer and better outcome. In a good team diversity of opinions and voices are celebrated – mavericks are welcomed. He is missing out if he doesn’t give it a try.
Joy: Panels are clearly not inferior to individuals selecting the best books. The diversity of voices from multiple judges has time and time again resulted in a better outcome.
DK: There are plenty of examples of great books and authors that missed out on being nominated by panels but became best sellers.
Craig: This may be true but he doesn’t consider the proportion of books that succeed that are identified by panels for awards compared to those that don’t. It would be reasonably easy to review this by comparing sale results to the books identified by the relevant panels. I am happy to put a $50 bet on with David that on balance the former would be the case.
Joy: Literary awards and panels are not the only way to identify, promote and celebrate great books. The Archibald has the Packing Room prize – we also have equally as important general public or bookshop popularity awards and prizes. However, literary prizes and awards also clearly have a positive impact on the sales of books and provide a reputable way to introduce quality books to the broader public.
DK: There are better alternatives. The examples he gives are:
Use individual judges rotated each year.
Stop at the long list of twelve books.
Stop at the short list of five books.
Joy: Should the process be truncated? It is true that a lot of the hard work is done to get to the long list stage and there have typically been 3-4 panel sessions to winnow out books to a quality list. Also, generally, many or most books on a shortlist could be a winner. The impact of being shortlisted cannot be under-estimated. Although all awards are different, the impact of obtaining a CBCA short list nomination for children’s books is very significant. The awards may be announced 12 months after a book’s first publication date and the award means that the publishers often re-print at least the original run.
However, the process of then announcing a winner provides another opportunity to publicise great books. Sales of quality books are positively impacted at both the short list and winner stages of the panel process.
DK: The best and surest measure is the wisdom of the crowd with the verdict of posterity selecting great books.
Joy: I agree that there is wisdom in crowds, and books are successful whether they have been shortlisted by an award or not. However, this is complementary not a replacement for literary awards and panels. The better question is whether there is a better alternative to literary awards and panels, or would we be better off without anything. For the reasons outlined above, and in my extensive experience with judging adult and children’s book awards over many years, this is not the case. They make a positive and material difference to the identification, publication, celebration and demand for quality books.
Craig: There is a saying that ‘sword sharpens swords’. Let’s hope that this debate leads to a superior outcome for literary panels and awards, and how we all benefit from quality books – without just stabbing each other in the back.
by Craig Lawn
paperbarkwords.blog 
One thought on “Are Literary Panels a Bad Idea?”