Predictive Coding: Machine Learning Disrupts Discovery | Law

Predictive Coding: Machine Learning Disrupts Discovery

OPINION: Associate Professor Michael Legg and Thomas Davey (UNSW Alumni), The Law Society Journal, April 2017.

The decision in McConnell Dowell Constructions (Aust) Pty Ltd v Santam Ltd (No 1) [2016] VSC 734 (‘McConnell Dowell’) signals the emerging judicial acceptance of predictive coding and should encourage practitioners in New South Wales to embrace the technology. Predictive coding provides solicitors with a rare opportunity to dramatically improve the time efficiency and cost effectiveness of court proceedings while also enhancing justice.

Machine learning, predictive coding and technology assisted review

Predictive coding is the application of machine learning to the process of discovery. It is a process which uses statistical modelling to make predictions about the relevance of documents in discovery in lieu of human review. Predictive coding has been frequently referred to as ‘technology assisted review’ or ‘TAR’ by the legal community. Unfortunately, this term obfuscates the range of ways in which technology assists in the review process. TAR is frequently and indiscriminately used to refer to keyword searches, concept searching, predictive coding, or all of the above. In practice, each of these technologies works differently, produces different results, and achieves different levels of accuracy. This article champions predictive coding, the most groundbreaking and disruptive of those technologies, which is changing the way in which large scale electronic discovery takes place.

Predictive coding addresses the challenges posed by electronically stored information (‘ESI’) during the process of discovery. ESI is the cause of significant cost and delay in the discovery process due to its volume, duplicability, and dispersion. Rapid and continuing developments in information technology have increased the number of potentially relevant documents that need to be analysed. Estimates suggest that 90 per cent of business records are already held in electronic format only and individuals within businesses will send and receive an average of 140 emails a day by 2018.

Predictive coding is concerned with the collection phase of discovery in which documents are gathered and reviewed for relevance. Frequently, the desired documents only make up a small proportion of the documents initially obtained. This process has been traditionally undertaken by junior associates and paralegals who manually review documents to identify those which may be relevant. Predictive coding, on the other hand, uses machine learning to identify relevant documents. The process sees a senior lawyer or small team review and code a ‘seed set’ of documents. A computer then identifies similarities and patterns within those documents and attempts to predict the coding for additional samples. When the coding of the human reviewers and the computer sufficiently coincides, the computer is able to make confident predictions for the balance of the documents.

The total number of documents reviewed by the senior lawyer is typically only a few thousand. Studies have determined that predictive coding can yield more accurate results than exhaustive manual review and, in some cases, offer a fifty-fold saving in terms of documents reviewed manually (see Maura Grossman and Gordon Cormack, ‘Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review’, (2011) 17 Richmond Journal of Law and Technology 11).

Has TAR been judicially approved?

The first judicial decision concerning the use of predictive coding was made by Magistrate Judge Peck in the United States decision of Da Silva Moore v Publicis Groupe et al (2012) 287 F.R.D 182 (‘Da Silva Moore’). In that case, Judge Peck held that predictive coding was better able to process the three million electronically stored documents than any of the available alternatives. Statistical evidence examined during the trial revealed that the concept of manual review as the ‘gold standard’ was a myth and computerised searches were ‘at least as accurate, if not more so.’

Judge Peck criticised the utility of keyword searches (which are widely used in Australia). He stated that such searches require guesses to be made by legal practitioners about which keywords might support their clients’ case. The results are often inaccurate and, in many cases, counsel doesn’t know what is in its own client’s documents. Keyword searches frequently achieve a mere 25 per cent recall and 25 per cent precision. In contrast, the machine learning technology that underpins predictive coding sees the computer learn from experience over time and produce results that are better in every measure.

When used in discovery, predictive coding extrapolates human judgments to a broad collection of documents, emulating human decision-making and reducing the cost and time of review. Similar technology has been used in selfdriving cars, language translation, and speech and vision recognition.

In what may be a relief to many lawyers, Judge Peck held that the Court should be less interested in the science behind the ‘black box’ of the predictive coding vendor’s software than its production of responsive documents with reasonably high recall and precision.

The Da Silva Moore ruling was made in pre-trial discovery proceedings, of which Magistrate Judge Peck was supervising. District Judge Carter went on to affirm Peck’s decision stating that it was well reasoned and considered the potential advantages and pitfalls of the predictive coding software.

From novel technology to black letter law

In 2014, the United States Tax Court held that predictive coding had become a widely accepted method for facilitating effective discovery of electronically stored information without undue burden (see Dynamo Holdings Ltd. P’Ship v Comm’r of Internal Revenue, 2014 WL 4636526). In Rio Tinto PLC v Vale S.A 306 F.R.D 125 (S.D.N.Y. 2015) (‘Rio Tinto’), three years after Da Silva Moore, Judge Peck stated that the use of predictive coding in the United States had become ‘black letter law’.

This string of cases was picked up in Ireland (see Irish Bank Resolution Corporation Limited v Quinn [2015] IEHC 175) and England (see Pyrrho Investments Ltd v MWB Property Ltd [2016] EWHC 256 (‘Pyrrho’)) where the courts acknowledged the connection between the advantages of predictive coding and the broader purposes of the judicial system. Namely, that it provides a proportionate solution to the discovery of expansive ESI.

Australia’s cautious approach

Ultimately, it was from the ratio of Pyrrho that Justice Vickery drew the majority of his reasoning in McConnell Dowell. In that case, the central issue was how discovery of approximately four million electronic documents could and should be managed consistently with the principles of proportionality and s 9 of the Civil Procedure Act 2010 (Vic).

Vickery J noted that principles of civil procedure referred to in Pyrrho were closely aligned to those in Victoria. The vast quantity of documents and the fact that it would be near impossible to accurately review them through traditional means also reflected the issues in Pyrrho. Vickery J appointed a senior barrister as a Special Referee to answer questions as to the appropriate management of discovery. In answering the questions, the parties came together to undertake a ‘due diligence’ examination of predictive coding. The conclusion of the special referee, the parties and the Court was that just, efficient, timely and cost-effective discovery would be best achieved through the sophisticated process of predictive coding.

Vickery J held that predictive coding facilitated a reasonable search for documents and that the costs associated with doing so were proportionate. Not only was the process more sophisticated than a word search facility it was also able to dramatically reduce the search volume and bring it within ‘reasonable and manageable bounds.’ The fact that traditional review of the documents would have taken approximately 583 working weeks clearly demonstrated that manual discovery was not appropriate.

The law as it stands

Vickery J’s judgment foreshadowed Practice Note SC Gen 5 – Technology in Civil Litigation which was issued earlier this year. The note states that the use of technology in civil litigation should facilitate the just, efficient, timely and cost-effective resolution of the real issues in dispute. The Court expects parties to ‘acquit their obligation to ensure costs are reasonable and proportionate by employing technology to save time and cost wherever possible.’ The note also states that ‘the use of common technologies is a core skill for lawyers and a basic component of all legal practice’ and that ‘the inability or reluctance of a lawyer to use common technologies should not occasion additional costs for other parties’. In relation to discovery, it states that ‘technology assisted review will ordinarily be an accepted method of conducting a reasonable search in accordance with the Rules of Court.’ The note makes direct reference to the decision in Pyrrho and states that the costs of manually searching documents in larger cases may not be reasonable and proportionate.

The Federal Court and the Supreme Court of New South Wales

Given the acceptance of predictive coding in Victoria and across the world, it is likely that the technology will receive judicial recognition in other Australian jurisdictions soon. With judicial approval mounting, the relevant practice notes of the Federal Court of Australia and the Supreme Court of NSW both appear to be broad enough to permit TAR without express judicial intervention. Practice Note SC Gen 7 of the NSW Supreme Court and the Technology and the Court Practice Note of the Federal Court of Australia both encourage the use of technology in civil litigation and require all parties to consider the prospect of using technology at various stages of the trial. The Federal Court note in particular states that technology should be used to assist in achieving the quick, inexpensive and efficient resolution of proceedings.


It may be that solicitors are now under an obligation to consider predictive coding in cases where large quantities of ESI exist. Whichever path is chosen, solicitors will need to satisfy the court that their selected process meets the standards of each jurisdiction. When appropriately implemented, predictive coding should enable a document review process that is more just, time efficient, and cost effective than traditional manual review.


Predictive coding can enhance the just resolution of a dispute by improving the quality of discovery and, indirectly, by reducing cost and delay. The quality of predictive coding in relation to large sets of documents is superior to that of manual review. As Judge Peck said in Da Silva Moore, ‘even if all parties here were willing to entertain the notion of manually reviewing the documents, such review is prone to human error and marred with inconsistencies from the various attorneys’ determination of whether a document is responsive.’

When it comes to large data sets, ‘technology assisted review using predictive coding is at least as accurate as, and, probably more accurate than, the manual or linear method in identifying relevant documents’. Researchers have concluded that human reviewers often suffer from fatigue, inattention and boredom and that predictive coding exceeds the effectiveness of human review with a small fraction of the effort (see Maura Grossman and Gordon Cormack, ‘A Tour of Technology-Assisted Review’ in Jason Baron, Ralph Losey and Michael Berman (eds), Perspectives on Predictive Coding (American Bar Association, 2016) chapter 3).

Time efficiency

The concept of justice is also closely intertwined with cost and delay. Courts have long struggled with the causes of delay, but have recognized justice is illusory if it cannot be achieved within a reasonable time. In Aon Risk Services Australia Ltd v Australian National University (2009) 239 CLR 175 it was held that there is ‘an irreparable element of unfair prejudice in unnecessarily delaying proceedings.’ Predictive coding presents a solution to delays caused by expansive discovery. The cost of the computer processing power required to conduct predictive coding has become affordable to a large percentage of litigants and the bulk of the time required for it to work can be attributed to the human review component. With the reductions in human review on offer, predictive coding is able to significantly reduce the time of the discovery process.

Cost effectiveness

It is estimated that document review accounts for 70 per cent of all discovery costs. The suitability of any given technology in litigation should be measured with reference to its ability to make the proceedings more costeffective. Doing so enhances justice for both the participating parties and the community at large. It prevents premature settlement and improves access to justice for those who would otherwise be unable to bear the burden of an ESI discovery.

Predictive coding reduces costs by simply reducing the number of hours needed for fee-earners to review documents. Using standard rates, it would take over 1,000 hours and cost over $250,000 for a human to review 100,000 documents, at a speedy rate of 100 documents an hour. The same review completed using predictive coding would cost a mere $50,000 in labor as humans review just one fifth of the original set. In many cases, even greater savings may be achieved.



Like all cloud based technologies, predictive coding presents a cyber security risk. Stacey Blaustein, the head of IBM’s eDiscovery program, has noted that the volume and severity of attacks on ESI has increased exponentially. According to Blaustein, 80 per cent of the United States’ largest law firms have experienced a data security incident. The risk can be mitigated by pre-engagement risk analysis, the implementation of appropriate cyber security measures, and the taking out of suitable cyber insurance policies (see: www.insurancelawtomorrow. com/2016/12/how-to-protect-yourbusiness-from-cyber-threats ‘).

Procedural risks

The overriding purpose of the justice system is facilitated through careful case management by the presiding judge. In this regard, predictive coding presents unique challenges for the judiciary.

Complicated ESI and varied technologies make case management a difficult task. There is a risk that the benefits of predictive coding will not be realised if judges are unable to encourage parties and their lawyers to cooperate. It is likely that the parties rather than the judge will be best placed to determine what procedure is appropriate. In Rio Tinto, the approved protocol incorporated a collaborative process of review which saw both parties check the relevancy of seed documents. This process assisted in satisfying the court and the requesting party that the computer had been appropriately trained. Such collaboration may not be common in Australian courts; however, it may be that documents produced through a process of predictive coding without collaboration are more likely to be challenged. If the judge is unable to maintain genuine cooperation between the parties the overriding purpose may not be met and the benefits of predictive coding may be lost.

Ethical obligations

The availability of predictive coding presents ethical risks for technophobic lawyers. Solicitors are under a prevailing and paramount duty to the court and the administration of justice. To this end, they are duty bound to adhere to instructions of the court, including those contained in practice notes. Interestingly, a recent survey of US Federal Court judges revealed a belief that a majority of lawyers are failing to make proper use of technology when it comes to discovery. It is likely that this holds true for the Australian legal profession. This trend poses a risk for lawyers.

It may be that solicitors are obliged to consider the use of predictive coding in all cases that have a large amount of ESI. The Supreme Court of Victoria already ‘expects parties to be in a position to address questions [of technology in review] at the point at which discovery is sought.’ The solicitor’s obligation to deliver services competently, diligently and as promptly as reasonably possible may encompass an obligation to advise clients of the avenues available to manage discovery and their respective costs. Considering predictive coding is often able to provide significant cost savings, practitioners may be under an ethical obligation to consider its use. The American Bar Association has recently updated their rules to ensure lawyers keep abreast of benefits and risks of relevant technology.


It is certainly not the case that predictive coding will be appropriate in all circumstances, but its widespread adoption suggests that it should be seriously considered in cases where large amounts of ESI are likely to cause significant cost or delay.

Article from LSJ, April 2017, Issue 32, pp.82-84.

**Thomas Davey is a UNSW Law alumni. Thomas wrote his major essay on technology assisted review, also called predictive coding, in 'Complex Civil Litigation' taught by Associate Professor Legg. After the course was finished, Associate Professor Legg helped Thomas to rework his essay for the LSJ.