Recap: Human-AI Cooperation in E-Discovery, the First Twenty-ish Years with David Lewis - Institute for Experiential AI

Recap: Human-AI Cooperation in E-Discovery, the First Twenty-ish Years with David Lewis

  By Tyler Wells Lynch

The Institute for Experiential AI welcomed Dave Lewis, Ph.D., executive vice president for AI research, development, and ethics at Reveal, to discuss advances in AI-assisted legal discovery, including prospects for other industries given the 20-year history of machine learning in law. The lecture is part of EAI’s Distinguished Lecturer Series. Watch the full replay or read on for an event summary.



In legal proceedings, “discovery” refers to the process of gathering facts that are relevant to a case, mostly in the form of documents. “Electronic discovery” or “e-discovery” is a form of Technology-Assisted Review (TAR) involving information of digital or electronic origin. Discovery may involve teams of reviewers, complex workflows, and potentially millions of documents. To lessen the load, judges, attorneys, and legal services have turned to AI.

The use of AI in legal discovery has two purposes: To minimize costs and improve accuracy. Some firms may prioritize cost savings while others care more about finding the most relevant documents. Debates surrounding AI-assisted discovery further hinge on the ethical and efficiency pressures of TAR in an increasingly digitized information landscape.


A Brief History

Basic forms of electronic discovery date back to the 1980s with keyword filtering and boolean queries. As firms introduced document scanners, the task of discovery came to involve a hybrid approach involving both computers and banker boxes. By 2006, changes to the Federal Rules of Civil Procedure (FRCP) cemented the inclusion of email in corporate records, expanding legal datasets to include millions of documents. The shift sparked commercial and scientific interest in supervised or AI-assisted workflows.

From 2005 to 2010, e-discovery focused on improving the querying process through a mix of query formulation, thesauri, and natural language processing. Eventually, it became clear that the best path to developing the technology was in iterative or active learning of text classifiers, which refers to using algorithms that can query supervisors to classify and label text. (The cost-efficiency balance, coupled with a variety of legal ramifications, largely preclude the possibility of fully automated e-discovery.)


What does the e-discovery workflow look like?

In its modern format, electronic discovery is an example of human-AI teaming. It begins with one or more professionals or supervisors labeling documents used to train a predictive model. The model is then used to select more documents to show to the supervisor, and the workflow iterates until the discovery phase is done.

But when is the work done? The whole point of e-discovery is to assist in reviewing a portion of a sum of documents. That means a decision has to be made at some point that enough information has been identified. Two kinds of workflows influence this decision.

One-phase workflows (sometimes referred to as TAR 2.0) consist of a linear format, in which a supervisor labels some data, trains a predictive model, and uses the model to select more data (as described above). The stopping rules here are generally based on some heuristic.

An alternative, known as a two-phase workflow, treats the process more like a generalization task. Once the final predictive model is reached, it is used to train a large number of documents that then go to a team of reviewers. For firms, this raises a number of questions about cost-effectiveness, as a two-phase workflow may involve hiring more people to review documents, which raises labor costs while simultaneously lowering the cost per sample. The financials likely vary depending on the firm and the project in question.

Lewis discusses a number of implications for human review, namely in the effectiveness measures that are used to assess models and workflows. One is the “recall,” which is the prevalence of interesting documents that a classifier found.

An alternative effectiveness measure known as “elusion” refers to the proportion of relevant documents that were missed by a classifier, which is then compared with the precision measurement. The goal, then, is to have a low elusion measurement. It’s an odd metric because the original collection might have had a low proportion of relevant material, but the appeal is mostly driven by the social phenomenon of attorneys not wanting to spend a lot of time reviewing negative samples.


Who’s involved in e-discovery?

Lewis identifies the key players involved in e-discovery and how the technology appeals to different parties for different reasons. At the top of the food chain are the legislators and regulators who create laws and rules governing civil procedures, largely at the advice of judges.

Courts, attorneys, legal service providers, technology companies, and academics also have a stake in e-discovery, as they are constantly churning out white papers, law review articles, affidavits, speeches, interviews, blog posts, etc.

What’s interesting is that the abundance of content makes for an unusually robust and legible record of a human-centered AI project. As Lewis says, there’s a great deal to learn about how the introduction of human-AI teaming in an industry tends to play out. The 20-year history of e-discovery serves as a bellwether for successful (or problematic) implementation of these systems.


Controversies in e-discovery

Lewis concludes his lecture with a look at some of the debates surrounding human-assisted AI in legal discovery, and what their outcomes might portend for other fields. He phrases these debates in the form of questions:

  1. As a methodology, is TAR both effective and ethical for the purposes of discovery?
  2. Can producing parties be compelled to use TAR in large cases?
  3. Is it wise for producing parties to filter on keywords and use supervised learning to filter them down even more?
  4. What is the role of the requesting party in e-discovery? Producing parties have great discretion, so without keyword filtering is the requesting party getting what they wanted? Should they participate at all in the labeling of training documents?
  5. How does deep learning factor into e-discovery?

Lewis addresses each of these questions in-depth, including to what extent they’ve been resolved in the e-discovery community. To hear his responses, watch the video replay or read his Q&A.


About Dave Lewis:

Dave Lewis, Ph.D. is executive vice president for AI research, development, and ethics at Reveal. He has variously in his career been a corporate researcher, startup development team leader, research professor, freelance consultant, expert witness, and software company co-founder. In 2006, he was elected a Fellow of the American Association for Advancement of Science for foundational work in text analytics. In 2017 he received an ACM SIGIR Test of Time Award for his paper with Gale introducing uncertainty sampling.