John Bliss 12/30/23.
Legal AI is a fast moving field. In just the past year, we saw OpenAI’s foundation models improve from failing the Unified Bar Exam (miserably, with GPT-3.5 scoring below the 1st percentile) to passing it (comfortably, with GPT-4 scoring at roughly the 62nd percentile among first-time takers).[1] We have seen similar jumps on law school exams, from GPT-3.5 barely passing to GPT-4 reaching the A- to A range with the best-performing prompts.[2]
This technology is now being integrated in the leading applications for legal research (e.g. Lexis+ AI), word processing (e.g. Microsoft 365 Copilot), and internet search (e.g. Google’s Search Generative Experience). These tools can automate the drafting of legal documents and assist with a wide range of legal tasks. Legal AI start-ups claim waitlists of thousands of law firms.[3] A recent survey found that 73% of lawyers plan to incorporate generative AI in their practice over the next year.[4] This technology is poised to become a standard tool, assistant, and collaborator in the process of legal research and writing. Yet, there is great uncertainty about the quality of legal AI outputs, which is an ongoing subject of inquiry among researchers and legal practitioners.
It is possible that this blog will end up serving as documentation of the rapid ascent toward legal AGI (“artificial general intelligence”), aka robot lawyers, rivaling or surpassing the proficiency of human lawyers in many respects. But it is also possible that AI progress will be more gradual, yielding tools that moderately assist rather than replace lawyers in most capacities. The research reviewed in this blog provides an up-to-date look at the current state of legal AI advancement, and the implications for the quality and efficiency of legal work, the well-being of lawyers, legal ethics, inequality in the legal profession, access to justice, and the changing nature of the lawyer role.
We will need to update these discussions as the technology advances. This might all look very different with GPT 5, 6, 7, or whatever the next leading AI systems will be called, including non-GPT systems (e.g. analytical, predictive, rule-based, robotic, symbolic, knowledge-representing, quantum, affective, and other future approaches to AI). And this may all look very different as new legal applications are developed on top of foundation models. Headlines and social media discourse can be misleading, and the messages from the tech industry are often overly hyped. What is needed—the premise for this blog—is a central place to find regularly updated research-based understandings of this rapidly evolving field.
Subscribe below. Please reach out with comments, suggestions, and contributions: jbliss@law.du.edu.
[1] Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo, GPT-4 Passes the Bar Exam (Working Paper, March 15, 2023), https://ssrn.com/abstract=4389233; Eric Martínez, Re-Evaluating GPT-4’s Bar Exam Performance (Working Paper, May 8, 2023), https://ssrn.com/abstract=4441311.
[2] Jonathan Choi & Daniel Schwarcz, AI Assistance in Legal Analysis: An Empirical Study (Working Paper Aug. 16, 2023), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4539836(finding that GPT-4 when prompted well passed law exams above the median, achieving an A- and an A under the best-performing prompts); Jonathan H. Choiet al., ChatGPT Goes to Law School, 71.3 J. Legal Educ. 387 (2022) (finding that GPT 3.5 passed law exams with a grade of roughly C+)
[3] Sara Merken, Legal AI Race Draws More Investors as Law Firms Line Up, Reuters (Apr. 26, 2023), https://www.reuters.com/legal/legal-ai-race-draws-more-investors-law-firms-line-up-2023-04-26/; Julie Sobowale, Law’s AI Revolution is Here, CBA Nat’l. Mag. (Mar. 27, 2023), https://nationalmagazine.ca/en-ca/articles/legal-market/legal-tech/2023/law-s-ai-revolution-is-here.
[4] Wolters Kluwer, Future Ready Lawyer 2023 Report (2023), https://www.wolterskluwer.com/en/news/future-ready-lawyer-2023-report.
[5] See https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/.
[6] Id.
[7] Max Roser, AI Timelines: What Do Experts in Artificial Intelligence Expect for the Future?, available at https://ourworldindata.org/ai-timelines.
Empirical Studies of AI Lawyering (e-SAIL) 