A recent Stanford Law School-led study found that law professors overwhelmingly preferred answers generated by artificial intelligence over responses written by fellow professors when evaluating questions commonly asked by law students.
The study, titled “Law Professors Prefer AI Over Peer Answers,” examined whether large language models could serve as effective tutors in contracts law courses, a subject that frequently requires nuanced judgment, interpretation and analysis rather than straightforward factual recall.
Researchers conducted a blind evaluation involving 16 law professors from universities across the United States. The professors reviewed nearly 3,000 anonymized comparisons without knowing whether each response had been written by another professor or generated by an AI system.
The AI-generated answers were preferred in approximately 75% of the head-to-head comparisons. The systems also performed comparably to the highest-rated human instructor participating in the study.
The research was led by Stanford Law School Professor Julian Nyarko, who directs the school’s Legal Innovation through Frontier Technology Lab, known as liftlab. The paper was co-authored by researchers affiliated with Stanford, Yale University, New York University, the University of Chicago and other institutions.
The study focused on contracts law because legal education tests capabilities that are often difficult to measure through conventional AI benchmarks. Legal analysis can require students to identify competing interpretations, apply doctrine to new circumstances, evaluate the strength of different arguments and reach defensible conclusions in situations where there may not be a single correct answer.
Previous assessments of AI systems have often concentrated on questions with clear solutions, such as mathematical problems, standardized examinations or factual knowledge tests. The Stanford-led researchers instead sought to determine whether AI-generated legal analysis could meet the professional standards that legal educators use when evaluating reasoning and instruction.
For the study, participating professors created 40 representative questions that contracts students might ask during office hours or after class. The professors then prepared their own answers to those questions.
AI systems were also used to produce responses. The researchers calibrated those responses to resemble the length and general structure of the professor-written answers, helping prevent evaluators from determining their origin based on formatting or writing style alone.
The professors then assessed the anonymized responses based on their accuracy, reasoning, clarity and educational value. They also evaluated whether an answer could mislead, confuse or otherwise harm a student’s understanding of the subject.
In addition to receiving higher overall preference scores, AI-generated answers were substantially less likely to be identified as potentially harmful from an educational perspective.
Professors flagged AI responses as pedagogically harmful in approximately 3.5% of evaluations. By comparison, professor-written responses were flagged in about 12% of evaluations.
The difference suggests that AI-generated explanations may sometimes provide students with clearer, more complete or better-structured guidance than an answer produced quickly by an instructor responding to an individual question.
However, the researchers did not conclude that AI systems should replace professors or classroom instruction. Instead, they said the findings support further examination of how AI tutors could complement legal education by providing students with additional, on-demand guidance.
Such systems could potentially answer questions outside normal office hours, offer alternative explanations of difficult concepts and provide more students with access to individualized assistance. This could be especially valuable in large classes or at institutions where faculty members have limited time to provide personalized support.
The study also raises questions about how AI could expand access to high-quality legal education. Students who cannot regularly attend office hours or afford additional tutoring may be able to receive immediate explanations from AI systems trained or configured for specific courses.
Researchers said the technology could be particularly useful when it is connected to assigned readings, lecture materials and course-specific resources. This approach could help ensure that AI responses reflect the doctrine, terminology and analytical methods emphasized by a particular professor.
The study evaluated several AI products and configurations, including commercial tutoring systems and Google’s NotebookLM. Performance varied between the systems, and some responses were affected by limitations in the context or source material available to the model.
Even when those limitations affected an answer, however, professors frequently preferred the AI response to the alternative written by another instructor.
The findings arrive as law schools continue to determine how generative AI should be incorporated into legal education. Some schools have introduced courses, research initiatives and experimental programs focused on AI-assisted legal work, while others remain concerned about plagiarism, hallucinations, excessive reliance on automation and the potential weakening of students’ independent analytical skills.
These concerns are particularly significant in law, where students must learn to evaluate evidence, interpret precedent, recognize ambiguity and construct persuasive arguments. An AI system that provides an answer too readily could prevent students from developing those capabilities if the technology is used as a substitute for their own reasoning.
The researchers emphasized that their study measured the quality of AI-generated answers rather than the long-term educational effects of using AI tutors. Additional research will be needed to determine whether students who regularly receive AI assistance develop stronger legal reasoning skills or become overly dependent on the technology.
Effective implementation could require safeguards that encourage students to examine the reasoning behind an answer, compare competing interpretations and verify the underlying legal authorities rather than simply accepting an AI-generated conclusion.
The study nevertheless challenges the assumption that AI-generated legal guidance is inherently less accurate or less useful than instruction produced by experienced educators.
It also suggests that the debate surrounding AI in legal education may need to move beyond whether the technology can produce credible answers. The more important questions may involve how AI tutors should be designed, how their outputs should be supervised and how they can support students without replacing the intellectual work required to become effective lawyers.
KEY QUOTES:
“This study challenges important assumptions about AI’s role in legal education. We focused on law precisely because it requires judgment, nuanced reasoning, and the ability to navigate ambiguity—not just factual recall. We were frankly surprised by the magnitude of the results. These weren’t just simple questions with obvious answers. Many of them required synthesizing complex material, applying it to new situations, and explaining legal concepts in ways that would help students develop their own analytical skills. We designed this study to be as rigorous as possible because the stakes are so high. Legal education is about training future lawyers to think critically, argue persuasively, and navigate ethical complexities. Our study makes important steps toward finding out whether AI could support that mission. Our study evaluates the quality of answers given by AI tools. But how to implement these tools to most effectively improve student learning is still an open question. So we’re not advocating for wholesale adoption of AI tutors. But our data suggests that blanket skepticism may be equally unwarranted. The conversation should shift from whether AI can give accurate, high-quality responses to how we can deploy it responsibly to the benefit of our students.”
Julian Nyarko, Professor at Stanford Law School and Director of liftlab
“In most fields where AI gets tested, there’s a right answer. In law, there often isn’t. Two opposing arguments can both be good. What we wanted to know is whether AI can meet the latent professional standard that lawyers use to evaluate each other’s arguments. In this case, the answer was yes.”
Sarath Sanga, Professor at Yale Law School and Study Co-Author
“Our study shifts attention to what AI tutoring can contribute to learning in judgment-rich fields like law. We find that, when evaluated by legal educators, AI tutors can offer high-quality, on-demand support that complements classroom instruction and may broaden access to expert guidance.”
Alejandro Salinas, Researcher at Stanford Law School’s liftlab and First Author of the Study
