OpenAI's Astra for Law Beats GPT-6 by 40% on Legal Research Bench
OpenAI launched Astra for Law, combining GPT-6 Astra with a legal search index covering over 230 million URLs. On Vals AI's Legal Research Bench, it passed 54.0% of questions versus 38.7% for GPT-6 Astra with web search alone—a 40% relative improvement. The offering includes Zero Data Retention, 26 partner plugins, and custom workflows from firms like Sullivan & Cromwell and Ropes & Gray.
Astra for Law passed the evaluation's overall correctness check on 54.0% of questions, compared with 38.7% for GPT-6 Astra using web search alone – a 40% relative improvement.
- ivraatiems
I know someone who works in law and deals particularly with an area of US benefits and healthcare law. One of their workflows for lower-level employees at their firm involves taking in documents from healthcare plans and organizations, analyzing them for certain kinds of data, and then importing that data into an internal system they use to analyze and provide guidance on plans. The internal system can contain hundreds of documents for an individual client. All of the documents have the same information (roughly) but in totally diverse formats and styles. Once it's in the system, it's easy to compare and analyze across documents and the research process is much faster.
They recently bought a Claude subscription and began using Claude to do the initial read of the documents and output JSON they can import into their internal systems. The work still must be reviewed by an attorney - Claude is nowhere near making the kinds of judgments a lawyer would make about this content - but it has increased their throughput from 2-3 documents an hour to 8-10 documents an hour by killing the busy work.
LLMs have great advantages for this kind of work - but not for decision-making. I just don't see OpenAI ever admitting that.
(I've left some details intentionally vague because this is a very specific area of law and I don't want my friends to be identified without their consent.)
- DannyBee
Lawyer here (non practicing so to be clear none of this affects me):
most comments I read here don't seem to realize that different areas of law have very very different economic models and don't even mention which one they think will be affected or why, they just sort of lump it all together.
For example: It is highly unlikely llms will have any meaningful effect on high value personal injury law - I don't see a 5 million dollar case being handed to an LLM when the majority of the cost is in trial aids and not even lawyers. It may affect where and how they advertise. It may affect how they work. But it seems really unlikely to put any of them out of business any time soon by people doing it themselves.
Will it affect other areas more? Maybe. Probably? But so far I haven't seen a ton of comments that make specific enough arguments that they could really be debated or responded to effectively with a useful opinion
- halamadrid
The need for actual lawyers will persist I think from my own experience. I attempted drafting a contract with some points myself using AI, but after several edits I wasn't sure if it was correct. Sending it to an actual lawyer ended up in so many corrections I couldn't imagine the first time. One big thing was the overly excessive protective clauses which didn't make sense for reality or conflicted with another.
Its just like code I suppose, if you can read and understand and validate, you can use it to scale and otherwise it could end up being a vibe effort.
- piker
Second paragraph:
> API customers including Harvey and Legora will be able to build on Astra for Law, bringing this intelligence into their own products and workflows.
In other words: "no, no, we're not eating our children to prep for the IPO. Don't worry."
- jumploops
The courts are about to overrun with AI-generated lawsuits (even more so than they have been[0]).
[0]https://www.technologyreview.com/2026/06/04/1138391/courts-c...
- kevinsundar
Can someone explain to me the economics of AI and how it intersects with billing by the hour?
I think there would be a strong incentive not to use tools that speed up your work because you'd effectively be able to bill less time?
I'm sure there are some firms out there with more work than people, but still wouldn't it be more effective to hire another human who can then bill at a high rate for many hours?
- ayushiyerji
What we contributed to Reddit and Arxiv voluntarily before, what we contribute as traces right now are all being used to build business verticals by OpenAI and Anthropic. All these business verticals are being used for is more trace collection which would only strengthen these models and render most humans useless because the ceiling for 1000 swarms of Agents to learn is much higher than an average human.
These companies clearly can identify the relevant traces for these business verticals which only strengthens the argument that NS was built on someone else's traces.
- cbg0
No word on model hallucinations in the blog post.
https://artificialanalysis.ai/models/gpt-6-astra?omniscience...
- chr15m
"We have dangerous AGI that can destroy humanity."
"Also, all of your sensitive legal documents will be totally safe with us."
"Also, for some reason even though we have AGI and selling tokens is a fine business, we need to sell a new product specifically targeted at a very high margin and lucrative industry."
- Digory
This seems like a very similar set of tools to Anthropic. But I’ve enjoyed using the various frontier models to criticize each other.
Using recursive loops, the output has gone from a high school level intern to a 2nd year lawyer in about a year. It still doesn’t beat the experts, but so much legal work is (legally significant) pedantry, not legal philosophy.
AI will not kill off lawyers, or reduce the amount of litigation. It will increase volume and velocity.