OpenAI CPO Kevin Weil says their o1 model can now write legal briefs that previously were the domain of $1000/hour associates: "what does it mean when you can suddenly do $8000 of work in 5 minutes for $3 of API credits?" pic.twitter.com/MotT9Oo9rv— Tsarathustra (@tsarnick) October 19, 2024 OpenAI's Chief Product…
The bar exam is just one part of a larger set of qualifications
The bar exam is just a (closed-book) proxy for the actual skills and knowledge being tested. While a reasonable proxy for humans, it is a poor proxy for computers
The short answer is being admitted by the bar; we already trust them to certify humans.
If for some reason I were arbiter, I would say a convincing record of doing actual legal work, vetted by existing lawyers. The legal profession already has a well-defined model of how non-lawyers can contribute to the work, so there is no need for a quantum leap up to being a lawyer.
Ok. What test would an LLM need to pass to convince you that it capable of being a lawyer?
The short answer is being admitted by the bar; we already trust them to certify humans.
If for some reason I were arbiter, I would say a convincing record of doing actual legal work, vetted by existing lawyers. The legal profession already has a well-defined model of how non-lawyers can contribute to the work, so there is no need for a quantum leap up to being a lawyer.
Draw a pentagon