Legal AI Wins Independent Test It Wrote, Marked and Sponsored

LexNonsense has been named the highest-performing legal AI in a new independent study published by an institute that shares its office, reception staff and former head of sales.

The report’s forty-page methodology establishes a strict separation between the sponsor and the research. The sponsor was responsible for the questions, scoring framework, acceptable answers and final approval. The researchers were responsible for the page numbers.

The Unbillable has reviewed the methodology, reproduced below with its disclosures restored to the same size as the findings.

1. Selection of tasks

Researchers chose one hundred representative legal tasks from a library supplied by LexNonsense. The library was developed during the product’s training and demonstration process, ensuring each question reflected a problem the software had already had an opportunity to solve.

Unfamiliar tasks were excluded because their inclusion might introduce variability.

2. Definition of accuracy

Responses were scored against model answers approved by the sponsor. Where a competitor reached the same legal conclusion using different language, assessors deducted points for divergence from the validated response.

One question asked which platform a modern legal department should procure. LexNonsense received full marks. A competing system requested information about the department’s requirements and was penalised for failing to answer.

3. Testing conditions

All products were given access to the same public documents. LexNonsense was also given its standard implementation support, configured data sources and a specialist available to correct unsuccessful runs.

Competitors were tested using the free accounts researchers could obtain before lunch. The report describes this as an evaluation of real-world accessibility.

4. Independent oversight

The study’s director previously led sales at LexNonsense but now works for the institute. Her independence is confirmed by a separate employment contract and a different email signature.

The institute’s university-style crest contains a book, a torch and a Latin phrase selected by the sponsor’s design agency. It does not represent an affiliation with a university.

5. Publication and sponsorship

LexNonsense had no influence over the findings beyond defining what constituted a correct answer and requesting changes necessary to prevent misunderstanding.

These included removing two unsuccessful tasks, replacing ‘vendor-selected’ with ‘industry-relevant’ and clarifying that the report could be published once the sponsor was satisfied with its independence.

A law-firm procurement committee has accepted the study as evidence that its preferred supplier outperforms the market. A request to test the product on the firm’s own documents has been deferred: the independent report already runs to forty pages.

The committee’s decision is now available as a customer validation. LexNonsense intends to include it in the next benchmark.

This benchmark, institute, methodology and all reported results are fictional and were created for satire.