AI is good at looking right. It reads a document, returns a tidy answer, and the answer usually seems about right. In mortgage lending, about right isn't good enough. A tax figure that's off by one quarter or an insurance coverage pulled from the wrong endorsement changes the loan.
So Vontive built a way to measure it. Working with Vals AI, an independent AI evaluation company, Vontive has released the mortgage industry's first benchmark for how accurately large language models handle mortgage data processing. The benchmark is public, so any lender, vendor, or capital allocator can use it to test the systems they rely on.
What the benchmark tests
The benchmark is built on an expert-annotated dataset from Vontive covering 23 types of documents that show up in real underwriting files. These include county tax certificates, insurance binders with their endorsements and coverages, rental leases, full title reports, LLC formation documents, operating agreements, and HUD forms.
It tests two kinds of work. The first is semantic extraction: finding the right fact in a long, messy document. The second is numerical extraction, where the model has to do math as well as read. Annualizing a property tax bill from quarterly or semi-annual county assessments is a good example. A person does it without thinking. A model often gets it wrong.
Every answer is scored against what a human expert says is correct. The only measure that counts is accuracy.
What the results showed
The best model in the study was Anthropic's Claude 3.7 Sonnet, running without extended reasoning, at 80.6% accuracy. It did well on both semantic and numerical tasks. Claude 4.0 with extended reasoning scored 62.5%. Meta's Llama 3 scored 55.3%.
Two findings stood out. First, moving to larger, more expensive models produced only minimal gains. Second, smaller models lost accuracy in ways that mattered. Bigger isn't automatically better, and cheaper can cost more in errors.
The benchmark also showed where the gains really come from. After Vontive re-ran the evaluation with Vals AI using improved extraction prompts, accuracy rose to 90%, in line with human data entry. Vontive's production system, which pairs the models with its own validation layer, reaches 95% accuracy across every document type it supports.
Why build it, and why share it
Vontive didn't set out to build a benchmark. It set out to process mortgage documents correctly, and it found that no reliable way existed to tell AI that works from AI that only looks like it works. Vontive expected every lender adopting AI to run into the same problem, so it built the measurement itself and made it available to the industry.
The idea borrows from how AI research has always moved forward. Stanford's question-answering dataset, known as SQuAD, gave natural language processing a shared test that everyone could measure against. Progress became visible, and claims became checkable. Mortgage lending deserves the same thing. With a common benchmark, a lender evaluating an AI vendor can ask for a score instead of accepting a demo.
Built on years of real loans
The benchmark grew out of work Vontive has been doing for a long time. Vontive has structured mortgage data since its first loan in 2017, and every file on the platform lives in one common data foundation. That history is what made an expert-annotated dataset possible in the first place.
Vontive's AI Underwriter went into production on July 23, 2024. In its first year, it parsed 35,000 documents for 1,700 borrowers across 6,845 loans, representing $1.713 billion in requested loan balances. It supports more than 40 document types, covering everything typically required in a mortgage underwriting file. Every extracted value is then checked against thousands of criteria before the file moves forward.
The software gathers documents, verifies data, and prepares the file. Human experts supervise the work and make every credit decision. The benchmark is how Vontive makes sure the software half of that partnership is as accurate as the people relying on it.
What's next
Vontive will keep testing new models against the benchmark as they're released to expand and improve its data foundation, agentic workflows, and AI infrastructure.
Lenders and capital allocators that want underwriting built on verified data, and engineers who want to work on these problems, can reach Vontive at vontive.com.
Media contact: media@vontive.com
Press coverage
- Vontive sets mortgage industry's first benchmark for AI performance on data processing tasks — Business Wire, July 2025
- Vontive sets mortgage industry's first AI benchmark — FF News
- Mortgage tax benchmark — Vals AI