Show Your Work: Can Small Open Models Read SEC Filings with Evidence a Machine Can Check?
Source
arxiv.org
Author
Krishna Nandula
Date
Why it matters
Evidence-citation checks catch many wrong answers but not all. 57% of wrong answers that slip through are scale or unit errors, so extraction pipelines need checks beyond quote matching.
Key takeaways · AI-distilled
Filing-Evidence has 1,000 questions over 90 recent 10-K and 10-Q filings from 45 US companies. An answer needs the page and a verbatim quote containing the number, and answer keys come from the filers' inline XBRL tags.
Ten instruct models of 7B to 21B parameters, run on one RTX 4090, scored 17.1% to 82.7% with checkable evidence in the plain setup. The best reported result was 84.5%, from Qwen3.5-9B.
Between 16 and 167 answers per 1,000 gave a wrong value for an answerable question and still passed every automatic check. In the plain setup, 30% of these took another period or column from the right line.
Reasoning mode significantly reduced these check-passing errors for 4 of the 5 models that offer it. Retrying after a failed check raised accuracy by up to 3.9 points but never reduced them.