Harnessing LLMs for Document-Guided Fuzzing of Python Libraries

VistaFuzz uses an AI model to read Python library documentation and generate test inputs that satisfy the library's requirements. Its particular focus is relationships between arguments: checking each argument separately can miss conditions that depend on another argument. This matters for fuzzing, which exercises software with generated inputs to uncover problems. The paper evaluates the approach across twelve libraries.

For the tested interfaces with such dependencies, resolving those relationships produced valid inputs more than 95% of the time. Disabling that step reduced the rate to 31.6–52.8%. The comparison measures whether generated inputs satisfy requirements, rather than the percentage of bugs the system detects.

“Testing their APIs requires inputs that satisfy both per-parameter constraints and dependencies among parameters. Existing approaches either leave such constraints implicit in generated programs or rely on library-specific parsing rules.”

Bin Duan, Tarek Mahmud, Meiru Che, Yan Yan, Naipeng Dong, Dan Dongseong Kim, Guowei Yang

“This paper introduces VistaFuzz, a document-guided fuzzing technique that uses a locally served open-sourced LLM to extract parameter specifications from API documents and generate inputs that satisfy both parameter constraints and inter-parameter dependencies.”

Bin Duan, Tarek Mahmud, Meiru Che, Yan Yan, Naipeng Dong, Dan Dongseong Kim, Guowei Yang

“We evaluate VistaFuzz on 7,718 APIs across twelve Python libraries. Inter-parameter relationships occur in 40.1\% of tested APIs, and disabling their resolution reduces the valid generation rate on those APIs from above 95\% to 31.6\%--52.8\%.”

Bin Duan, Tarek Mahmud, Meiru Che, Yan Yan, Naipeng Dong, Dan Dongseong Kim, Guowei Yang

“VistaFuzz reports 74 issues, of which 43 have been confirmed by developers and 29 have been fixed.”

Bin Duan, Tarek Mahmud, Meiru Che, Yan Yan, Naipeng Dong, Dan Dongseong Kim, Guowei Yang