BigCodeBench

About

A programming benchmark from the BigCode project that scores models on 1,140 tasks requiring compositional use of many libraries and function calls, rather than the self-contained puzzles earlier code benchmarks used. It runs in two modes: Complete, which tests completion from a docstring, and Instruct, which supplies only a natural-language instruction and tests whether the model turns intent into working code. Models rank by pass@1 with greedy decoding, and a roughly 150-task hard subset targets tougher real-world scenarios.