Instructions that work individually can interfere when combined in a coding prompt. Researchers tested 27 combinations of output format, assigned persona, and urgency across 164 programming problems and five OpenAI models. For the GPT-4o family, combined constraints reduced first-attempt success by more than the individual effects predicted. The GPT-4.1 family was largely resistant, while structured outputForcing a model's response to match a schema, so downstream code can parse it instead of guessing at prose.Full definition → sometimes helped o3-mini. The study separated failures to follow the requested format from failures to solve the programming problem. Its results support testing the complete production prompt, because isolated prompt experiments can miss interactions that depend on the model. Study abstract.
Terms in this piece · Glossary
structured output — Forcing a model's response to match a schema, so downstream code can parse it instead of guessing at prose.
articleDo Influence Tactics Matter? Investigating Prompt Framing Effects in LLM Code GenerationAlex Deaconu, Anubhav Gupta, Manaal Basha, Nicholas Haydu, Gema Rodr\'iguez-P\'erez
articleBeyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on QualityQipeng Xie, Zi Liang, Jiafei Wu, Yufei Chen, Weizheng Wang, Wenao Ma, Zhong Ming, Haiqin Yang, Kaishun Wu
articleA Study on the Impact of Natural Language Differences in Prompts on Automatic Code Generation Using LLMsHaruka Tokumasu, Masanari Kondo, Alexander Serebrenik, Dong Wang, Kei Koyanagi, Kotaro Noguchi, Naoyasu Ubayashi, Yasutaka Kamei