← All papers
First page of Eliciting Specifications: When Does Interactive Clarification Help?

Eliciting Specifications: When Does Interactive Clarification Help?

Mowafak Allaham, Aditya Vempaty, Ashish Jagmohan, Prasenjit Dey, Ravi Kokku

cs.SE Oct 5, 2026 · v1 cs.AI cs.CL
LLM agents clarify task descriptions before formalizing them into Lean 4 pre/postcondition specifications, evaluated on the Verina benchmark with Lean compilation and soundness/completeness checks.
LLMs can generate Lean specifications from task descriptions, but those descriptions are often underspecified or ambiguous and may omit verifier-relevant edge cases, yielding imperfect formal specifications. We study whether an elicitation agent (EA) can improve verifier-checked Lean specifications by seeking clarification from a reference-grounded simulated developer before formalization. The agent decomposes the task, identifies unresolved decisions, asks bounded clarification questions, and uses the answers to write a refined natural-language specification that an LLM formalizer then translates into Lean. On 189 Verina Lean 4 specification-generation tasks, we compare single-shot generation with elicitation under question budgets of k=3 and k=10, allowing at most three or ten EA questions per task, across same-model and cross-model configurations. The same-model Claude Sonnet 4.6 pipeline has negative strict pass@1 estimates at both budgets, showing elicitation is not a uniform accuracy booster. In contrast, the Sonnet 4.6-to-GPT-5.6 and Sonnet 4.6-to-Gemini 3.8 Flash pipelines have positive strict estimates. The largest strict result reaches 65.6%, a +6.3-point change (+12 tasks), in the Sonnet 4.6-to-GPT-5.6 pipeline. Six of eight conditions have positive compile-conditional estimates, with a maximum cross-model difference of +4.9 points among compiling outputs. Treatment estimates also differ across the basic and advanced splits; the largest advanced-task change occurs in the Sonnet 4.6-to-GPT-5.6 configuration at k=10, from 42.0% to 50.6%. Failure analysis identifies postconditions as the largest source of degradation. These findings suggest future autoformalization systems should treat elicited natural-language specifications as intermediate artifacts and optimize jointly for semantic correctness among compiling outputs and reliable translation of clarified information into Lean.

Task descriptions given to LLMs for Lean specification generation are often underspecified or ambiguous. The resulting Lean pre/postconditions can be imperfect because the model must silently guess at edge cases and representation choices.

An elicitation agent decomposes each task and identifies unresolved decisions. It then asks a reference-grounded simulated developer at most k=3 or k=10 clarification questions and writes a refined natural-language specification. A downstream LLM formalizer translates that refined specification into Lean 4. Single-shot generation is compared with elicitation on all 189 Verina SpecGen tasks, across same-model and cross-model pipelines, using strict pass@1 (compiles and passes soundness/completeness checks).

Effects depend on the configuration. The same-model Sonnet 4.6 pipeline loses accuracy, while the cross-model pipelines to GPT-5.6 and Gemini 3.8 Flash gain up to +6.3 points (reaching 65.6%). Postconditions are the largest source of degradation.

ConfigurationBaselinek=3k=10
Sonnet 4.6 same-model60.3%57.7%56.1%
Sonnet 4.6 → GPT-5.659.3%65.6%65.6%
Sonnet 4.6 → Gemini 3.8 Flash59.8%64.6%64.6%
Sonnet 4.6 → Claude Sonnet 563.5%64.0%60.3%
Strict pass@1 on Verina SpecGen (n=189)