← All papers
First page of ZonoGPT: Towards An Abstract Domain for Verifying Large GPT Models

ZonoGPT: Towards An Abstract Domain for Verifying Large GPT Models

Hai Duong, Thanh Le, ThanhVu Nguyen

cs.LG Sep 28, 2026 · v2 cs.SE
Soundness theorems for the abstract transformers (LayerNorm, affine GELU, fused MLP and attention error bounds) are mechanized in Lean 4.
Transformer-based models are widely used for reasoning, coding, and multimodal agentic tasks. To provide formal assurance of desirable behaviors, such as robustness, safety, and fairness, neural network verification techniques prove required properties and provide auditable guarantees before deployment. However, prior work remains limited to small or restricted Transformers, and maintaining precision across deep models remains challenging. In this work, we introduce ZonoGpt, an abstract domain for verifying large transformers that maintains a space complexity independent of network depth. ZonoGpt uses a structured zonotope and a generator reduction mechanism to efficiently preserve correlations. To maintain precision, it introduces block-specific fused transformations for Attention and LayerNorm that retain feature relations, along with an affine transform for GELU that preserves generator relations. These mechanisms enable ZonoGpt to be the first approach to verify standard architectures, scaling to official HuggingFace models up to GPT-2 Medium (24 blocks, 300M+ parameters) and successfully verifying 1,339 instances across text and vision tasks.

Neural network verification has been limited to small or simplified Transformers. Precision degrades across deep models, so existing tools do not scale to standard architectures such as GPT-2.

ZonoGpt is an abstract domain built on structured zonotopes. Shared generators capture cross-token relations, local generators capture token-specific relations, and generator reduction keeps memory independent of network depth. Fused transformations for Attention and LayerNorm, based on residual Taylor expansion, and an affine GELU transform preserve correlations. The soundness theorems for these transformations are mechanized in Lean 4 (e.g., layerNorm_image_subset_lnZN in ZonoGpt/LayerNorm.lean).

ZonoGpt verifies 1,339 of 2,880 instances on GPT-2 Small and Medium classifiers (up to 24 blocks, 300M+ parameters) for MNIST and SST. IBP, CoVeNN and alpha-beta-CROWN verify none: alpha-beta-CROWN and DeepZ run out of memory, while IBP and CoVeNN complete but solve no instances.

MethodSolvedTime (s)
IBP0688.8
CoVeNN0569409.8
ZonoGpt-F112083479.4
ZonoGpt-A1339157325.4
Total verified instances across all benchmarks (out of 2880)