CausalForge: A Formally Grounded, Self-Improving Agentic Framework for Automated Research in Causal Inference

Generative AI & LLMs
Published: arXiv: 2607.22511v1
Authors

Jiyuan Tan Vasilis Syrgkanis

Abstract

Automating theoretical research is constrained not only by the generation of candidate results, but also by their reliable evaluation. A common approach is to close the research loop with a large language model (LLM) reviewer. However, such reviewers remain empirically unreliable: they may accept fabricated papers and detect them at rates close to chance (Bad Scientist, 2025). We present CausalForge, a framework for automated theoretical research in causal inference grounded in the Lean proof assistant. CausalForge combines Causalean, a foundational Lean library for causal inference containing 7,035 machine-checked declarations developed with language-model assistance under human design and review, with CausalSmith, a self-improving agentic pipeline that selects research topics, proposes results, formalizes statements, constructs proofs, and presents the resulting artifacts for human inspection. Because a machine-checked proof establishes only that a formal statement follows from its assumptions, not that the statement faithfully captures the intended scientific claim, the pipeline augments kernel verification with a statement audit that compares each formal theorem against the informal claim it is intended to express. We evaluate the system using artifacts produced by completed autonomous research runs. The source code, formal library, and run records are available at https://github.com/Jiyuan-Tan/CausalForge.

Paper Summary

Problem
The main problem addressed in this research paper is the unreliability of large language models (LLMs) as reviewers of automated research in causal inference. LLMs can generate research artifacts quickly, but they often accept fabricated papers and detect them at rates close to chance, making them an unreliable evaluator of correctness.
Key Innovation
The researchers present CausalForge, a framework for automated theoretical research in causal inference grounded in the Lean proof assistant. CausalForge combines a foundational Lean library for causal inference, Causalean, with a self-improving agentic pipeline, CausalSmith. This pipeline selects research topics, proposes results, formalizes statements, constructs proofs, and presents the resulting artifacts for human inspection. What's new and unique about CausalForge is its ability to augment kernel verification with a statement audit that compares each formal theorem against the informal claim it is intended to express.
Practical Impact
The practical impact of CausalForge is significant. By providing a reliable evaluation mechanism for automated research in causal inference, CausalForge can help scientists and researchers trust the results generated by LLMs. This can accelerate the pace of scientific discovery and reduce the burden of expert verification. Additionally, CausalForge can be applied in various fields where causal inference is used, such as economics, medicine, and social sciences.
Analogy / Intuitive Explanation
Imagine a team of researchers working together to prove a mathematical theorem. In traditional research, one researcher might propose a result, while another researcher verifies it. However, with CausalForge, the research process is automated, and the pipeline proposes results, formalizes statements, constructs proofs, and presents the resulting artifacts for human inspection. The statement audit is like a referee who checks that the formal theorem accurately reflects the intended scientific claim, ensuring that the research is reliable and trustworthy.
Paper Information
Categories:
stat.ML cs.AI cs.LG econ.EM
Published Date:

arXiv ID:

2607.22511v1

Quick Actions