Meta turns Llama3.1 into a “Reasoning X-ray Machine” with new open-source tool

Meta AI Lab has released CoT-Verifier, an open-source model now available on Hugging Face that acts like an “X-ray” for chain-of-thought (CoT) reasoning in Llama3.1—exposing exactly where and why errors occur, rather than treating reasoning as a black box.

Unlike traditional methods that only judge final answers, CoT-Verifier analyzes the attribution graph behind each reasoning step. Meta found that correct and incorrect reasoning produce dramatically different graph structures—akin to distinct circuit boards. By training a lightweight classifier on these “graph fingerprints,” the system achieves state-of-the-art accuracy in predicting faulty steps. Crucially, each task type—math, logic, or commonsense—has its own identifiable “fault signature,” proving reasoning errors follow structured, classifiable patterns, not random noise.

Even more powerful: the tool doesn’t just diagnose—it intervenes. In tests, Meta applied targeted adjustments (like ablating or reweighting suspicious nodes) based on the attribution graph, boosting Llama3.1’s MATH dataset accuracy by 4.2%—without retraining the base model. This shifts error correction from “post-mortem analysis” to real-time “intra-operative navigation.”

Developers can now use CoT-Verifier with a single script: input any CoT trace, and receive a “structural anomaly score” per step plus identification of the most likely upstream failure point.

In its paper, Meta announced plans to extend this “white-box surgery” approach to code generation and multimodal reasoning, aiming to make transparent, graph-based intervention the new standard for large language models.