Designed and built ChainPilot, a closed-loop testing framework for enterprise AI agents, in 24 hours for the Hack-a-Claw NVIDIA Hackathon at UCSC. The system pairs a live supply chain simulation with a multi-agent pipeline powered by NVIDIA Nemotron and orchestrated via OpenClaw. A proposer agent, a critic agent, and an executor agent work together against a real backend with constrained APIs. Every decision gets evaluated on measurable downstream KPIs including service level, cost, emissions, and profit.
Most AI agent frameworks only ask whether an agent completed a task. In enterprise environments, that is not enough. An agent might call the right API but still make a decision that makes the overall system worse. Existing evaluation tools focus on reasoning and tool use, but none of them measure what actually happens after the agent acts.
Started by reading research on agentic AI topics like ReAct, Reflexion, and multi-agent critique to find the gap between task completion and decision quality. From there, designed a system where three specialized agents handle proposal, critique and revision, and constrained execution separately. Built the supply chain simulation with KPIs that intentionally conflict with each other, so agents have to reason through real tradeoffs instead of just solving a static prompt.
Keeping multi-agent state consistent across all three agent turns was tricky within a 24-hour build window. Making sure agents could not silently change simulation state required careful constraint enforcement at the API layer, so all execution had to flow through validated backend calls. Getting the real-time KPI comparison working on the dashboard, so users could see before and after impact clearly, also took several rounds of iteration under time pressure.
Delivered a fully deployed, working agent system within the 24-hour limit. The demo walked through a concrete supply chain scenario where Chicago had surplus inventory and the West Coast distribution center was at risk of stockout. The full propose, critique, and execute loop ran live, and the dashboard updated in real time showing measurable KPI changes after each agent decision.
Separating the simulator from the optimizer early on was the right call. It kept the agents honest and made the evaluation meaningful rather than circular. Defining the execution constraint model before writing agent logic also helped avoid a whole category of hard-to-debug bugs where agents were silently mutating state. Running a critic agent on top of the proposer noticeably improved decision quality, which was a good real-world validation of the core idea behind the project.