J’ai mis un Agent Claude dans ma CI pendant 3 mois , voici ce qu’il a vraiment fait

A reporter interviewed Claude about the experience of using an LLM agent for automating Terraform deployment deployments and the safety measures that had to be implemented. The automation idea arose from the mundane frustration of repeatedly rewriting the same code. An LLM could do this, but the question was if it could do it without supervision, in a pipeline, on an expensive infrastructure. Three months later, the answer was yes, but not in the way the reporter imagined.

The setup was simple - a $12 per month VPS with the agent's CLI installed and a Gitlab runner invoking it on a specific trigger when a Terraform plan fails on a merge request. The agent received three options: the error output, the merge request diff, and access to a read-only repository. It produced a patch proposal pushed to a dedicated branch. The approach worked well on mechanical errors like misspelled var.instance_type, missing outputs, and changed module signatures, achieving an 85% success rate.

Claude struggled with opaque AWS provider error messages, which it attempted to resolve by reading the request body from debug logs. Despite being able to read through 300 log lines in seconds, it wasn't more insightful than the reporter. On a Friday evening, the reporter's diagnostic quality collapsed, but Claude's did not. Claude proposed destroying a database, which highlighted an incident in the project. The agent had correctly proposed a technically sound plan, but it would have destroyed a live database still in use.

The lesson was not that AI is dangerous, but that it revealed existing technical debt. The discrepancy between the real infrastructure and version-controlled code existed before Claude, and he merely made it actionable and dangerous. Non-determinism is incompatible with CI - two executions with the same error do not produce the same patch.

Sometimes the variation is cosmetic, sometimes the agent chooses a structurally different approach. In a pipeline where the same input should produce the same output, this is confusing. The reporter could not resolve this issue, but circumvented it by preventing the agent from modifying the system state, instead suggesting solutions for human validation.

The costs were not as expected - around $0.40 per invocation for about 60 invocations per month, totaling $24. The real cost was the time spent reviewing proposals. A plausible but incorrect patch took longer to evaluate than no patch, as it required assessing reasoning rather than creating it. Safety measures were born out of incidents, with no initial permissions granted to write to AWS resources.

The agent produced text, never an apply, and the most it could produce was a bad suggestion. Even as permissions were restricted, the reporter resisted expanding the scope, fearing the agent would gain write access "just for this case."