Terraform : survivre à un state corrompu, guide de réparation sans tout détruire

Terraform state corruption can paralyze an entire production environment, often leading teams to the tempting but disastrous decision of "starting over" by wiping the state and re-importing everything. This article details the actual recovery maneuvers, in the order they should be attempted, starting with the most common scenario: backend storage on S3 with a DynamoDB lock.

Terraform's state file is a database, not a cache. It maintains a mapping between resource identifiers in the code and their real provider identifiers, plus the attributes read during the last refresh. Three properties make the state fragile: it's monolithic per workspace, versioned by a serial number, and protected by an external lock.

Corruption or improper handling can lead to a state that no longer reflects the real infrastructure, resulting in Terraform proposing to destroy production databases and locking the entire team.

Typical causes of state inconsistency include partial applies (some resources created but state not written), manual console modifications, module refactoring without moving, resource deletion outside Terraform, or concurrent writes. The recovery process involves several steps:

1. Freeze the situation: disable Terraform pipelines and announce a freeze on manual applies to prevent further corruption.

2. Save the current state: use `terraform state pull` to create a copy of the state without acquiring a write lock. Save it with a timestamp and note the serial and lineage numbers for later use.

3. Diagnose without writing: use `terraform plan -refresh-only -lock = false -out = drift.tfplan` to compare the real state with the current state without proposing changes. Analyze the output to identify any resource drift.

4. Inspect the DynamoDB lock: retrieve the LockID, operation, who, and created time of the lock using `aws dynamodb get-item`. Ensure the process responsible for the lock is truly dead before proceeding.

5. Force the unlock: use `terraform force-unlock -force <lock_id>` to release the lock. Be cautious, as forking an active lock can cause double writes.

6. Address the digest issue: the second DynamoDB item, often overlooked, contains an MD5 digest of the state file. If you restore an old state version through the S3 console without using `terraform state push`, you'll encounter a "state data in S3 does not have the expected content" error. Delete the digest item to allow Terraform to recalculate it during the next apply.

7. Consider using S3-native locking (Terraform 1.11+): with S3 backend storage, Terraform can set a lock via a .tflock object and If-None-Match header. Disable the DynamoDB backend and use `terraform { backend "s3" { ... use_lockfile = true } }`.