Daily Dose of DevOps — Terraform remote state explained

Terraform's state is not a cache that can be easily regenerated. Instead, it serves as a controller's mapping between abstract references and actual provider objects. This map contains essential information like dependency metadata and sensitive configurations.

When multiple teams or automation tools attempt to modify the state simultaneously, it creates competing writers. Without proper coordination, each run may generate a valid plan from the same starting point and then overwrite the other's changes, leading to lost-update anomalies.

To mitigate this, Terraform Remote State can be used. By locking the backend, it approximates single-writer serializability. However, this only works if all writers strictly adhere to the locking mechanism and handle lock leases safely.

While encryption at rest is necessary, it is not enough on its own. Additional security measures should be implemented, such as using narrowly scoped identities, transport encryption, access logs, object versioning, retention controls, and having tested recovery procedures. It is crucial to remember that state often contains sensitive information, including credentials or connection material, even when marked as sensitive in the configuration.

To manage state effectively, it should be partitioned based on failure domains. A single monolithic state file can lead to increased lock contention, longer plan latency, broader privilege access, and a larger blast radius. Conversely, excessive fragmentation can result in brittle cross-state dependencies and increased coordination overhead. It is advisable to create boundaries aligned with ownership, lifecycle, privilege, and failure domains.

When exchanging data with other components or systems, use stable identifiers rather than exposing entire state snapshots. This helps maintain consistency and reduces the risk of errors during data exchange.

In case of state recovery, it is essential to follow a disciplined approach. Never attempt to repair state by manually editing JSON files under pressure. Instead, first stop all writers, preserve the current object and its versions, compare the state with the provider's reality, and use supported operations such as import, moved blocks, or state move. It is also crucial to have a backend backup that has been tested through a restoration exercise to ensure credibility in terms of recovery point and recovery time objectives.