Cloud infrastructure management: the 90% nobody warns you about

The initial phase of establishing cloud infrastructure appears straightforward - clicking a console or executing a few Terraform files brings services online seamlessly. However, the critical 90% of cloud management begins thereafter, a phase that seldom garners attention until it's too late. Herein lies the essence of the matter, enumerated in the sequence of its potential complications.

Setting up cloud infrastructure constitutes but a paltry 10% of the task. You engage with a console or leverage Terraform scripts, and the setup materializes, functioning as intended; everyone is content. Yet, the true test lies in the remaining 90%, a phase that invariably unravels whether your setup remains a sound foundation or a gradual disaster, a facet seldom discussed until it's engulfed by the 3 a.m. chaos.

Therefore, allow me to elucidate this critical 90%, commencing with the order of its potential pitfalls. If an infrastructure's existence isn't codified, it simply does not exist. The distinction between teams that maintain composure and those that falter is fundamentally rooted in whether their infrastructure is codified. The expedience of manual console navigation, while seemingly advantageous on day one, is a deceptive harbinger of trouble.

Six months down the line, the rationale behind a security group's configuration remains elusive, modifications go undocumented, and resurrecting the environment transforms into an arduous excavation project. The practice of Infrastructure as Code (IaC) addresses this predicament, not for the ostensible reason it's often touted. While it facilitates repeatability, the paramount advantage lies in its ability to render your infrastructure scrutinizable and diffable.

Each alteration manifests as a pull request, enabling a forensics-like review prior to implementation, fostering peer scrutiny of potential errors, and preserving an immutable audit trail. Manual adjustments, in contrast, lack this depth of accountability. Cost, rather than a post hoc revelation, should be an integral design consideration.

The cloud bill unveils the efficacy of your architectural decisions, often revealing discrepancies too belatedly. The key insight is that cost permeates the design phase, not the post-mortem tuning phase. The selection of instance types, the employment of autoscaling, the extent of inter-regional data transfer, and the presence of dormant, unused resources all bear the seeds of future cost implications.

Ignoring these factors at the onset invariably engenders a hefty bill and exorbitant remediation efforts. Observability, not a reactive measure post-outage, but a proactive precept, is paramount. While consensus exists regarding the indispensability of monitoring, the majority invariably introduce it in the aftermath of an unexplained outage.

The prudent approach is to instill observability upfront - logging, metrics, tracing, and alerting mechanisms that judiciously notify the relevant personnel without inundating them with trivial alerts. Alert fatigue, the antithesis of effective monitoring, equates to a cacophony of noise that remains unheeded, rendering the system no more secure than one devoid of alerts altogether.

The proliferation of excessive permissions constitutes a pervasive conduit to breaches. Roles become over-extended, service accounts are granted undue administrative privileges, and credentials are generated and neglected. Each instance, seemingly innocuous, contributes to an intricate web that becomes the terrain of your subsequent incident.

The principle of least privilege transcends the realm of a one-time setup; it demands continuous vigilance. Regularly reassess the scope of permissions, scrutinize the entities wielding elevated access, rotate credentials judiciously, and pay particular heed to non-human identities - service accounts and automation scripts, which typically possess the most permissive access and the least scrutiny.

The unassuming truth surrounding cloud infrastructure management is fundamentally grounded in discipline, not innovation. Codifying your infrastructure, exercising fiscal prudence, instituting observability from the outset, and rigorously curtailing access to the bare minimum - these tenets collectively constitute the cornerstone of an infrastructure that runs itself, rather than one that incessantly besieges you.

The initial setup, while arduous, constitutes the simpler portion of the endeavor. The true test, the management, constitutes the arduous labor.