LLM Lifecycle

The lifecycle of a Large Language Model (LLM) parallels the deployment life cycle of an operating system (OS) for an Infrastructure Administrator. The first step is Pretraining, which involves building the "golden image" of the base model by learning general language understanding from large datasets. This is done once by the model provider and is a costly, centralized effort that many teams later build upon. It is similar to creating a golden Windows Server image.

The second step is Fine-Tuning, where the general-purpose base model is adapted to perform well on a specific task using the team's own labeled data. This is analogous to customizing a golden image for a specific business unit, such as installing department-specific software or applying tailored Group Policy Objects (GPOs). The customization can be done in two ways: Vanilla fine-tuning, which involves retraining most of the model's weights (slow and expensive), and T-Few fine-tuning, which only touches a tiny fraction of the model's weights (fast and cheap).

This stage is akin to applying targeted config patches or hotfixes to an existing image without altering the base.

The third step is Prompt Engineering, where the model's behavior is adjusted through runtime configuration, such as instructions, examples, and formatting. This is similar to adjusting a config file, environment variable, or GPO setting at the client/session level instead of rebuilding or repatching the image. It is fast, reversible, and carries no deployment risk, making it the preferred choice before resorting to more complex solutions like Retrieval-Augmented Generation (RAG) or fine-tuning.

The fourth step is Evaluation and Iteration, where the model's output is tested using metrics like accuracy (the number of correct predictions) and loss (the severity of errors). This step involves adjusting hyperparameters and retraining the model if necessary, similar to UAT/pilot testing and ongoing benchmarking before a full production rollout. The goal is to ensure the model meets defined key performance indicators (KPIs) and performs optimally in real-world scenarios.

The final step is Deployment and Ongoing Maintenance, which involves deploying the model for real inference traffic and monitoring its performance. If the model's performance degrades or new requirements arise, the cycle can be repeated by fine-tuning the model further. The lifecycle of AI clusters is also an important aspect of this process.

Fine-tuning clusters require more GPU horsepower than hosting clusters, similar to how a build/compile server needs more compute resources than a server solely serving traffic. Multiple custom model endpoints can share the same GPU cluster, much like several VMs sharing a hypervisor host's resources or multiple App Pools sharing an IIS server.

GPU memory reload overhead occurs when switching between unrelated models, akin to the vMotion or cold-migration overhead when moving between things that share the same base but become costly when switching to something fundamentally different.