Should Your Prompt Store Pick Your Model

Langfuse with Microsoft.Extensions.AI enables updating prompts without redeploying by fetching its config blob—model, tokens, temperature—which the code passes directly to the LLM. This approach is useful for prompt iteration, as users can make changes in the Langfuse UI and deploy them without code modifications or redeployment. However, the current setup puts a boundary on what could be better placed elsewhere, and moving it offers potential benefits.

The current setup couples prompt config with model selection, which can be problematic. The prompt config is an optional JSON object versioned alongside the prompt, allowing for easy deployment changes through the UI. While this provides a seamless experience for non-engineers, it introduces risks. The config is an arbitrary JSON without schema enforcement, and a missing key, stray max_tokens, or typo like "gpt4o" may not fail during build or deploy but could cause issues during live requests.

Additionally, prompt wording and model selection have different risk profiles, with prompt iteration being low risk and model selection being critical for infrastructure and cost considerations.

To address these concerns, a cleaner solution involves separating prompt configuration from infrastructure decisions. By using Microsoft.Extensions.AI, which binds the model identity when constructing the client, the app can leverage dependency injection to select the appropriate client based on pre-built providers. This approach allows for a typed, environment-aware registry mapping prompt aliases to concrete clients, with guardrails and validation at startup.

This ensures that model selection is handled separately from prompt configuration, reducing the risk of unintended behavior and providing a more robust and controlled system.