Text Generation Inference: One Title Match and a Real API-Key Boundary

Text Generation Inference (TGI) is a model-serving toolkit designed for large language models, offering generation, streaming, metrics, and model-management capabilities. When utilizing TGI, it's crucial to ensure that the API key is properly set and enforced, as this key controls access to protected routes such as streaming, metrics, and model-information endpoints.

The API key should not be relied upon for protecting generation endpoints, as separate request size, output token limits, concurrency, queue depth, and timeout controls should be implemented at the gateway and service level. It's essential to separate interactive and batch clients, ensuring that batch jobs consuming significant GPU time do not share resources with customer-facing assistants.

Logging should focus on request identifiers, model version, input and output token counts, latency, and status, while avoiding the logging of prompts, completions, system instructions, or retrieved data. Direct pod and service ports should not bypass the gateway, and verification of the direct listener should confirm that the model server port is not reachable from untrusted sources.