ML.NET inference pipelines in .NET Aspire with Redis caching

In .NET Aspire, ML.NET inference pipelines can achieve under 50 ms p99 latency with 10 k RPS by optimizing various components. The classic 300 ms stall is often caused by plumbing issues instead of the model itself. Breaking down the latency in production pipelines, 5 ms is spent on request parsing, 10 ms on message queue latency, 20 ms on Redis lookup, 15 ms on model load, and 30 ms on inference. To stay under 50 ms, there's limited margin for error.

Consider a fraud-detection microservice that enriches payment events with user context and returns a risk score. Running in Azure Container Apps, it uses a local sentiment model and publishes "risk-completed" events. During a flash-sale on a 50-million-user platform, the service experienced a 70 ms p99 latency spike, causing a 4% drop in checkout completion.

When choosing between local ONNX and Azure OpenAI, local inference provides deterministic cost but is limited by model size and GPU memory. Azure OpenAI can handle large token contexts and adaptive prompting but is more expensive and introduces a 50 ms network hop. The decision is to keep a small, frequently-used model locally and route only the tail of the distribution to Azure OpenAI.

Redis KV-cache and in-memory cache both have trade-offs. Redis enables cross-instance state sharing essential for horizontal scaling, while in-memory cache eliminates network hops but requires sticky routing and complicates failover. The decision is to use Redis for KV-cache and a small per-process in-memory LRU for hot keys.

GRPC streaming and HTTP polling have their advantages and disadvantages. GRPC reduces round-trip latency and allows streaming large ONNX blobs, while HTTP is easier to debug but adds ~5 ms per request. The decision is to use gRPC for production with fallback to HTTP in development. Batching and per-request inference also have trade-offs.

Batching reduces GPU kernel launch overhead but can increase per-request latency, while per-request guarantees O(1) latency but wastes GPU cycles. The decision is to use a 5 ms sliding window and fallback to single inference if the batch size exceeds 8.

Event-driven scaling with Service Bus and KEDA can handle traffic spikes with sub-second queuing time but introduces message latency and requires careful idempotency. Static scaling is simpler but cannot handle flash sales. The decision is to use Service Bus with KEDA on queue depth and maintain a minimum of 2 workers to absorb burst traffic.

Local ONNX and Azure OpenAI have different trade-offs. Local ONNX is cost-effective and has low operational overhead, while Azure OpenAI offers unlimited context length but is more expensive and has high operational overhead. The decision is to choose Local ONNX + Redis KV-cache when latency and cost control are priorities and fall back to Azure OpenAI for specific requests.

Avoiding common pitfalls is crucial. Examples include KV-cache eviction, inferenceSession leaks, too wide batch windows, and service bus partition exhaustion. Following the recommended approach, engineers can implement a Model Loader Service that watches for new ONNX blobs, uses Redis KV-cache with LZ4 compression, a batch collector with a sliding window, and KEDA scaling based on queue depth. This pattern has proven successful in production rollouts to 100k concurrent users.