Sub-Second Model Routing: How Shadow Avoids Vendor Lock-In Across Flux, SDXL, and Runware

Sub-Second Model Routing: How Shadow Avoids Vendor Lock-In Across Flux, SDXL, and Runware

Synthetic image workloads flow from the client into three specialized GPU pools. Each pool handles a distinct task: Flux 1.1 Pro provides deep scene understanding, SDXL performs global style transfer, and Runware Fast-Flux ensures high throughput. The sequence falters when one pool experiences a bottleneck; a single stalled API call can push total latency beyond 500 milliseconds, potentially breaking quality-of-experience guarantees.

To mitigate this issue, a dynamic routing policy with a 120 millisecond threshold is implemented, ensuring render uptime exceeds 99 percent while adhering to cost tiers.

The routing engine converts a render request into a model graph, optimizing for the lowest cost-latency combination that meets the Service Level Agreement (SLA) of 120 milliseconds or less. This process occurs in TypeScript using a functional pipeline. The function `route` accepts a `FeatureSet` object containing boolean values for properties such as grain, bokeh, and rim.

Depending on the feature set, the function selects an appropriate model. For example, if the `rim` property is set to true, the Runware model is chosen due to its shortest rim-lighting pipeline. The function then sorts the remaining candidate models based on a weighted combination of latency and cost, returning the model with the lowest combined value.

Each GPU pool is equipped with a circuit breaker to handle potential failures. When the breaker detects HTTP status codes 429 or 503, it records a dead-time window of 150 milliseconds. After this period, failed traffic is redirected to the next fastest pool. The circuit breaker class includes methods for making async calls and managing timeouts.

After the model returns a raw tensor, optical effects are applied to enhance the final image. These effects include adding film grain, blurring the image to create a shallow depth-of-field, and implementing rim lighting. The effects are applied conditionally based on the properties specified in the `FeatureSet` object.