Urgent.News

What's breaking now, across thousands of outlets.

Tech

Your model bill is a routing problem

A year into using language models, most companies have the same three problems. There are keys for five providers in a dozen places, it is hard to say what last month's spend bought, and every team has quietly picked the strongest model for everything, because nobody was ever told off for an answer that was too good. The fix for all three is the same piece of plumbing: one gateway that every call…

Using language models has led most firms to face three primary challenges: managing access to multiple providers, difficulty tracking spend, and each team selecting the most suitable model without oversight. To address these issues, a single gateway can be implemented, which serves as a central point for all model calls, regardless of the source.

By routing every call through this gateway, it ensures that all requests follow the same standardized format, allowing for better control and visibility. This centralized approach enables organizations to restrict access to approved models, set spending limits and clearly document each call's details, including token counts, costs, origins, and purposes.

To optimize routing, requests can be categorized into four tiers based on complexity, from simpler tasks to more advanced reasoning and agent-based queries. Each tier is associated with specific models and fallback options. A lightweight router, which only requires minimal processing power, determines the appropriate tier for each request and has demonstrated high accuracy in handling previously unseen prompts.

Initially, the classifier was trained based on the cheapest tier that could handle a given request, leading to suboptimal results. This was corrected by redefining the labels and their meanings, which improved the classifier's performance. The router offers an additional safety net by defaulting to a higher tier when uncertain, preventing poor answers and preserving user trust.

The implementation of a gateway adds minimal latency, typically around 14.5 milliseconds for a single caller. While cost savings can be achieved by using a simpler classifier, it may result in subpar responses and negatively impact user trust. To begin, companies should start by implementing a single gateway, recording each call's cost and purpose, and analyzing the data over time.

This will help identify the most effective routing policy and provide a foundation for further optimization.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

Moodle behind TLS offload needs $CFG->sslproxy

Moodle™ sees the request PHP received. If nginx, HAProxy, an F5, or Cloudflare terminates TLS and the PHP pool speaks HTTP on localhost, $_SERVER['HTTPS'] is empty.

  • $CFG- sslproxy must be set to true for external SSL appliance
  • Enables Moodle to generate HTTPS URLs via HTTP on localhost
  • Missing flag causes 302 loops, mixed-content warnings

The accessibility mistakes I keep finding on small business websites

I built a website accessibility scanner, and the first thing I did with it was point it at a lot of small business sites. Restaurants, dental clinics, a few agencies, a couple of gyms.

  • Only six or seven accessibility issues found repeatedly across 70 small business websites
  • Issues include poor readability, missing alternative text, unclear button/link names
  • Manual checks with keyboard and screen reader essential for true accessibility

I audited the top 20 iPhone teleprompter apps: they have 30,000+ ratings, I have 6

I make a small iPhone teleprompter app and it is nowhere in search for the word "teleprompter". Before writing more pages, I wanted to see who is there and where their traffic comes from.

  • App holds first position in US for "teleprompter offline" but absent from top 20 in other countries.
  • Author's app has only six ratings compared to Norton Five's 34,900 ratings.

Self-Host Gitea with Docker Compose: Git Service Setup

This tutorial was written with AI assistance from the project's official documentation. Verification status is stated at the end of the article.

  • Gitea is self-hosted Git service with repo hosting and issue tracking.
  • Requires Linux VPS, Docker Engine 24+, and Compose plugin.
  • Set up with .env file, compose.yml file, and Docker Compose.

Is Opus Clip Legit? What the Billing Complaints Really Mean

Opus Clip turns long videos into short clips automatically, and for creators drowning in footage, that sounds like a dream.

  • Opus Clip is AI-powered video editing tool launched in 2022.
  • Billing complaints arise from automatic trial-to-paid conversion.
  • Legitimate service with known founders and reputable coverage.

More from Wednesday 18 November →