Urgent.News

What's breaking now, across thousands of outlets.

Tech

Hybrid search for RAG: BM25 plus vectors, merged with reciprocal rank fusion

Hybrid search runs two retrievers on every query, BM25 keyword search and vector (embedding) search, and merges their results. It fixes the best-known weakness of embeddings: they capture meaning but blur exact tokens, so a query for invoice INV-20931 , error ECONNRESET or a person's name can return documents that are merely about invoices, network errors or people. The usual way to merge the two…

Hybrid search combines two retrievers: BM25 keyword search and vector (embedding) search, merging their results to address the weaknesses of each method. BM25 excels at capturing meaning but struggles with exact tokens, while embeddings excel at exact terms but lose context. Reciprocal rank fusion (RRF) merges the rankings by using ranks instead of scores, adding 1 / (k + rank) for each document in the two lists and summing the results.

The constant k is typically set to 60, requiring no score normalization or training. The hybrid search implementation is provided in plain Python.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

Idempotent webhooks in Node.js: handle every event once, even when it's delivered twice

Webhook providers deliver at least once : if your endpoint is slow, returns an error, or the response is lost on the way back, they send the same event again.

  • Webhook providers deliver events at least once, requiring idempotent handling.
  • Claim event ID in database with unique constraint to prevent duplicate processing.
  • Respond quickly and use provider's event ID to maintain idempotency and clean database.

Push to Talk: the Backend Problem Is Not Audio, It Is Partial State

Push to talk looks like a small feature. Record, send, play. The backend tells a different story, and the whole thing turns on one requirement that sounds impossible the first time you hear it.

  • Push to talk functionality is more complex than it appears
  • Server must manage partial audio stream state for many messages
  • System uses shared database and sequence numbering for scalability

Hand Off Your Vector Search POC: A README and RECREATE.md for Amazon OpenSearch Service

The proof of concept worked in my account. The real test was whether the customer's team could build it again in theirs. A customer evaluating Amazon OpenSearch Service as a vector store asked for two things: proof that it could serve 300 queries per second against 4,096-dimension vectors with server-side latency under 50 milliseconds…

  • Amazon OpenSearch Service handled 300 queries per second with 4,096-dimensional vectors
  • CloudFormation template deployed OpenSearch 3.7 with r8gd.8xlarge data nodes in VPC
  • Faiss HNSW indexing with GPU acceleration achieved sub-50ms latency

More from Wednesday 18 November →