{
  "id": 3248239,
  "title": "How Hugging Face Inference Endpoints, Jobs, and Buckets Power Search on Papers with Code",
  "url": "https://urgent.news/2026/08/21/how-hugging-face-inference-endpoints-jobs-and-buckets-power-search-on",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-21T00:00:00.000Z",
  "source": {
    "name": "Hugging Face",
    "slug": "hugging-face",
    "url": "https://huggingface.co/blog/pwc-search"
  },
  "original_language": "en",
  "account": "Three months ago, Papers with Code underwent a revival (see the announcement tweet) to make open AI research more accessible, allowing users to discover artifacts related to papers, identify state-of-the-art models across AI domains, and share research. The system's powerful search engine enables users to find relevant work quickly through the website or pwc search CLI command. Unlike regular text searches, Papers with Code's search must handle complex queries and tolerate typos, all while delivering results swiftly, even if a model service is cold or temporarily unavailable. The search system employs a hybrid approach, combining keyword search for exact matches and vector search for semantically similar terms, and further enhanced with rerankers for improved results. To support dense embeddings, the system utilizes pgvector and the RRF algorithm to combine keyword and vector search results. The system maintains dense embeddings for over 110,000 papers sourced from arXiv and Daily Papers. In production, embeddings are generated using a dedicated Hugging Face Inference Endpoint behind the scenes. If the endpoint is unavailable, the system falls back to full-text retrieval. This separation ensures the search system remains both powerful and fast. The embedding pipelines are treated as a versioned API, with each paper encoded as a unique identifier. The Qwen/Qwen3-Embedding-0.6B model, pinned to a specific revision, is used for production generation. The embedding process follows a sequential pipeline, starting with batch generation on a GPU using Hugging Face Jobs. The corpus build is a large batch workload that runs on an NVIDIA L4 GPU, encoding about 75 papers per second at 1024 dimensions. The generated vectors are stored in a mutable Storage Bucket, which serves as a boundary between three systems with distinct lifecycles, ensuring immutability and data integrity for a seamless user experience.",
  "summary": null,
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}