{
  "id": 3674084,
  "title": "Building a Reliable AI Transcription Pipeline on Cloudflare Workers",
  "url": "https://urgent.news/2026/08/27/building-a-reliable-ai-transcription-pipeline-on-cloudflare-workers",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-27T05:26:21.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/bill_king_d4cd78085ee37d2/building-a-reliable-ai-transcription-pipeline-on-cloudflare-workers-5hie"
  },
  "original_language": "en",
  "account": "The post outlines the architecture patterns employed to create a dependable AI transcription pipeline on Cloudflare Workers. Instead of attempting to complete the transcription directly in the HTTP request, the system validates the request, persists the intent, and establishes a durable handoff. The pipeline employs explicit job states to manage task progress, ranging from awaiting media to completed or failed.\n\nTo handle large media files, the system adopts a durable handoff approach. This involves creating a task record, uploading the media to private object storage, and placing a small message on the queue containing the task ID and object key. The worker retrieves the object when capacity is available, and the object is either deleted or retained based on an explicit retention policy. By separating the media from the actual transcription, the system ensures the web process remains responsive while the backend independently reports the progress of preparation and transcription.\n\nEvery callback is designed to mitigate the risk of duplicate delivery. Retries are common in distributed systems, and callbacks may arrive multiple times due to various reasons such as worker timeouts, queue retries, or repeated webhooks. To address this, the callback handler must be idempotent. The handler verifies the signature, retrieves the task from the database, and checks if the status is already completed. If so, it returns the existing result. Otherwise, it updates the task in a database transaction, settles any billing, and returns the updated task. This approach ensures that the system does not charge twice or create duplicate outputs in case of a duplicate callback.\n\nBatches, rather than being viewed as larger loops, are treated as separate entities with individual ownership and aggregate status. Each batch record tracks the ownership and overall status of the batch, while each item record maintains an independent state and error information for every source. This approach allows for a more nuanced handling of partially completed batches, improving fairness by enabling the scheduler to distribute work across batches instead of blocking all single-item requests.\n\nA crucial aspect of the pipeline is preserving word timing as the primary data. Instead of storing only a large text blob, the pipeline maintains a normalized word timeline. This timeline consists of objects with text, start time, end time, and speaker ID (if applicable). From this timeline, the application can derive various views, such as readable paragraphs, speaker-aware sections, standard subtitle cues, shorter social caption cues, and search matches for TXT, document, and timed subtitle exports. This approach ensures that the user can switch between different output formats without incurring additional transcription costs, and it maintains consistency across all views by keeping the data aligned with the source timeline.\n\nIn the event of failures, the system adopts a product-centric approach to error handling. Internal errors may indicate specific issues such as decoder failures, exhausted queue retries, or rejected media files. These details are valuable for internal logging but can be harmful to the user experience if exposed directly. Instead, the system should present stable, actionable categories such as unsupported source, inaccessible or private media, no audio track, file too large, transcription failed, or processing timeout. This separation allows the system to change providers without altering the user-facing error language, ensuring a consistent and user-friendly experience.",
  "summary": "I recently shipped HiTranscript , a web app that turns public video URLs and local media uploads into searchable transcripts and subtitle files. The transcription model was not the hardest part. The hard part was building a pipeline that stays correct when uploads are large, requests are retried, callbacks arrive twice, a batch partially fails, or a deployment needs to be rolled back. This post…",
  "key_points": [
    "System validates request, persists intent, establishes durable handoff",
    "Handles large media files via durable handoff approach",
    "Idempotent callbacks mitigate risk of duplicate delivery"
  ],
  "editors_take": "This development allows for a more reliable and user-friendly AI transcription service by decoupling media processing from the web process, ensuring responsiveness, and providing a consistent experience across output formats.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}