My universal document converter learned that 'auto-detect' is a lie when the file extension lies

The universal document converter initially relied on auto-detect based on file extensions. However, this approach proved flawed when a PDF named report.docx was received. The initial sniffing logic, which checked the file extension first, led to the processing of binary noise from a PDF header instead of the intended Markdown output.

To rectify this, the converter stopped trusting file names and now relies on detecting magic bytes such as %PDF- for PDF files, PK\x03\x04 for OOXML formats like DOCX, and ID3 or \xFF\xFB for MP3 files. Extension remains a tiebreaker when magic bytes are ambiguous, which is rarely the case for the supported formats. Another mistake was underestimating the shared structure between EPUB and DOCX files, both being PK zips with XML payloads.

The converter initially rendered a Word document as "chapter 1: [Content_Types].xml," although technically accurate, this was practically useless. The third issue lies with plain text files. When magic bytes offer no clue, the default is to treat them as text/plain. This works well for .txt files but fails for files like CSVs, JSONs, or Python scripts that lack distinctive delimiters or structure in their initial bytes.

The converter now checks for these common delimiters and JSON structures before reverting to plain text. However, there exist certain types of files where it remains impossible to determine their nature without examining the entire content. The current behavior of the universal document-to-Markdown endpoint can be accessed at https://x402.freeq.one/tools/document_to_markdown.html, processing files through magic bytes first, structure sniffing second, extension as a secondary hint, and plain text as the last resort.