What My FastAPI CSV Upload Service Got Wrong: Four Bugs the Test Suite Never Caught

The FastAPI CSV upload service had four bugs in its upload path that the test suite failed to catch. The first issue was a size limit check that incorrectly raised a 500 Internal Server Error instead of the expected 413 Request Entity Too Large response. This was due to the use of a non-existent status constant in Starlette's status module.

To fix this, the service now raises a custom ClientError with the correct HTTP_413_REQUEST_ENTITY_TOO_LARGE status code, which is handled by a dedicated exception handler and converted into the API's standard JSON error response.

The second problem stemmed from the lack of a policy for handling encodings and binary files. The service assumed that any file with a .csv extension could be treated as UTF-8 text, leading to decoding errors and a 500 response when encountering binary files. The fix involved implementing a decode_bytes() function that first rejects files containing NUL bytes, then tries UTF-8 decoding (with or without a BOM), and finally falls back to Latin-1 if UTF-8 fails. This provides a clear decoding policy and returns a 400 error for binary files.

The third issue was the size check being executed after the entire file had been read into memory. This meant that oversized files could consume memory before being rejected, potentially causing performance issues. The solution was to introduce two separate checks: one using a middleware to inspect the Content-Length header before the request reaches the endpoint, and another to verify the actual file size once it's read. This ensures the size limit is enforced more efficiently and prevents excessive memory usage.

Lastly, the service had a CORS configuration that was more permissive than initially intended, allowing potentially unsafe requests. The fix involved tightening the CORS settings to ensure secure communication between the client and the server. By addressing these four issues, the updated FastAPI CSV upload service now has a more robust error handling system, a clear decoding policy, efficient size checks, and secure CORS configuration.