Bank statement PDFs to CSV, part 4: why pdfplumber, and the fixture bug that almost hid a real one

This is the fourth part of a series on building a converter that transforms bank statement PDFs into CSV format. The previous sections detailed the converter's functionality and process, including a graphical user interface, a Windows installer, and the handling of scanned PDFs through OCR. This post focuses on the decision-making process behind selecting a PDF library for the converter and a specific test fixture that initially obscured a real issue.

The converter utilizes a single runtime dependency: pdfplumber==0.11.10. This choice was made over PyMuPDF and pypdf for two key reasons. First, pdfplumber's line-based table extraction is particularly suited to bank statements, which typically feature structured grids with debit and credit columns. Second, pdfplumber is written in pure Python on top of pdfminer.six, resulting in a straightforward deployment with just one package to install.

While PyMuPDF remains part of the project, it is exclusively used for generating test fixtures and does not participate in the conversion process itself. This separation is crucial, as the fixture generator is where the significant bug under scrutiny emerged.

To ensure accurate testing of the converter, synthetic statements were generated and each extracted row was compared field by field with the corresponding ground truth. Initially, the converter attempted to use reportlab with an embedded NotoSansCJK font; however, the resulting extracted text displayed as null characters (\x00) for every Chinese character due to the font's encoding limitations.

A second attempt was made using PyMuPDF with the same font, which successfully extracted Chinese characters. Nonetheless, a subtle issue arose: the hyphen (-) symbol was incorrectly mapped to U+2011 (non-breaking hyphen) in some cases, and certain Chinese characters were placed on compatibility code points. This discrepancy could have led the converter to incorrectly handle U+2011 hyphens as normal, or worse, pass the tests while failing to process real PDFs with standard ASCII hyphens.

The resolution to this issue involved rendering the test fixtures in two distinct parts: Chinese characters (CJK) text using PyMuPDF's built-in china-s font, and ASCII text using Helvetica. Moreover, the conversion process now incorporates NFKC normalization on all extracted text before parsing. This normalization step converts various hyphen representations, such as U+2011, back to standard hyphens, thereby preventing any potential mismatches during the matching phase.

The test fixtures encompassed four digital layouts with 34 rows each, spanning two-decade debit/credit tables across two pages and 12 rows. The converter's performance was evaluated by comparing every field against the known ground truth. The results were highly successful, with 12/12 matches for two-column debit/credit layouts, 8/8 matches for single amount columns with plus and minus signs, 8/8 matches for reordered columns with month-day dates derived from the title, and 6/6 matches for plain text lines extracted through a fallback parser.

The scanned path, where OCR was utilized, was already validated in Part 3 of this series. The inclusion of these four layouts serves to test various code paths: primarily table extraction, with a fallback to text-line parsing when tables are not present. Furthermore, the comparison includes four distinct layout scenarios, each exercising different aspects of the converter's functionality.

It is important to note that the four layouts listed represent a subset of potential statement formats. The converter's capabilities extend beyond these scenarios, as evidenced by the successful handling of a scanned path through OCR in Part 3. However, the focus here is on the decision to use pdfplumber as the primary PDF library and the revelation of a critical issue in the test fixtures.

The converter is designed to be versatile, accommodating different column recognition methods based on header keywords, and it includes a text-line fallback parser for instances where tables are not applicable.

It is crucial to emphasize that this analysis does not claim pdfplumber as the ultimate PDF library for all document types. In fact, the converter's design intentionally favors ruled tables, a single dependency, and the absence of system binaries, which aligns with the specific requirements of bank statement processing. Nevertheless, pdfplumber's performance in this context underscores its suitability for this particular use case.

The test fixtures, while synthetic, provide a robust foundation for validating the converter's functionality, yet they do not encompass every possible bank statement variation. Banks may exhibit diverse header wording, column arrangements, summary rows, and multi-page tables, which have not been directly tested by this system.

In summary, this post illuminates the rationale behind the pdfplumber library selection and the significance of the test fixture bug that was uncovered. The normalization step proved instrumental in resolving an underlying issue that could have otherwise masked a real problem. The converter has demonstrated its capability to handle a variety of statement layouts, yet it is imperative to recognize the limitations inherent to synthetic test cases.

The full source code for the converter is available at its GitHub repository, and the presale page offers comprehensive details regarding the project's status, scope, and refund policies.