Our invoices lost their accents when we moved to a new CI system
In February we moved the build for our invoicing service from Jenkins to GitHub Actions. The pipeline was rewritten step by step, every test passed, the image deployed, and the first run of monthly invoices went out on the first of March. By lunchtime our Polish and Czech customers were sending us screenshots. Every ł, ś and ř on their invoices was an empty box. The service renders PDFs with…
In February, the invoicing service's build system was transitioned from Jenkins to GitHub Actions. The process involved rewriting the pipeline, passing all tests, and deploying the updated image. The first monthly invoices were sent out on March 1st. However, Polish and Czech customers began sending screenshots of their invoices, with accented characters replaced by empty boxes.
The service rendered PDFs using bundled fonts because the base image had limited resources. The fonts were stored in the repository using Git LFS, a small text pointer being stored in Git and the actual file on the LFS server. Jenkins had the LFS plugin enabled years ago, fetching the fonts on every checkout. In contrast, GitHub Actions did not automatically download the LFS content, resulting in only pointer files being copied into the image.
The PDF library attempted to load these pointer files, failed to parse them, logged a warning, and fell back to a built-in font that supported only Latin letters. This led to English and German invoices appearing mostly correct. The tests ran against a mocked renderer, and a single test verified the validity and page count of the generated PDFs.
The image size also decreased by nine megabytes, a metric rarely compared. To resolve the issue, the team rebuilt the image with LFS enabled, regenerated the affected invoices, and sent them again that same evening. A pipeline step then ran 'git lfs fsck' and searched the workspace for any remaining files starting with the LFS pointer header, failing the build if any were found.
The font loader refused to start the service if a bundled font couldn't be parsed, rather than falling back as before. A rendering test produced an invoice with names in Polish, Czech, Greek, and Turkish, comparing it to an approved image. The pipeline failed if the image size changed by more than five percent from the previous release without a label on the pull request explaining the change.
Moving a pipeline is a complex translation, and the old system had conveyed information we never documented. Jenkins had been fetching our fonts for six years, with the only record being a checkbox. This situation was explained by Sergey Shinder.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
