I built 100 compliance screening pages with one command (OFAC, PEP, entity)

This article details the process of building 100 compliance screening pages using a single command, which includes screening against OFAC, EU, UK, UN sanctions lists, PEP screening, and entity verification. The system is built using a Cloudflare Worker, a JSON list, and a generator script to produce 100 SEO pages, a landing page, and a Suby product. The live version of this system can be accessed at https://jsonexlab.com/compliance.

The main functionality of this system is to screen any person or company against various sanctions lists and PEPs. It checks against OFAC SDN (US Treasury, 19,416 entries), OFAC Consolidated (non-SDN), UK OFSI (UK sanctions), EU Consolidated (EU sanctions), UN Security Council (UN sanctions), and PEP (Politically Exposed Persons). Entity verification is also included, using company registries from various countries.

The article highlights three significant challenges encountered during the development of this system:

1. Cloudflare Workers cannot directly fetch OFAC data due to SSL handshake errors or missing User-Agent. The solution was to move the ingestion process to GitHub Actions, which has a larger RAM capacity (6 GB) and a normal TLS stack.

2. The XML file (30 MB) cannot fit within the 128 MB Worker RAM. To address this issue, GitHub Actions downloads the XML file, parses it in Node.js, and inserts the data directly into Cloudflare D1 via the D1 HTTP API.

3. The D1 free tier allows only 100,000 row writes per day. A single OFAC ingestion requires ~39,000 writes, which would exceed the quota if performed twice in a day. To resolve this, the ingestion process is scheduled to run once per day at 01:00 UTC, ensuring that the quota resets and is not hit.

The architecture of this system involves GitHub Actions for scheduling and automating tasks, Node.js for parsing and processing data, and Cloudflare D1 for storing the data. The system is composed of several components, including a GitHub Actions workflow, a Node.js script for OFAC ingestion, a Cloudflare Worker for handling requests, and a Cloudflare Pages site for displaying the 100 SEO pages.