How We Built a Zero-Cost Unchanged Run SEC Enforcement API
SEC.gov publishes two feeds that matter a lot to anyone tracking enforcement risk: litigation releases and administrative proceedings. Neither has an API. Both are static web pages you're expected to refresh by hand, and neither one tells you the actual EDGAR-registered company behind the respondent's name — just whatever name the release itself used. This post walks through how I built a small,…
SEC.gov publishes two feeds crucial for tracking enforcement risk: litigation releases and administrative proceedings. Neither of these feeds has an API, and both are static web pages that require manual refreshing. Furthermore, the feeds do not reveal the actual EDGAR-registered company behind the respondent's name – they only use the name provided in the release.
The author explains how they built a small API actor that transforms these feeds into a structured, delta-tracked stream. Three key design decisions impacted the outcome: fingerprint-based delta tracking, automatic EDGAR CIK resolution, and honest failure classification.
The first design decision is fingerprint-based delta tracking. Each record, whether a release or proceeding, undergoes canonicalization (consistent field ordering, whitespace normalization, date formatting) and hashing. This hash is compared against a persisted state store from the previous run. If the hash matches, the record is dropped before reaching the billing layer.
This approach ensures that re-scraping is wasteful, and only genuinely new records are billed at $0.05 each, resulting in a zero-cost run when all records are unchanged.
The second design decision is automatic EDGAR CIK resolution. Each respondent's name is cross-referenced with EDGAR's filer database, including known former company names (mergers, rebrands). The match comes with a numeric confidence score, rather than a blind first-result match. This allows downstream consumers to decide whether to trust the match or flag it for manual review.
The third design decision is honest failure classification. This concept originated from a similar project. An outage at the UK's Health and Safety Executive register was detected, which would have surfaced as a generic "run failed" error otherwise. To address this issue, the actor classifies network-level failures as upstream outages, distinct from internal parsing or logic errors. This distinction is crucial for downstream users who need to decide whether to wait for the issue to resolve or report it as a bug.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
