Building a Reliable FMCSA Carrier Data Scraper with Python and Playwright
TL;DR: Public FMCSA carrier records are useful for logistics research, but a fragile script will get you blocked or produce dirty data. This guide walks through a production-minded Playwright scraper: respectful rate limits, retries with backoff, MC/USDOT hygiene, and clear compliance boundaries—without inventing shortcuts past access controls. Why FMCSA data, and why Playwright? U.S. freight and…
Publicly available FMCSA carrier records are a valuable resource for logistics research, but they require a robust scraper to avoid being blocked or receiving inaccurate data. This guide outlines a production-ready Playwright scraper that follows best practices such as respecting rate limits, implementing retries with exponential backoff, maintaining carrier and USDOT hygiene, and adhering to compliance boundaries.
FMCSA data is useful for logistics teams seeking information on carriers, including their status, legal name, address, and equipment details. The challenge lies in maintaining a stable collector amidst changing pages, network issues, and ensuring clean, deduplicated data for CRM or directory integration. Playwright is an ideal choice because it uses modern Chromium (and Firefox/WebKit) with a stable API, offers predictable waits for network and DOM loading, and provides request interception, tracing, and session storage.
The article focuses on authorized public-page collection, not including login bypasses, CAPTCHAs, paywalls, or private APIs. Always obtain proper permission when a site's terms require it. A stable scraper typically consists of four layers: seed list, fetcher, normalizer, and sink. The seed list contains MC/USDOT numbers or search queries, while the fetcher uses Playwright (or HTTP for stable APIs).
The normalizer parses, types, cleans, and deduplicates data, and the sink handles CSV, SQLite, or Postgres storage, along with a run log documenting successes, failures, and reasons.
To maintain privacy and professionalism, keep secrets out of the repository. Store rate limits and user-agent policies in configuration files rather than hard-coding them. Configure the following in `config.py`:
```python
RATE_LIMIT_SECONDS = 2.5
MAX_RETRIES = 4
BACKOFF_BASE = 2.0
REQUEST_TIMEOUT_MS = 45000
USER_AGENT = (
"INDUSResearchBot/1.0 (+https://www.induswebagency.com; research; contact=induswebagency@gmail.com)"
)
```
When fetching carrier pages, employ Playwright's auto-waiting locators instead of fixed `time.sleep` calls. Incorporate a small jitter in the `polite_pause` function to avoid perfectly periodic traffic. Load Playwright's `sync_playwright` module and catch `PlaywrightTimeout` exceptions. Use `main`, `carrier-details`, or `#carrier-details` selectors to wait for stable carrier record blocks on the page.
Implement retries using the Tenacity library to handle transient errors and flaky networks. Create a `TransientFetchError` exception and apply retry logic with exponential backoff between 2 and 60 seconds. Detect soft error banners in the HTML content when a site is temporarily unavailable. Handle `PlaywrightTimeout` exceptions during retries and log every attempt, including the URL, attempt number, exception type, and elapsed time in milliseconds.
Normalize MC and USDOT codes before storing them to ensure consistency and avoid errors. Use regular expressions to match and format MC and USDOT numbers, stripping leading zeros via `int()`. By following these guidelines, you can build a reliable FMCSA carrier data scraper using Python and Playwright that complies with access controls and produces clean, usable data for logistics research.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
