Urgent.News

What's breaking now, across thousands of outlets.

Tech

scrapy-playwright

scrapy-playwright 是 Scrapy 官方插件组织维护的下载处理器,用 Playwright for Python 驱动真实浏览器来抓取请求。它的定位是处理需要执行 JavaScript 的页面,同时保留 Scrapy 原有的调度和 item 处理流程。和前面聊的 curl_cffi 方案相比,它不是"模拟"浏览器指纹,而是直接用真浏览器,所以 TLS、HTTP/2、JS 环境天然一致,代价是慢得多、吃内存。 安装与启用 pip install scrapy-playwright playwright install chromium # settings.py DOWNLOAD_HANDLERS = { "http" : "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler" , "https" :…

scrapy-playwright is an official Scrapy plugin maintained by the Scrapy organization, which uses Playwright for Python to drive a real browser for request fetching. Its purpose is to handle pages that require JavaScript execution, while preserving Scrapy's existing scheduling and item processing flow. Compared to the previous curl_cffi solution, scrapy-playwright does not simulate browser fingerprints but instead uses actual browsers, resulting in slower speeds and higher memory usage.

To install and enable scrapy-playwright, use the following command: pip install scrapy-playwright playwright install chromium. In your settings.py file, update the DOWNLOAD_HANDLERS as follows:

DOWNLOAD_HANDLERS = {

'http' : scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler,

'https' : scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler,

}

Set the TWISTED_REACTOR to twisted.internet.asyncioreactor.AsyncioSelectorReactor.

Basic usage involves setting the playwright key in Request.meta for requests that should be handled by the browser. Other requests will still be processed by the regular Scrapy downloader. This allows for a mixed approach, with browser requests for listing pages and normal requests for detail interfaces.

PageMethod is a Playwright page method that can be executed after the page loads and before the response is returned. Examples include waiting for elements, clicking, scrolling, taking screenshots, etc. The response is the rendered HTML, and parsing remains unchanged. If more complex interactions are needed, set playwright_include_page to True and access the page object from response.meta['playwright_page'] in the callback.

Remember to close the page with await page.close() and include an errback to close the page, otherwise memory leaks can cause the concurrent process to hang.

Common configurations include PLAYWRIGHT_BROWSER_TYPE (chromium, firefox, webkit), PLAYWRIGHT_LAUNCH_OPTIONS (headless, channel), PLAYWRIGHT_CONTEXTS (predefined browser contexts with separate cookies, proxies, user agents, viewport), PLAYWRIGHT_MAX_CONTEXTS and PLAYWRIGHT_MAX_PAGES_PER_CONTEXT (limits on context and concurrent pages), PLAYWRIGHT_ABORT_REQUEST (intercepts images, fonts, tracking scripts to save bandwidth), PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT (navigation timeout in milliseconds), and PLAYWRIGHT_CDP_URL (connects to an existing remote browser instead of launching a local one).

Proxy settings can be configured in the launch options or context meta keys, allowing for isolation of each proxy/account into a separate context. These settings should be referenced from the official documentation's Settings page for accuracy.

The installation of scrapy-playwright is resource-intensive, so it's recommended to lower the CONCURRENT_REQUESTS from a regular Scrapy crawler. Typically, a few dozen concurrent pages per node are sufficient. Note that scrapy-playwright shares some limitations with scrapy-redis, including the requirement for serializable requests, possible issues with PageMethod objects when pickled, and the need for simple values like playwright: True and context names to be properly serialized.

While scrapy-playwright is not a reflection of anti-detection measures, it does not solve the issue of automated features such as navigator.webdriver or HeadlessChrome UA. Common mitigations include using the chrome channel to start a regular version of Chrome, using head mode, or connecting via CDP to a fingerprinted browser.

Resource consumption is significant, and the speed is significantly slower than curl_cffi, requiring attention to memory usage during long-term operation. Support for Windows is limited due to historical event loop issues, and Linux is recommended for production environments.

In terms of selection, scrapy-playwright is not suitable when a simple interface can be achieved using curl_cffi. A practical combination involves using scrapy-playwright to handle JavaScript challenges, obtaining cookies or tokens, and then delegating subsequent requests to scrapy-impersonate, ensuring that both use the same browser version and proxy IPs.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

Every Rule We Added Had a Bug. So We Stopped Adding Rules.

We wrote a handful of small rules on November 12. Each one was a line or two. Each one was obviously right. A read-only review found a real bug in nearly every one of them before they shipped.

  • Every rule added to Since tool had a bug.
  • Rules were used for questions beyond code review scope.
  • Team stopped adding rules, relying on code analysis instead.

Scope Was Always a Test-Design Question

Originally published at tddbuddy.com . Related reading: Agent PRs Do Not Converge Under Design Review argued the current-PR review turn is the wrong artifact for an agent author.

  • Agentic PRs revert due to scope creep in 2.66% of cases
  • Agents rewrite code without considering broader impact
  • Scope assertions can prevent inadvertent modifications

Fixing the Daily Content Generator’s Bluesky Payload – A Code‑First Walkthrough

Fixing the Daily Content Generator’s Bluesky Payload – A Code‑First Walkthrough TL;DR: I rewrote the payload builder for the daily post generator so it correctly writes bluesky_es.json and updates…

  • CI pipeline skipped git add and commit steps due to payload file location issue
  • Utility created to manage locales for correct file name generation
  • Metadata file updated atomically to record published status and URIs

More from Monday 16 November →