# scrapy-playwright and Cloudflare Turnstile: JS-Rendered Forms > When the Cloudflare Turnstile widget only exists after JavaScript runs, let scrapy-playwright render the page, then solve the token, fill the form and parse. - Source: https://zerocaptcha.io/blog/scrapy-playwright-cloudflare-turnstile - Published: 2026-10-01 - Author: ZeroCaptcha Engineering Use **scrapy-playwright** for a Cloudflare Turnstile form when the widget only exists after JavaScript runs: its sitekey is not in the HTML Scrapy downloads, or the page reads the token through a JavaScript callback rather than the form field. Request the page with `meta={"playwright": True, "playwright_include_page": True}`, wait for the widget in the Playwright page, read its sitekey, get a token from a solving API, write it into `cf-turnstile-response`, call the page's callback if it has one, submit, and close the page. If the sitekey is already in the server's HTML, [plain Scrapy](https://zerocaptcha.io/blog/scrapy-cloudflare-turnstile) does the same job without a browser. This tutorial shows the spider, the settings it needs, and how to keep browser pages from piling up. Crawl only sites you are allowed to automate: see [responsible captcha automation](https://zerocaptcha.io/guides/responsible-captcha-automation). ## Set up ```sh pip install scrapy scrapy-playwright httpx playwright install chromium export ZEROCAPTCHA_API=… # the API's base URL export ZEROCAPTCHA_KEY=… # your API key ``` scrapy-playwright needs two settings: its download handler for the schemes you crawl, and Twisted's asyncio reactor. The spider below sets both in `custom_settings`, so it runs on its own. ## The spider Save this as `search_spider.py`. It uses Scrapy's async `start()` method, added in Scrapy 2.13; on older versions, Scrapy's docs say to "define also a synchronous `start_requests()` method that returns an iterable". ```python import asyncio import os import uuid import httpx import scrapy API = os.environ["ZEROCAPTCHA_API"] KEY = os.environ["ZEROCAPTCHA_KEY"] FILL_TOKEN = """([token, callbackName]) => { for (const input of document.querySelectorAll('[name="cf-turnstile-response"]')) input.value = token; const callback = callbackName && window[callbackName]; if (typeof callback === "function") callback(token); }""" async def solve_turnstile(page_url, sitekey, action=None, cdata=None): task = {"type": "TurnstileTaskProxyless", "websiteURL": page_url, "websiteKey": sitekey, "metadata": {}} # The widget's data-action and data-cdata go in metadata, only when it sets them: many sites # check both when they verify the token. if action: task["metadata"]["action"] = action if cdata: task["metadata"]["cdata"] = cdata async with httpx.AsyncClient(base_url=API, timeout=15) as client: # One Idempotency-Key per task: a retried create with it returns the same task. created = (await client.post("/createTask", json={"clientKey": KEY, "task": task}, headers={"Idempotency-Key": str(uuid.uuid4())})).json() if created["errorId"]: raise RuntimeError(f"createTask: {created['errorCode']}") for _ in range(90): # 90 polls, 2 seconds apart: 3 minutes at most await asyncio.sleep(2) reply = await client.post("/getTaskResult", json={"clientKey": KEY, "taskId": created["taskId"]}) result = reply.json() if result["errorId"]: raise RuntimeError(f"getTaskResult: {result['errorCode']}") if result["status"] == "ready": return result["solution"]["token"] raise TimeoutError("no token within 180 seconds") class SearchSpider(scrapy.Spider): name = "search" custom_settings = { "DOWNLOAD_HANDLERS": { "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler", "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler", }, "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor", "PLAYWRIGHT_BROWSER_TYPE": "chromium", "PLAYWRIGHT_MAX_PAGES_PER_CONTEXT": 4, } async def start(self): yield scrapy.Request( "https://shop.example.com/search", meta={"playwright": True, "playwright_include_page": True}, errback=self.close_page, ) async def parse(self, response): page = response.meta["playwright_page"] try: widget = await page.wait_for_selector("form#search [data-sitekey]", timeout=15_000) token = await solve_turnstile( page.url, await widget.get_attribute("data-sitekey"), await widget.get_attribute("data-action"), await widget.get_attribute("data-cdata"), ) await page.fill("form#search input[name=q]", "running shoes") await page.evaluate(FILL_TOKEN, [token, await widget.get_attribute("data-callback")]) await page.click("form#search button[type=submit]") await page.wait_for_selector(".result", timeout=30_000) html = await page.content() finally: await page.close() for result in scrapy.Selector(text=html).css(".result"): yield { "title": result.css("h2::text").get(), "url": response.urljoin(result.css("a::attr(href)").get()), } async def close_page(self, failure): page = failure.request.meta.get("playwright_page") if page is not None: await page.close() ``` Run it with `scrapy runspider search_spider.py -O results.json`. ## How it works - **The page comes back open.** With `playwright_include_page`, the response carries the live Playwright page in `response.meta["playwright_page"]`, so the callback can wait, click and run scripts in it. The README is firm that you must close it: "Always close pages when finished", with an errback for requests that fail before your callback runs. - **Wait for the widget, not a fixed time.** `wait_for_selector` returns as soon as the widget's element exists, which on a JavaScript-rendered page may be well after the first response. - **Solve late.** The token is valid for 300 seconds and works once, so the spider asks for it right before it submits, after the page has loaded. See [Cloudflare Turnstile token expiry](https://zerocaptcha.io/guides/cloudflare-turnstile-token-expiry). - **Fill the field and call the callback.** In a real browser, the widget does both on success. Pages that pass an inline callback to `turnstile.render` need that function called instead; see [submit a Cloudflare Turnstile token](https://zerocaptcha.io/guides/submit-cloudflare-turnstile-token). - **Parse the final HTML with Scrapy.** `page.content()` gives the page after the form's results have rendered, and a `Selector` parses it as usual. ## Settings that keep it stable | Setting | Why | | --- | --- | | `PLAYWRIGHT_MAX_PAGES_PER_CONTEXT` | Caps open tabs. Each tab waiting on a token is a whole page in memory; start at 4 and raise it as your machine allows. | | `CONCURRENT_REQUESTS` | Keep it close to the page cap: queued browser requests hold nothing useful while they wait. | | `PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT` | In milliseconds. Raise it for slow sites, so navigation doesn't fail while the token is still good. | | `PLAYWRIGHT_LAUNCH_OPTIONS` | Such as `{"headless": False}` while you debug the selectors. | Only requests with `"playwright": True` in their meta go through the browser. Everything else, such as result pages that are plain HTML, still uses Scrapy's fast downloader in the same spider. ## When not to use a browser A browser costs far more per page than Scrapy's downloader. If the sitekey is in the HTML and the form posts `cf-turnstile-response`, use [Scrapy with form2request](https://zerocaptcha.io/blog/scrapy-cloudflare-turnstile) instead. If every page answers "Just a moment…" with HTTP 403, the site shows a Cloudflare challenge page, not a widget, and a token does not help: a challenge task returns the cf_clearance cookie with the user agent it is bound to, earned through your proxy; see the [Cloudflare WAF and 5-second challenge solver](https://zerocaptcha.io/cloudflare-challenge-solver). The [Cloudflare Turnstile solver for Playwright](https://zerocaptcha.io/cloudflare-turnstile-solver/playwright) page has more Playwright examples, and the [Cloudflare Turnstile solver](https://zerocaptcha.io/cloudflare-turnstile-solver) page covers every language. ## Sources - [scrapy-playwright](https://github.com/scrapy-plugins/scrapy-playwright), README (checked 1 October 2026). - [Scrapy: spiders](https://docs.scrapy.org/en/latest/topics/spiders.html), for the async `start()` method, "Added in version 2.13" (checked 1 October 2026). - [Playwright for Python: Page](https://playwright.dev/python/docs/api/class-page), for `wait_for_selector`, `evaluate` and `content` (checked 1 October 2026). - [Cloudflare challenges: supported browsers](https://developers.cloudflare.com/cloudflare-challenges/reference/supported-browsers/) (checked 1 October 2026). - [Cloudflare Turnstile: client-side rendering](https://developers.cloudflare.com/turnstile/get-started/client-side-rendering/) (checked 1 October 2026). - [Cloudflare: Error 403](https://developers.cloudflare.com/support/troubleshooting/http-status-codes/4xx-client-error/error-403/) (checked 1 October 2026). ## Questions ### When do I need scrapy-playwright for a Cloudflare Turnstile form? When the widget is rendered by JavaScript, so its sitekey is not in the HTML Scrapy downloads, or when the page reads the token through a JavaScript callback. If the sitekey is in the HTML and the form posts cf-turnstile-response, plain Scrapy is lighter. ### Why do my scrapy-playwright pages leak memory? A page requested with playwright_include_page stays open until you close it. Close it in a finally block in the callback, and in an errback for requests that fail, as the scrapy-playwright README recommends. ### Does scrapy-playwright pass Cloudflare Turnstile by itself? No. It renders pages in a Playwright browser, and Cloudflare says automated browsers are not supported for solving production challenges. Fill the widget's field with a token from a solving API instead.