# Scrapy and Cloudflare Turnstile: Submit Forms Without a Browser > A Scrapy spider that reads a Cloudflare Turnstile sitekey from the HTML, gets a token from an API without blocking the crawl, and posts the form. - Source: https://zerocaptcha.io/blog/scrapy-cloudflare-turnstile - Published: 2026-09-30 - Updated: 2026-10-01 - Author: ZeroCaptcha Engineering Scrapy cannot run the Cloudflare Turnstile widget, because it does not execute JavaScript, but it does not need to. The widget's sitekey is in the page's HTML, a solving API turns the sitekey into a token, and the form accepts that token in its `cf-turnstile-response` field. The one thing to get right is waiting for the token without stalling the crawl: use an `async def` callback, the asyncio reactor and an async HTTP client. This tutorial builds that spider. Only automate sites you are allowed to: your own, a client's, or one whose terms permit it. See [responsible captcha automation](https://zerocaptcha.io/guides/responsible-captcha-automation). ## What the page looks like A form protected by Cloudflare Turnstile carries the widget as a `div` with the sitekey, and the widget adds a hidden input when it runs in a browser: ```html ``` Scrapy sees the `div` but never the token. If the sitekey is not in the HTML at all, the page renders the widget from JavaScript with `turnstile.render()`: read the sitekey from the script, or use a browser crawler such as [Crawlee](https://zerocaptcha.io/blog/crawlee-cloudflare-turnstile). ## The spider It needs Python 3.10 or later. Install Scrapy, httpx and form2request (`pip install scrapy httpx form2request`), set `ZEROCAPTCHA_API` and `ZEROCAPTCHA_KEY`, and save this as `search_spider.py`: ```python import asyncio import os import uuid import httpx import scrapy from form2request import form2request API = os.environ["ZEROCAPTCHA_API"] KEY = os.environ["ZEROCAPTCHA_KEY"] async def solve_turnstile(page_url, sitekey, action=None, cdata=None): task = {"type": "TurnstileTaskProxyless", "websiteURL": page_url, "websiteKey": sitekey, "metadata": {}} # The widget's data-action and data-cdata go in metadata, only when it sets them: many sites # check both when they verify the token. if action: task["metadata"]["action"] = action if cdata: task["metadata"]["cdata"] = cdata async with httpx.AsyncClient(base_url=API, timeout=15) as client: # One Idempotency-Key per task: a retried create with it returns the same task. created = (await client.post("/createTask", json={"clientKey": KEY, "task": task}, headers={"Idempotency-Key": str(uuid.uuid4())})).json() if created["errorId"]: raise RuntimeError(f"createTask: {created['errorCode']}") for _ in range(90): # 90 polls, 2 seconds apart: 3 minutes at most await asyncio.sleep(2) reply = await client.post("/getTaskResult", json={"clientKey": KEY, "taskId": created["taskId"]}) result = reply.json() if result["errorId"]: raise RuntimeError(f"getTaskResult: {result['errorCode']}") if result["status"] == "ready": return result["solution"]["token"] raise TimeoutError("no token within 180 seconds") class SearchSpider(scrapy.Spider): name = "search" start_urls = ["https://shop.example.com/search"] custom_settings = { "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor", } async def parse(self, response): widget = response.css("form#search [data-sitekey]") if not widget: self.logger.warning("no Cloudflare Turnstile widget on %s", response.url) return token = await solve_turnstile( response.url, widget.attrib["data-sitekey"], widget.attrib.get("data-action"), widget.attrib.get("data-cdata"), ) form = form2request( response.css("form#search"), {"q": "running shoes", "cf-turnstile-response": token}, ) yield form.to_scrapy(callback=self.parse_results) def parse_results(self, response): for result in response.css(".result"): yield { "title": result.css("h2::text").get(), "url": response.urljoin(result.css("a::attr(href)").get()), } ``` Run it with `scrapy runspider search_spider.py -O results.json`. ## How it works - **The asyncio reactor.** Scrapy can run coroutine callbacks on Twisted's asyncio reactor, which lets `await` call any asyncio library, httpx included. Scrapy uses this reactor by default since 2.13; the `TWISTED_REACTOR` setting states it for older projects too. - **Only the callback waits.** While `parse` awaits the token, Scrapy keeps downloading and parsing other requests. A blocking `requests.post` in the same place would freeze the whole crawl for the length of every solve. - **`form2request`** reads the form's action, method and fields, including the empty `cf-turnstile-response` input the widget would fill, and the data you pass overrides them with your query and the token. Scrapy deprecated `FormRequest.from_response()` in favor of this library in 2.16; its docs now use `form2request` for the same job. - **Cookies carry over.** Scrapy's cookie middleware sends the session cookies the form page set, so the site sees the submission come from the same visitor that loaded the form. ## Keep the token fresh A Cloudflare Turnstile token is valid for 300 seconds and for one verification. In Scrapy, that means: - Solve inside the callback that submits the form, as above, never in `start_requests`. - Keep `CONCURRENT_REQUESTS` and your download delay such that the `FormRequest` goes out soon after the token arrives. A deep queue of pending requests can hold it past its lifetime. - If the site answers the form with an error, the token has been spent. Solve again before the retry. [Cloudflare Turnstile token expiry](https://zerocaptcha.io/guides/cloudflare-turnstile-token-expiry) explains the `timeout-or-duplicate` error the site sees otherwise. ## Scaling it Each coroutine waiting on a token holds nothing but a small object, so a spider can have dozens of solves in flight. To keep your read calls inside the API's budget, open one `httpx.AsyncClient` for the whole spider (create it in `open_spider` of an extension or in the spider's `__init__`, close it on `spider_closed`), cap solves in flight with an `asyncio.Semaphore`, and honor `Retry-After`. The [httpx tutorial](https://zerocaptcha.io/blog/python-httpx-cloudflare-turnstile) has that client, with safe retries through an `Idempotency-Key`. ## When Scrapy is the wrong tool - **The widget is rendered by JavaScript** and the sitekey only exists at runtime: use a browser crawler, or read it from the bundle. - **Every page answers "Just a moment…"** with HTTP 403 and a `cf-mitigated: challenge` header: that is a Cloudflare challenge page, not a Turnstile form. A token does not help. See [Cloudflare challenge page vs Cloudflare Turnstile](https://zerocaptcha.io/guides/cloudflare-challenge-vs-turnstile) and the [Cloudflare challenge solver](https://zerocaptcha.io/cloudflare-challenge-solver). - **The site answers error 1020 or 1015**: those are firewall and rate-limit blocks. See [Cloudflare error 1020](https://zerocaptcha.io/blog/cloudflare-error-1020) and [error 1015](https://zerocaptcha.io/blog/cloudflare-error-1015). ## Sources - [Scrapy: asyncio support](https://docs.scrapy.org/en/latest/topics/asyncio.html) and [coroutines](https://docs.scrapy.org/en/latest/topics/coroutines.html) (checked 1 October 2026). - [Scrapy release notes](https://docs.scrapy.org/en/latest/news.html), for 2.13 (the asyncio reactor by default, `start()`), 2.16 and 2.17 (`FormRequest.from_response()` deprecated in favor of form2request), and 2.19.0, the current release (checked 1 October 2026). - [Scrapy: requests and responses](https://docs.scrapy.org/en/latest/topics/request-response.html), for form2request (checked 1 October 2026). - [Cloudflare Turnstile: client-side rendering](https://developers.cloudflare.com/turnstile/get-started/client-side-rendering/), [Cloudflare: detect a challenge response](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/detect-response/) and [Error 403](https://developers.cloudflare.com/support/troubleshooting/http-status-codes/4xx-client-error/error-403/) (checked 1 October 2026). ## Questions ### Can Scrapy solve Cloudflare Turnstile on its own? No. Scrapy does not run JavaScript, so the widget never produces a token. A spider can read the sitekey from the HTML, get a token from a solving API, and post it in the cf-turnstile-response field. ### Does calling a CAPTCHA API from a Scrapy callback block the crawl? Not if the callback is a coroutine and the API client is asynchronous. With Scrapy's asyncio reactor, await an httpx call and the other requests keep flowing while the token is prepared. ### Does a Turnstile token help with a Cloudflare 'Just a moment' page in Scrapy? No. That page is a Cloudflare challenge, not a Turnstile widget in a form. It is passed with a cf_clearance cookie, which is a different task.