# Go Colly and Cloudflare Turnstile: A Scraper Tutorial

> A Go Colly collector that finds a Cloudflare Turnstile sitekey with OnHTML, gets a token from a solving API over net/http, and posts the form.

- Source: https://zerocaptcha.io/blog/go-colly-cloudflare-turnstile
- Published: 2026-09-30
- Updated: 2026-10-01
- Author: ZeroCaptcha Engineering

Colly does not execute JavaScript, so it never sees a Cloudflare Turnstile token, but a Colly
scraper can still submit a Turnstile-protected form: read the widget's `data-sitekey` in an
`OnHTML` callback, ask a solving API for a token over plain `net/http`, and send it as
`cf-turnstile-response` with `Collector.Post`. This tutorial builds that collector in one file,
with no dependencies beyond Colly.

Scrape only sites you are allowed to. See
[responsible captcha automation](https://zerocaptcha.io/guides/responsible-captcha-automation).

## Set up

```sh
mkdir search-scraper && cd search-scraper
go mod init example.com/search-scraper
go get github.com/gocolly/colly/v2
export ZEROCAPTCHA_API=…   # the API's base URL
export ZEROCAPTCHA_KEY=…   # your API key
```

## The collector

Save this as `main.go`:

```go
package main

import (
	"bytes"
	"crypto/rand"
	"encoding/json"
	"errors"
	"fmt"
	"log"
	"net/http"
	"os"
	"time"

	"github.com/gocolly/colly/v2"
)

var (
	apiURL = os.Getenv("ZEROCAPTCHA_API")
	apiKey = os.Getenv("ZEROCAPTCHA_KEY")
	api    = &http.Client{Timeout: 15 * time.Second}
)

type reply struct {
	ErrorID   int    `json:"errorId"`
	ErrorCode string `json:"errorCode"`
	TaskID    string `json:"taskId"`
	Status    string `json:"status"`
	Solution  struct {
		Token string `json:"token"`
	} `json:"solution"`
}

func call(method string, body map[string]any, idempotencyKey string) (reply, error) {
	body["clientKey"] = apiKey
	payload, err := json.Marshal(body)
	if err != nil {
		return reply{}, err
	}
	req, err := http.NewRequest(http.MethodPost, apiURL+"/"+method, bytes.NewReader(payload))
	if err != nil {
		return reply{}, err
	}
	req.Header.Set("Content-Type", "application/json")
	if idempotencyKey != "" {
		req.Header.Set("Idempotency-Key", idempotencyKey)
	}
	resp, err := api.Do(req)
	if err != nil {
		return reply{}, err
	}
	defer resp.Body.Close()
	var r reply
	if err := json.NewDecoder(resp.Body).Decode(&r); err != nil {
		return reply{}, fmt.Errorf("%s: %w", method, err)
	}
	if r.ErrorID != 0 {
		return r, fmt.Errorf("%s: %s", method, r.ErrorCode)
	}
	return r, nil
}

func solveTurnstile(pageURL, sitekey, action, cdata string) (string, error) {
	// The widget's data-action and data-cdata go in metadata, only when it sets them: many sites
	// check both when they verify the token.
	metadata := map[string]string{}
	if action != "" {
		metadata["action"] = action
	}
	if cdata != "" {
		metadata["cdata"] = cdata
	}
	task := map[string]any{"type": "TurnstileTaskProxyless", "websiteURL": pageURL, "websiteKey": sitekey, "metadata": metadata}
	// One Idempotency-Key per task: a retried create with it returns the same task.
	created, err := call("createTask", map[string]any{"task": task}, rand.Text())
	if err != nil {
		return "", err
	}
	deadline := time.Now().Add(180 * time.Second)
	for time.Now().Before(deadline) {
		time.Sleep(2 * time.Second)
		result, err := call("getTaskResult", map[string]any{"taskId": created.TaskID}, "")
		if err != nil {
			return "", err
		}
		if result.Status == "ready" {
			return result.Solution.Token, nil
		}
	}
	return "", errors.New("no token within 180 seconds")
}

func main() {
	c := colly.NewCollector(colly.AllowedDomains("shop.example.com"), colly.Async(true))
	if err := c.Limit(&colly.LimitRule{DomainGlob: "*", Parallelism: 4}); err != nil {
		log.Fatal(err)
	}

	c.OnHTML("form#search", func(e *colly.HTMLElement) {
		sitekey := e.ChildAttr("[data-sitekey]", "data-sitekey")
		if sitekey == "" {
			log.Printf("no Cloudflare Turnstile widget on %s", e.Request.URL)
			return
		}
		token, err := solveTurnstile(e.Request.URL.String(), sitekey,
			e.ChildAttr("[data-sitekey]", "data-action"), e.ChildAttr("[data-sitekey]", "data-cdata"))
		if err != nil {
			log.Print(err)
			return
		}
		target := e.Request.AbsoluteURL(e.Attr("action"))
		if err := c.Post(target, map[string]string{"q": "running shoes", "cf-turnstile-response": token}); err != nil {
			log.Print(err)
		}
	})

	c.OnHTML(".result a[href]", func(e *colly.HTMLElement) {
		fmt.Println(e.Text, e.Request.AbsoluteURL(e.Attr("href")))
	})

	if err := c.Visit("https://shop.example.com/search"); err != nil {
		log.Fatal(err)
	}
	c.Wait()
}
```

Run it with `go run .`.

## How it works

- **`OnHTML("form#search", …)`** fires once for the form. `ChildAttr` reads the sitekey and the
  optional action from the widget's `div` inside it, the attributes Cloudflare's implicit
  rendering uses.
- **`solveTurnstile`** is a plain `net/http` client for the compatible `createTask` and
  `getTaskResult` calls. It checks `errorId` on every reply, because the format answers HTTP 200
  even when a call fails, and it stops after three minutes instead of waiting forever.
- **`c.Post`** sends the form URL-encoded, like a browser. Colly keeps cookies between requests
  of the same collector, so the submission carries the session the form page set.
- **The second `OnHTML`** runs on the results page that the post returns.

- **`c.Wait()`** blocks until every request, the post and its results page included, is done.

## Going parallel

With `colly.Async(true)` the collector fetches several pages at once, and each callback runs in its
own goroutine, so one callback waiting on a token does not hold up the others. The `LimitRule`
caps the collector at four requests at a time, so the post for a solved form goes out at once
rather than behind a long queue. To crawl many forms, call `c.Visit` for each start page before
`c.Wait()`; raise `Parallelism` only as far as the site allows.

A Cloudflare Turnstile token is valid for 300 seconds and for one use: solve in the callback that
posts, as above, and never share a token between two posts. See
[Cloudflare Turnstile token expiry](https://zerocaptcha.io/guides/cloudflare-turnstile-token-expiry).

## Errors you may see

- **`createTask: ERROR_ZERO_BALANCE`**: add funds; nothing was held.
- **`getTaskResult: ERROR_CAPTCHA_UNSOLVABLE`**: the task failed and costs nothing. Check the page
  URL and sitekey.
- **The post returns the form again with an error**: the site rejected the token or another
  field. Solve again before retrying; the token has been spent.
- **Every page answers HTTP 403 with "Just a moment…"**: that is a Cloudflare challenge page, not
  a Turnstile form. See [Cloudflare challenge page vs Cloudflare Turnstile](https://zerocaptcha.io/guides/cloudflare-challenge-vs-turnstile).

The [Go page of the Cloudflare Turnstile solver](https://zerocaptcha.io/cloudflare-turnstile-solver/go) shows the tested net/http program and the coming Go SDK, which add
retries and idempotency keys to the same calls. Every error code is in the
[errors reference](https://zerocaptcha.io/docs/reference/errors).

## Sources

- [Colly documentation](https://go-colly.org/docs/) and the
  [colly v2 package reference](https://pkg.go.dev/github.com/gocolly/colly/v2) (checked 1
  October 2026).
- [Cloudflare Turnstile: client-side rendering](https://developers.cloudflare.com/turnstile/get-started/client-side-rendering/)
  (checked 1 October 2026).

## Questions

### Can Colly solve Cloudflare Turnstile?

Not by itself: Colly does not run JavaScript, so the widget never produces a token. Colly can read the sitekey from the HTML, and a solving API returns a token to post with the form.

### Does the Turnstile token go in a cookie or in the form?

In the form, as the field cf-turnstile-response. Colly's Post sends it with the other fields, and Colly's cookie handling keeps the session the form page set.

### How do I keep an async Colly collector from wasting tokens?

Solve inside the callback that submits the form, and limit parallelism with a LimitRule so submissions are not queued behind other requests while the token's 300 seconds run out.