Selenium can control a real browser, but collecting search-result URLs and visible text is usually a data-retrieval problem rather than a browser-testing problem. Launching Chrome, managing browser processes, waiting for rendered elements, and maintaining selectors add moving parts that may not contribute to the dataset you actually need.

A direct HTTP workflow is simpler: submit a complete Google Search URL to a search-results endpoint and receive structured JSON. This guide shows how to scrape Google Search results without Selenium while preserving result order, nested links, locale settings, pagination, and failure information.

Why browser automation adds unnecessary work

Selenium's WebDriver is designed to drive a browser as a user would. That is valuable when the browser experience itself is the object of the task—for example, testing an interactive flow or capturing a rendered state.

For structured search collection, a browser introduces additional state:

  • Browser and driver versions
  • Process and memory management
  • Page-load and element-wait logic
  • HTML selectors tied to presentation
  • Cookie and session state
  • Screenshot, rendering, and JavaScript behavior

If the required output is an ordered collection of result text and destination URLs, a JSON response is easier to store, validate, and analyze.

Make one direct HTTP request

Reserp accepts a complete https://www.google.com/search URL in a JSON request body. Store the API key in an environment variable and send it in the Authorization header:

curl --request POST 'https://api.reserp.ai/v1/serp' \
  --header "Authorization: Bearer $RESERP_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
    "url": "https://www.google.com/search?q=renewable+energy+storage&gl=gb&hl=en"
  }'

The Google URL keeps the search context explicit:

  • q contains the query
  • gl supplies country context
  • hl supplies interface-language context
  • start selects a later Google result offset when paginating

Keep the API key out of the Google URL. URLs are routinely stored in logs and research datasets; credentials belong in headers.

Understand the JSON response

A successful response has a compact public structure:

{
  "ok": true,
  "url": "https://www.google.com/search?q=renewable+energy+storage&gl=gb&hl=en",
  "finalUrl": "https://www.google.com/search?q=renewable+energy+storage&gl=gb&hl=en",
  "results": [
    {
      "text": "Example visible result text",
      "url": "https://example.org/energy-storage",
      "children": [
        {
          "text": "Related page",
          "url": "https://example.org/energy-storage/research"
        }
      ]
    }
  ],
  "pagination": {
    "start": 0,
    "nextStart": 10,
    "nextUrl": "https://www.google.com/search?q=renewable+energy+storage&gl=gb&hl=en&start=10"
  },
  "billed": true
}

Each visible result block can contain text, a destination url, and nested children. Fields can be absent when a block has no visible text or external URL, so consumers should treat them as optional.

The top-level array preserves returned visible blocks. It should not automatically be renamed “organic rankings,” because a modern result page can contain several kinds of linked content.

Scrape Google Search results with Node.js

Node.js can make the request with its built-in fetch implementation:

const endpoint = "https://api.reserp.ai/v1/serp";
const apiKey = process.env.RESERP_API_KEY;

if (!apiKey) {
  throw new Error("RESERP_API_KEY is missing");
}

function googleSearchUrl(
  query,
  { country = "us", language = "en", start = 0 } = {},
) {
  if (!Number.isInteger(start) || start < 0 || start % 10 !== 0) {
    throw new Error("start must be a non-negative multiple of 10");
  }

  const url = new URL("https://www.google.com/search");
  url.searchParams.set("q", query);
  url.searchParams.set("gl", country);
  url.searchParams.set("hl", language);
  if (start) url.searchParams.set("start", String(start));
  return url.toString();
}

async function fetchGoogleResults(url) {
  const response = await fetch(endpoint, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ url }),
    signal: AbortSignal.timeout(30_000),
  });

  const rawBody = await response.text();
  let data;

  try {
    data = JSON.parse(rawBody);
  } catch {
    throw new Error(`Search returned non-JSON HTTP ${response.status}`);
  }

  if (!response.ok || data.ok !== true) {
    const error = new Error(data.error ?? `HTTP ${response.status}`);
    error.retryable = data.retryable === true;
    error.billed = data.billed === true;
    throw error;
  }

  return data;
}

const page = await fetchGoogleResults(
  googleSearchUrl("renewable energy storage", {
    country: "gb",
    language: "en",
  }),
);

console.log(page.results);

Reading the body as text before parsing makes malformed or empty responses distinguishable from valid JSON. A usable success requires both an appropriate HTTP status and ok: true.

Walk nested result blocks

Looking only at results.map(result => result.url) can omit linked children. Traverse the result tree recursively:

function* walkResults(results = [], parentPath = []) {
  for (let index = 0; index < results.length; index += 1) {
    const result = results[index];
    const path = [...parentPath, index];
    yield { result, path };
    yield* walkResults(result.children ?? [], path);
  }
}

const rows = [...walkResults(page.results)]
  .filter(({ result }) => result.url)
  .map(({ result, path }) => ({
    path,
    url: result.url,
    text: result.text ?? "",
  }));

The path retains the parent-child position. Keep the original response as well as any flat table derived from it; flattening is convenient for analysis but should not destroy the source structure.

Paginate with the supplied next URL

Google pagination advances in organic-result offsets of ten. The number of visible blocks or recursively extracted URLs can be greater or smaller than ten.

Fetch another page with:

const secondPage = await fetchGoogleResults(page.pagination.nextUrl);

Do not calculate the next offset from page.results.length. Following pagination.nextUrl preserves the submitted query and locale parameters while advancing the correct offset.

Set a page or request budget before collection:

async function collectPages(firstUrl, maximumPages = 3) {
  const pages = [];
  const seenPageUrls = new Set();
  let nextUrl = firstUrl;

  while (
    pages.length < maximumPages &&
    !seenPageUrls.has(nextUrl)
  ) {
    seenPageUrls.add(nextUrl);
    const current = await fetchGoogleResults(nextUrl);
    pages.push(current);
    nextUrl = current.pagination.nextUrl;
  }

  return pages;
}

A hard limit prevents a collection loop from growing beyond its intended scope.

Normalize URLs after preserving them

The same destination can appear with a fragment or known analytics parameter. Store the original URL, then create a separate comparison value:

function comparisonUrl(rawUrl) {
  const url = new URL(rawUrl);
  url.hash = "";

  for (const key of [...url.searchParams.keys()]) {
    const lowerKey = key.toLowerCase();
    if (
      lowerKey.startsWith("utm_") ||
      ["gclid", "fbclid"].includes(lowerKey)
    ) {
      url.searchParams.delete(key);
    }
  }

  return url.toString();
}

Do not remove every query parameter. Parameters can identify genuinely different products, documents, languages, or application states.

Handle empty results and failures separately

A robust Google Search scraping workflow distinguishes:

  • A valid response containing result blocks
  • A valid response with an empty results array
  • An invalid request
  • Failed authentication
  • A rate limit
  • A retryable service failure
  • A malformed or non-JSON response

Documented error bodies expose ok: false, a stable error code, retryable, and billed. Retry only failures marked as retryable, cap the number of attempts, and use backoff with jitter. A failed request must not be stored as if Google returned zero results.

Record enough context to reproduce the collection

For every request, preserve:

  • Exact query text
  • Complete submitted Google Search URL
  • Country and language settings
  • Pagination offset
  • UTC collection timestamp
  • Request and retry number
  • Raw response
  • URL-normalization version used for derived rows

This produces an auditable search snapshot rather than an unexplained list of links.

Choose the method that matches the output

Use browser automation when you need to test browser interactions or study a rendered interface. Use a structured HTTP workflow when the intended output is search-result text, URLs, pagination state, and reproducible data.

That distinction is the practical answer to how to scrape Google Search results without Selenium: remove the browser from the client workflow and request a structured result representation directly.

For language-specific examples, read How to Get Google Search Results as JSON in Python or How to Query Google Search from Node.js and TypeScript. For multi-page collection, see How to Paginate Google Search Results Without Creating Duplicates. The Google Search API documentation contains the current request, response, pagination, and error contract.