How to Paginate Google Search Results Without Creating Duplicates

Pagination looks straightforward until one search page contains more visible URLs than organic positions. A result page can include ordinary listings, nested sitelinks, news items, discussions, or other linked modules. Counting those URLs does not tell you Google's next page offset.

The safe rule is: follow the pagination URL returned with the response, and deduplicate destinations independently.

Why the result count is not an offset

Google pagination advances in organic-result increments:

• First page: start=0 or no start parameter
• Second page: start=10
• Third page: start=20
• Fourth page: start=30

A response containing 16 recursively visible URLs can still have pagination.nextStart equal to 10. Setting the next page to start=16 would mix two unrelated units.

Reserp responses therefore include:

{
  "pagination": {
    "start": 0,
    "nextStart": 10,
    "nextUrl": "https://www.google.com/search?q=example&start=10"
  }
}

Use nextUrl rather than constructing the next request from results.length.

A bounded page collector

The following JavaScript function accepts a Google Search URL and fetches a fixed maximum number of pages:

const endpoint = "https://api.reserp.ai/v1/serp";
const apiKey = process.env.RESERP_API_KEY;

async function fetchPage(url) {
  const response = await fetch(endpoint, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ url }),
    signal: AbortSignal.timeout(30_000),
  });

  const data = await response.json();
  if (!response.ok || data.ok !== true) {
    throw new Error(data.error ?? `HTTP ${response.status}`);
  }
  return data;
}

async function collectPages(firstUrl, maxPages = 3) {
  const pages = [];
  const seenPageUrls = new Set();
  let nextUrl = firstUrl;

  while (pages.length < maxPages && !seenPageUrls.has(nextUrl)) {
    seenPageUrls.add(nextUrl);
    const page = await fetchPage(nextUrl);
    pages.push(page);
    nextUrl = page.pagination.nextUrl;
  }

  return pages;
}

The hard page limit matters. A collection job should have a known request budget even if every response offers another URL.

Extract every destination without erasing structure

Use a recursive iterator for analysis:

function* walkResults(results = [], parentPath = []) {
  for (let index = 0; index < results.length; index += 1) {
    const result = results[index];
    const path = [...parentPath, index];
    yield { result, path };
    yield* walkResults(result.children ?? [], path);
  }
}

The path records where an item appeared in the original tree. Preserve it even if the next step creates one flat table.

Normalize conservatively

The same page can appear with a fragment or known tracking parameter. A comparison key can remove those obvious variants:

function comparisonUrl(rawUrl) {
  const url = new URL(rawUrl);
  url.hash = "";

  for (const key of [...url.searchParams.keys()]) {
    if (
      key.toLowerCase().startsWith("utm_") ||
      ["gclid", "fbclid"].includes(key.toLowerCase())
    ) {
      url.searchParams.delete(key);
    }
  }

  return url.toString();
}

Do not delete every query parameter. On many sites, parameters identify different products, documents, languages, or search states.

Deduplicate while preserving occurrences

Deleting repeated rows immediately loses information about where a destination appeared. Instead, keep every occurrence and add a stable comparison key:

function deriveRows(pages) {
  const firstOccurrence = new Map();
  const rows = [];

  pages.forEach((page, pageIndex) => {
    for (const { result, path } of walkResults(page.results)) {
      if (!result.url) continue;

      const normalizedUrl = comparisonUrl(result.url);
      const duplicateOf = firstOccurrence.get(normalizedUrl) ?? null;
      const rowId = `p${pageIndex + 1}:${path.join(".")}`;

      if (!duplicateOf) firstOccurrence.set(normalizedUrl, rowId);

      rows.push({
        rowId,
        page: pageIndex + 1,
        path,
        originalUrl: result.url,
        normalizedUrl,
        duplicateOf,
        text: result.text ?? "",
      });
    }
  });

  return rows;
}

This gives you two valid views:

• The occurrence table, which preserves the search observation
• The unique-URL view, which groups rows by normalizedUrl

Neither needs to replace the other.

Choose stopping conditions before collection

Useful stopping rules include:

• A fixed number of pages
• A fixed request budget
• A maximum elapsed time
• A documented threshold for pages adding no new comparison URLs
• A terminal error under a bounded retry policy

If “no new URLs” is used, require more than one page before stopping. Temporary duplication on an adjacent page does not prove that later pages contain nothing new.

Keep failures separate from empty pages

A timed-out request, invalid JSON body, and successful response with no qualifying URLs are different outcomes. Do not convert all three into an empty array and continue as if the page was observed.

Record each page request with:

• Submitted Google URL
• UTC collection time
• HTTP outcome
• Public error code, when present
• Retry number
• Whether the response contributed rows

Correct pagination is therefore two separate operations: follow Google's supplied offset, then decide how your application identifies duplicate destinations. The Google Search API documentation describes the current pagination object and valid request format.