How to track keyword rankings without scraping Google

Why checking Google yourself doesn't work

Google personalises results by location, device, language and your own search history. The position you see in your browser is rarely the one your customers see, and it changes from one search to the next. A rank tracker checks every keyword the same way each time, from the location and device you choose, so a change in the numbers means something changed on Google.

Why scraping Google is a dead end

The obvious shortcut is a script that loads Google's result pages and reads the positions. It breaks quickly:

• Google's terms of service forbid automated queries, and Google actively blocks them with CAPTCHAs and IP bans.
• You end up managing proxies, rotating user agents and fixing the parser every time the result page changes.
• Your own server's IP can get flagged, which can affect other things you run on it.

The reliable way: SERP APIs

SERP data providers such as Serper, SerpApi, DataForSEO and ValueSERP run this infrastructure as their business. You send a keyword with a country, language, device and optional city; they return the full result page as structured data: organic positions, URLs, titles and the SERP features on the page.

What to look for in a provider:

• Location targeting down to the city, if you do local SEO.
• Desktop and mobile results.
• Result depth: the top 10 is cheaper, the top 100 shows keywords that are still climbing.
• Price per search at your volume. Costs differ a lot between providers and plans, and most give free credits to test.

Pair it with Google Search Console, which is free and official: it shows the clicks and impressions your pages actually received. Rankings tell you where you are; Search Console tells you what it's worth.

Pick the right keywords

Start with 20 to 50 phrases that matter for the business:

• Money keywords: the products and services you sell ("trail running shoes", "dentist in leeds").
• Problem keywords: what people search before they know your product ("how to choose running shoes").
• Brand keywords: your name, so you notice when someone else outranks you for it.

Tag them (product, blog, local) so you can see which part of the site is moving, and add a target URL so you notice when Google ranks the wrong page.

Choose country, city and device

If you sell nationally, track the country. If you serve a city, track that city: local results, with their map pack, differ a lot from national ones. Track mobile as well as desktop for important keywords, because Google ranks the two separately.

How often to check

Daily checks catch problems within a day and show the effect of a change quickly; weekly checks are enough for stable, long-term keywords and cost less. A good setup checks money keywords daily and the long tail weekly.

Read trends, not noise

Positions move a little every day. Look at trends over weeks:

• Visibility sums up all keywords into one number weighted by expected clicks, so a move from 11 to 4 counts far more than a move from 48 to 41.
• Best position shows how high a page has ever been.
• SERP features (AI overviews, featured snippets, local packs, videos) explain why a top-3 position may still get few clicks.
• Competitors show whether you dropped or someone else climbed.

Act on what you see

A keyword that drops sharply usually means a technical problem (a page returning an error, a stray noindex, a broken redirect) or a stronger competitor. Run a site audit, compare the ranking URL with your target page, and look at who moved above you. Set alerts for top 3 and top 10 entries and exits, so the important moves reach you without opening a dashboard.