advanced-keyword-scraper

Advanced Keyword Scraper v1.9.11

v1.9.11 fixes

• Research mode now reserves Google request budget for SERP/PAA enrichment instead of allowing keyword discovery to consume the full budget first.
• Topic clustering now uses TF-IDF + cosine similarity with representative phrase labels instead of first-token labels.
• Opportunity scoring now has better differentiation from lexical specificity, intent, SERP exact-title prevalence and optional provider metrics.
• Flipkart/Meesho generic product queries are classified as Product Discovery rather than automatically as Commercial Research.
• CSV workflow exports use standard UTF-8 without a BOM-prefixed header artifact.

Security & stability hardening (introduced in v1.9.8)

• SSRF protection and redirect validation for SEO audits.
• 2 MB HTML response cap and HTML content-type validation.
• API key authentication, local-only fallback, and in-process rate limiting.
• Bounded TF-IDF similarity/clustering to prevent O(n²) resource exhaustion.
• Project-aware keyword persistence with duplicate protection on fresh databases.
• Improved intent fallback (unknown instead of assuming navigational).
• robots.txt and sitemap.xml checks in SEO audits.

Multi-source SEO / Keyword Intelligence

Version 1.9.11 includes the hardened multi-source workflow: Google, YouTube Creator suggestions, Bing and Amazon autocomplete can be queried individually or together. Google/YouTube traffic remains behind the safety gateway; Bing/Amazon use their public suggestion endpoints and do not bypass access controls.

Multi-source features

• --engine all for Google + YouTube + Bing + Amazon in one keyword run.
• --engines google,youtube,bing,amazon for explicit source selection.
• Source-aware deduplication: the report keeps a combined Source field when the same keyword appears across providers.
• Expanded modifier discovery: geo/local, comparison, content and buyer-intent patterns with --include-modifiers.
• --min-words / --max-words keyword-length filtering.
• --content-brief adds recommended format, title angle and content angle.
• Research workbook now includes Source Coverage.
• More tolerant parsing for Bing OSJSON and Amazon completion payloads.
• Version metadata, CLI output, changelog, and package documentation are synchronized to 1.9.11.

Example:

keyword-scraper "cotton kurti" --engine all --depth 2 --include-modifiers --content-brief --format excel

For only the YouTube creator workflow:

keyword-scraper "cotton kurti" --engine youtube --depth 2

Google SERP safety/persistence upgrade with conservative global pacing, persistent SQLite SERP cache, duplicate suppression, progressive 429 cooldown, CAPTCHA/unusual-traffic stop, Google circuit breaker, per-run request budget, and console telemetry.

This safety layer is designed to reduce request volume and stop on provider abuse signals; it is not an anti-bot bypass mechanism.

Example:

keyword-scraper "cotton kurti" --mode serp --threads 2 --google-min-interval 3 --google-jitter 1.5 --google-max-requests 100

SERP cache defaults to <output>/google_serp_cache.sqlite3 and is valid for 72 hours.

Historical multi-source hardening (v1.9.6)

• Multi-source aggregation preserves Google, YouTube, Bing, and Amazon provenance for duplicate keywords.
• Source Count and per-provider Found flags are included in keyword outputs.
• Research Opportunity Score uses the observed source count instead of a fixed value.
• Depth-2 expansion preserves provider lineage.
• YouTube Creator and Amazon buyer-intent intelligence are included when their modes are selected.
• Non-Google retry routing is provider-aware: Bing/Amazon requests use their own provider identity when selecting retry/throttling delays. Failed Bing/Amazon responses are not cached as empty results.
• Regression coverage includes multi-provider integration, depth-2 lineage, provider-specific retry routing, export schema, and version consistency.

Historical baseline: v1.7.0

A local-first keyword research CLI with multi-source discovery, heuristic intelligence, optional SERP analysis, resilient HTTP handling, and export workflows.

Requirements

• Python 3.10 or newer
• Windows, macOS, or Linux

Install

Recommended on Windows:

py -m pip install --upgrade pip
py -m pip install -e .

If py is unavailable, use python instead.

Verify the package

python -m unittest discover -s tests -t . -v

The repository includes a tests/__init__.py bootstrap so tests also work directly from this source-layout checkout before an editable install.

Run keyword research

keyword-scraper "saree" "kurti" --depth 2 --format all

Or without installing the console script:

python -m keyword_scraper.cli "saree" "kurti" --depth 2 --format all

Run SERP research

keyword-scraper "saree" --mode serp --format excel

Run full keyword intelligence research

The research mode combines the existing multi-source discovery engine with optional SERP/PAA enrichment, related searches, local-first keyword clustering, transparent opportunity scoring, and optional provider metrics.

keyword-scraper "cotton kurti" --mode research --depth 2 --serp-limit 20 --format excel

Optional competitor-gap analysis:

keyword-scraper "cotton kurti" --mode research --serp-limit 30 ^
  --target-domain example.com --competitor-domain competitor.com

Optional metrics hydration (no metrics are invented):

keyword-scraper "cotton kurti" --mode research --metrics-file metrics.csv

The metrics CSV can contain:

keyword,search_volume,kd,cpc,competition,competition_level
cotton kurti,12000,28,0.42,0.31,LOW

Research report sheets

The Excel report can include:

• All Keywords
• Opportunity Shortlist
• KPI Summary
• Keyword Clusters
• PAA Questions
• Related Searches
• SERP Analysis
• Competitor Gap
• Intent Summary / n-grams / topic summaries

Opportunity Score is a transparent heuristic. Search volume, KD and CPC are only used when supplied by a provider metrics CSV; the package never fabricates those values.

Workflow modes

These modes create local action-oriented CSV plans from the supplied keywords; they do not claim exact search volume or guaranteed rankings.

keyword-scraper "budget smartphone" --mode youtube --mode-output youtube.csv
keyword-scraper "wireless earbuds" --mode ecommerce --mode-output ecommerce.csv
keyword-scraper "best accounting software" --mode seo --mode-output seo.csv
keyword-scraper "project management software" --mode copywriting --mode-output copywriting.csv

Notes

• SERP providers may change markup, throttle requests, or restrict automated access.
• Opportunity scores are heuristics, not search-engine ranking scores.
• SERP/PAA extraction depends on changing Google markup and may return partial results.
• Clustering is deterministic lexical/intent clustering, not a claim of semantic ML understanding.
• The project does not attempt to bypass provider controls.
• The project does not attempt to bypass provider controls.

Google gateway hardening (v1.9.2)

• Exactly one half-open circuit probe is admitted after breaker cooldown; concurrent probes are blocked.
• Google transport uses a stable per-run User-Agent and does not rotate proxies.
• Proxy rotation, if configured for other providers, is not passed into the Google gateway.
• CAPTCHA/429 remains a hard-stop/cooldown signal; the gateway never changes proxies or User-Agent to retry around it.

Multi-source intelligence

Keyword output now preserves provider provenance. Duplicate keywords discovered by multiple sources are merged into one row with Source Count, Google Found, YouTube Found, Bing Found, Amazon Found, and Cross-Source Signal fields. Research opportunity scoring uses the actual source count when available.

YouTube workflow output includes creator opportunity and search-intent fields. Ecommerce workflow output includes buyer intent and product-attribute angles.

AI-free SEO Intelligence Stack

v1.9.11 includes an optional-but-packaged intelligence/API layer with no external AI provider:

• scikit-learn — TF-IDF, cosine similarity and deterministic clustering
• RapidFuzz — fuzzy/near-duplicate keyword matching
• httpx — modern HTTP client for SEO audits and integrations
• BeautifulSoup4 + lxml — fast HTML parsing and on-page SEO analysis
• pandas — keyword analytics and report dataframes
• FastAPI + Pydantic — typed REST API
• SQLAlchemy — persistent projects/keyword storage
• diskcache — local result caching
• Uvicorn — local API server

Start the API

keyword-scraper-api

or:

python -m keyword_scraper.api

Default: http://127.0.0.1:8000

Useful endpoints:

• GET /health
• POST /keywords/analyze
• POST /keywords/clusters
• POST /keywords/similarity
• POST /keywords/matches
• POST /keywords/score
• POST /seo/audit
• POST /projects
• GET /keywords/recent

Swagger documentation is available at /docs when the API server is running.

The intelligence layer is deliberately algorithm-first: no OpenAI/Gemini/Claude API key is required.

Marketplace Seller Layers (v1.9.11)

Two local-first marketplace planning layers are included:

keyword-scraper "cotton nighty" --mode flipkart --mode-output flipkart.csv
keyword-scraper "cotton kurti" --mode meesho --mode-output meesho.csv

These layers generate buyer intent, listing priority, title angle, product attributes, bullet angle, catalog action and platform-specific research notes. They are research/planning layers, not official Flipkart/Meesho search-volume APIs and do not bypass authentication, CAPTCHAs or access controls.