#community-help

Fixing Typesense/Docsearch-Scraper Issue in SPA with Hash Navigation

TLDR Nargiz encountered an issue with typesense/docsearch-scraper in SPA using hash navigation. Jason shared a helpful link, and Nargiz provided a solution using specific config settings.

Powered by Struct AI

1

3
7mo
Solved
Join the chat
Mar 07, 2023 (7 months ago)
Nargiz
Photo of md5-838ca51cf399939b4c0dd325708e1ada
Nargiz
02:46 PM
Hi
I am having a problem running typesense/docsearch-scraper in SPA, which has navigation using hash (#) in URLs. It doesn't index the pages and shows this in the console.
Ignored: from start url <https://url.before.hash/comes_here>

Any idea, how can I tell the scrapper to keep visiting pages with hash in url and indexing them?

Thanks in advance.
Mar 08, 2023 (7 months ago)
Nargiz
Photo of md5-838ca51cf399939b4c0dd325708e1ada
Nargiz
03:46 PM
Thanks for the link.
If someone has auto-generated OpenAPI spec pages in their docs and their pages get ignored by the scraper, we can set the below settings in config.json. This will scrap the page after the javascript is rendered (SPA navigation problem) and treat urls with anchors as a separate page/document.

"strict_redirect": false,
"use_anchors": true,
"js_render": true,

1