#community-help

Fixing Typesense/Docsearch-Scraper Issue in SPA with Hash Navigation

TLDR Nargiz encountered an issue with typesense/docsearch-scraper in SPA using hash navigation. Jason shared a helpful link, and Nargiz provided a solution using specific config settings.

7mo

Join the chat

Mar 07, 2023 (7 months ago)

Photo of md5-838ca51cf399939b4c0dd325708e1ada

Nargiz

02:46 PM

Hi
I am having a problem running typesense/docsearch-scraper in SPA, which has navigation using hash (#) in URLs. It doesn't index the pages and shows this in the console.
Ignored: from start url <https://url.before.hash/comes_here>

Any idea, how can I tell the scrapper to keep visiting pages with hash in url and indexing them?

Thanks in advance.

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

04:52 PM

May be this helps: https://discourse.algolia.com/t/docsearch-scraper-with-anchors/10811/2

Mar 08, 2023 (7 months ago)

Nargiz

03:46 PM

Thanks for the link.
If someone has auto-generated OpenAPI spec pages in their docs and their pages get ignored by the scraper, we can set the below settings in config.json. This will scrap the page after the javascript is rendered (SPA navigation problem) and treat urls with anchors as a separate page/document.

"strict_redirect": false,
"use_anchors": true,
"js_render": true,

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2786 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Docsearch Scrapper Metadata Configuration and Filter Problem

Marcos faced issues with Docsearch scrapper not adding metadata attributes and filtering out documents without content. Jason helped fix the issue by updating the scraper and providing filtering instructions.

6mo

Configuring Docusaurus and Typesense for a Documentation Site

Apoorv had trouble adding search functionality to a Docusaurus documentation website with Typesense. Jason worked through several troubleshooting steps, identified issues with Apoorv's setup, and ultimately provided solutions that successfully implemented the search bar function.

27mo

Solving Typesense Docsearch Scraper Issues

Sandeep was having issues with Typesense's docsearch scraper and getting fewer results than with Algolia's scraper. Jason helped by sharing the query they use and advised checking the running version of the scraper. The issue was resolved when Sandeep ran the non-base regular docker image.

22mo