Fixing Typesense/Docsearch-Scraper Issue in SPA with Hash Navigation
TLDR Nargiz encountered an issue with typesense/docsearch-scraper in SPA using hash navigation. Jason shared a helpful link, and Nargiz provided a solution using specific config settings.
1
Mar 07, 2023 (7 months ago)
Nargiz
02:46 PMI am having a problem running typesense/docsearch-scraper in SPA, which has navigation using hash (#) in URLs. It doesn't index the pages and shows this in the console.
Ignored: from start url <https://url.before.hash/comes_here>
Any idea, how can I tell the scrapper to keep visiting pages with hash in url and indexing them?
Thanks in advance.
Jason
04:52 PMMar 08, 2023 (7 months ago)
Nargiz
03:46 PMIf someone has auto-generated OpenAPI spec pages in their docs and their pages get ignored by the scraper, we can set the below settings in config.json. This will scrap the page after the javascript is rendered (SPA navigation problem) and treat urls with anchors as a separate page/document.
"strict_redirect": false,
"use_anchors": true,
"js_render": true,
1
Typesense
Indexed 2786 threads (79% resolved)
Similar Threads
Docsearch Scrapper Metadata Configuration and Filter Problem
Marcos faced issues with Docsearch scrapper not adding metadata attributes and filtering out documents without content. Jason helped fix the issue by updating the scraper and providing filtering instructions.
Configuring Docusaurus and Typesense for a Documentation Site
Apoorv had trouble adding search functionality to a Docusaurus documentation website with Typesense. Jason worked through several troubleshooting steps, identified issues with Apoorv's setup, and ultimately provided solutions that successfully implemented the search bar function.
Solving Typesense Docsearch Scraper Issues
Sandeep was having issues with Typesense's docsearch scraper and getting fewer results than with Algolia's scraper. Jason helped by sharing the query they use and advised checking the running version of the scraper. The issue was resolved when Sandeep ran the non-base regular docker image.