Hi I am having a problem running typesense/docsearch-scraper in SPA, which has navigation using hash (#) in URLs. It doesn't index the pages and shows this in the console.

Ignored: from start url <https://url.before.hash/comes_here>

Any idea, how can I tell the scrapper to keep visiting pages with hash in url and indexing them? Thanks in advance.

May be this helps: https://discourse.algolia.com/t/docsearch-scraper-with-anchors/10811/2

Thanks for the link. If someone has auto-generated OpenAPI spec pages in their docs and their pages get ignored by the scraper, we can set the below settings in config.json. This will scrap the page after the javascript is rendered (SPA navigation problem) and treat urls with anchors as a separate page/document.

"strict_redirect": false,
"use_anchors": true,
"js_render": true,