Hey everyone. I'm using typesense-docsearch-scrape...
# community-help
b
Hey everyone. I'm using typesense-docsearch-scraper on a large website and half of the pages need
js_render=true
. The scraper takes about 1h to complete, mainly because of
js_render
. Are there any tips on how to make this faster? Ideally I would only enable
js_render
on specific paths but I don't think that's possible right now. Thank you for any suggestions! Should I run two scrapers in parallel with 2 different indexes? I haven't explored this and not sure what limitations I might get especially because I use faceting.
f
Hey Bogdan, By faceting, are you referring to the website being scraped or while using search?
b
yes, I have a few field_definitions set up as
facet=true
in the scraper config and later in the frontend I use
algoliaHelper.addFacetRefinement(fieldName, value)
. I haven't investigated yet if it is possible to still use faceting this way when data is spread over multiple indexes. But I might be getting ahead of myself, if there is a way to make the scraper run faster without using multiple indexes whilst
js_render=true
then happy to try something simpler
f
Running separate instances of the docscraper with and without using
js_render
should be an option. If you face any issues, feel free to reply on this thread so we can take a look. Ideally, if issues arise, we would advise sending us a DM so we can test the scraper against the website.
b
Hey @Fanis Tharropoulos thanks for taking the time to reply. If I run multiple instances of the scraper then that means I would have to create 2 separate indexes, is that correct?
f
Yes. You would afterwards merge them manually. We could introduce a new config parameter for a per-URL
js_render
as well, though.
b
I will try your suggestion, I think it should work. And yes, I think if it would be possible to configure
js_render
per each
start_url
that would work great and simplify things quite a bit. Thanks for the help @Fanis Tharropoulos
f
Can you open up a feature request on the repo? We want to see if other users also would benefit from this feature
b