typesense

I am trying to create an index locally with DocSearchScraper and distribute it to different systems. As I understand the scraper, a collection is created first, which is given to the Typsense service at the end. Is there a way to interrupt at this point and create a JSON file with the index that can be imported later on other machines or is it necessary to do an export at the end of the indexing.

A collection is first created in Typesense, and then as the scraping happens, each page is imported into Typesense as documents

Here’s the file that manages all communication with Typesense: <https://github.com/typesense/typesense-docsearch-scraper/blob/master/scraper/src/typesense_helper.py>

You could edit this as needed to write to a local JSONL file

Or if you’re also using Typesense for search, you could let the scraper do it’s thing and the use the documents/export endpoint to export the JSONL file from Typesense: <https://typesense.org/docs/0.24.0/api/documents.html#export-documents>

Thank you, Jason, i think the export option will be my solution