#community-help

Approximate Time to Scrape Website Documentation

TLDR Jean-Baptiste asked about the average time for website documentation scraping. Jason shared that scraping the Typesense docs site takes around 15 minutes.

Powered by Struct AI

2

3
3mo
Solved
Join the chat
Jun 29, 2023 (3 months ago)
Jean-Baptiste
Photo of md5-01b4600434aa419becd17a9f7773e2ff
Jean-Baptiste
12:38 PM
also I wonder about a rough average of a common documentation website scrap's time (I know there can be 1 page to 1000 🙂 but tryin' to get a really vague idea)
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:39 PM
For scraping the Typesense docs site, it takes roughly 15 minutes IIRC, and generates ~10K search records

1

Jean-Baptiste
Photo of md5-01b4600434aa419becd17a9f7773e2ff
Jean-Baptiste
05:07 PM
Thank you 😊

1