Discussing Full-Text Indexing of a Website
TLDR Mac asked for advice on crawling a website with varying structures. Jason suggested using a SaaS service.
May 12, 2022 (18 months ago)
Mac
02:08 AMI am running a cluster for our team, my collections are all well designed on the schema, but I have a new team who want to full text index a website that has had many developers working on it over time.
So as you can imagine the structure and naming styles are a bit of a mixed bag and we have no set crawler tags.
What is the best way to crawl this?
Jason
02:17 AMMac
03:06 AM
Typesense
Indexed 2779 threads (79% resolved)
Similar Threads
Discussing Full-text Indexer
Joe was clarifying what a full-text indexer is. Jason confirmed the explanation.
Discussing Search Functionality in Custom Blogging Platform
Martin discussed his blogging platform's search functionality and its use of Typesense for full text search. Jason provided feedback and suggested a hybrid solution for the search bar. Improvements will be made based on further user feedback.
Discussing Document Indexing Speeds and Typesense Features
Thomas asks about the speed of indexing and associated factors. The conversation reveals that larger batch sizes and NVMe disk usage can improve speed, but the index size is limited by RAM. Jason shares plans on supporting nested fields, and they explore a solution for products in multiple categories and catalogs.