#community-help

Discussing Large Document Indexing in Word Files

TLDR robert asked about indexing large word files. Kishore Nallan advised splitting into smaller documents for improved performance.

17mo

Join the chat

Aug 03, 2022 (17 months ago)

Photo of md5-0ca37054c6c9042aa04fcfb92cc7d99c

robert

02:29 PM

Is there any documentation on large document indexing. Say word files of 4-5k words. Can i put the entire content of the word file into a field to be indexed? I would love to grab snippets of it via the instantsearch Snippet feature. What's the performance implication?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

02:31 PM

Recommended approach is to split into smaller documents and during query, do a group by on a parent ID that's common to all splits.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3005 threads (79% resolved)

Join Our Community

Similar Threads

Optimal Indexing and Querying of Large Documents

Robert asks about the best practice for indexing large documents and the ideal size of subdocuments. Jason suggests experimenting with 10K words in a single document and performance testing.

2mo

Discussing Maximum Words Per Field and Performance Impacts

robert sought advice on the performance impacts of large text fields in search response, and Kishore Nallan advised reading from disk would be slow. They also highlighted a limitation in Typesense.

17mo

Estimating RAM Requirements for Indexing Documents

Epi asked about index sizes in relation to document sizes and RAM requirements for their dataset. Kishore Nallan suggested indexing a sample and extrapolating results, and confirmed suitability for indexing large documents like Wikipedia articles in Typesense.

12mo