Full Text Search Across Long Books: Chunking vs. Single Document

TLDR Epi asked about providing FTS for long books. Kishore Nallan suggested breaking the books into chunks for better performance and query results.

Powered by Struct AI
Join the chat
Dec 03, 2022 (9 months ago)
Photo of md5-949f018f94fc2432eeec74b4c51e0aaa
08:03 PM
If I'd like to provide FTS across a collection of long books, is it necessary to break each book into chunks for performance, or can a single document be the whole book itself?
Dec 04, 2022 (9 months ago)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:48 AM
Definitely you need to break it into chunks because that's when you can provide multiple relevant highlights for a given query. You can always use group by to group the results meaningfully at a per chapter or page level if needed.