#community-help

Preparing Data for Indexing in TypeSense

TLDR Charles sought advice for preparing diverse data for TypeSense indexing. Jason suggested creating documents for similar fields and extracting text from PDF into paragraphs.

18mo

Join the chat

Jun 21, 2022 (18 months ago)

Photo of md5-3a8311fe91486b1c95f9ab3c7ca33f1c

Charles

03:53 PM

Hello everyone, I am just starting with TypeSense I was wondering how I should prepare the data for the indexing, I have many with different layout... How would you proceed? thanks in advance

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

03:56 PM

In general, you'd typically want to put all documents with the same fields and attribute types in one collection. Similar to what you'd do in a relational database. A table is roughly equivalent to a collection

Charles

03:56 PM

Let's imagine I have books in PDF, how would you proceed?

Jason

03:57 PM

You want to extract the text out of the PDF into paragraphs (there are libraries that do this for you), then create one document per paragraph in Typesense to get the most relevance

Charles

03:59 PM

let's imagine this as an example:

03:59

Charles

03:59 PM

03:59

Charles

03:59 PM

and many pages like this

04:00

Charles

04:00 PM

you would create blocks (heading+paragraphs) together

Jason

04:00 PM

Yup

Charles

04:00 PM

then here we have several paragraphes, would you ut the same heading for several paragraphes?

Jason

04:01 PM

If you need to show the paragraph heading next to each search result, yes.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3011 threads (79% resolved)

Join Our Community

Similar Threads

Troubleshooting Typesense Document Import Error

Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.

10mo

Using Typesense to Index Large Amounts of Data

Rafael wants to use Typesense to index 100M documents currently in MongoDBAtlas. Jason affirmed Typesense can handle it and asked for more details.

17mo

Deleting and Creating Documents in Typesense

Priyank asked for help with deleting and creating documents in Typesense and found the problem to be an issue with their own code. Jason offered support.

10mo

Improving Speed of Document Batch Writing in Typesense

Narayan asked for advice on speeding up document batch writing. Jason recommended a resource and suggested a 4vCPU minimum to increase performance.

Nested Objects and Arrays in Typesense

Robert seeks advice on managing nested objects in Typesense. Kishore Nallan informs of upcoming support for nested objects and provides a current workaround. Robert indicates interest in tracking this feature. Kishore Nallan provides a link to follow the issue.

18mo