#community-help

Dramatic Improvement with Typesense Indexing.

TLDR Tim shares their excitement over the greatly improved indexing speed they experienced with Typesense, and Jason asks about their use of faceted numerical fields and limitations with long strings and nested fields.

8mo

Join the chat

Feb 14, 2023 (8 months ago)

Photo of md5-7b45f90b96b0340298166e895a906163

Tim

03:24 PM

Hi, I don’t know what has changed between the last time I ran a full index and now (still on a later rc version of 0.24.0)… But, I just finished indexing my 9.6M documents — about 15GB in RAM — faster than ever (about 30 minutes, down from 90+ minutes).

I’m super happy about this improvement, so thanks to the Typesense team for whatever it was that happened (I know I also made some changes on my end, but there is no way my tweaks made that big of a difference here)!

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

04:19 PM

Hmmm! That’s great to hear and quite a dramatic improvement! Do you have a lot of numerical fields that are faceted?

Tim

05:57 PM

There are 4 numerical fields that are faceted (not really sure why to be honest). Probably the biggest limiting factor I have are fields that can occasionally have very long strings or a nested field that has a lot of values (I ended up cutting off at 100 but could probably go lower if the data is actually not needed) — this means I can only send batches of about 10k documents at a time to be indexed.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Performance Issues with Typesense Faceting and Nested Fields

Stefan experiences performance issues when faceting against 23 fields in Typesense, especially with nested fields. Kishore Nallan is working on a solution and will follow up on the thread.

7mo

Improving Typesense Query Performance

Jonathan queried about slower than expected typesense query performance. Jason and Kishore Nallan offered solutions and explanations. After a series of tests, Jonathan found other queries returned results quickly, indicating the issue was specific to the original query.

11mo

Discussing Document Indexing Speeds and Typesense Features

Thomas asks about the speed of indexing and associated factors. The conversation reveals that larger batch sizes and NVMe disk usage can improve speed, but the index size is limited by RAM. Jason shares plans on supporting nested fields, and they explore a solution for products in multiple categories and catalogs.

20mo