#community-help

Dramatic Improvement with Typesense Indexing.

TLDR Tim shares their excitement over the greatly improved indexing speed they experienced with Typesense, and Jason asks about their use of faceted numerical fields and limitations with long strings and nested fields.

Powered by Struct AI

1

1

3
10mo
Solved
Join the chat
Feb 14, 2023 (10 months ago)
Tim
Photo of md5-7b45f90b96b0340298166e895a906163
Tim
03:24 PM
Hi, I don’t know what has changed between the last time I ran a full index and now (still on a later rc version of 0.24.0)… But, I just finished indexing my 9.6M documents — about 15GB in RAM — faster than ever (about 30 minutes, down from 90+ minutes).

I’m super happy about this improvement, so thanks to the Typesense team for whatever it was that happened (I know I also made some changes on my end, but there is no way my tweaks made that big of a difference here)!

1

Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:19 PM
Hmmm! That’s great to hear and quite a dramatic improvement! Do you have a lot of numerical fields that are faceted?
Tim
Photo of md5-7b45f90b96b0340298166e895a906163
Tim
05:57 PM
There are 4 numerical fields that are faceted (not really sure why to be honest). Probably the biggest limiting factor I have are fields that can occasionally have very long strings or a nested field that has a lot of values (I ended up cutting off at 100 but could probably go lower if the data is actually not needed) — this means I can only send batches of about 10k documents at a time to be indexed.

1