#community-help

Troubleshooting Create Index Issue on Typesense Instance

TLDR John was having issues creating new indices on a Typesense instance which Jason suggested could be due to old bugs or heavy search queries monopolizing CPU. After a restart, the issue was resolved. Jason also recommended upgrading the Typesense version.

Powered by Struct AI

1

13
1mo
Solved
Join the chat
Sep 02, 2023 (1 month ago)
John
Photo of md5-21545f1facb7836c149bc4c70752bd2b
John
04:52 PM
Hmm, seems like we can't create new indices on one of our Typesense instances anymore. It just times out, even with a small schema and a high timeout, whereas the same schema is instantly created on our other instances. There's nothing out of the ordinary about this instance, CPU and memory usage looks fine, logs look normal. Any idea what might be going on?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:56 PM
Could you DM me your cluster ID?
John
Photo of md5-21545f1facb7836c149bc4c70752bd2b
John
05:00 PM
Ah sorry for not being clear, we're running it ourselves
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:01 PM
Are the logs periodically outputting the latest sequence numbers?
05:02
Jason
05:02 PM
Also, in earlier versions of Typesense (before 0.25.0) there were some missing validation bugs that would cause bad data to get into the write queue and freeze it
05:03
Jason
05:03 PM
Could you post the last say 50 lines from the logs?
John
Photo of md5-21545f1facb7836c149bc4c70752bd2b
John
05:05 PM
Sure, here's a sample: https://pastebin.com/tPxQri3e
05:06
John
05:06 PM
> Also, in earlier versions of Typesense (before 0.25.0) there were some missing validation bugs that would cause bad data to get into the write queue and freeze it
Interesting, we've been running 0.25.0.rc25 for a while, maybe fixed after that?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:07 PM
Yeah... I would recommend upgrading to 0.25.1.rc6
05:08
Jason
05:08 PM
Although, this doesn't look like a write stall issue. There's potentially a set of heavy search queries that are monopolizing CPU... Notice how pending_queue_size is reducing very slowly
05:09
Jason
05:09 PM
Restarting would help flush that queue. We've also added safeguards to time-out stale search queries stuck in the queue for too long in recent builds
John
Photo of md5-21545f1facb7836c149bc4c70752bd2b
John
05:10 PM
CPU usage was at almost 0, but maybe something's still stuck. I restarted it now, let's see if that helps
05:15
John
05:15 PM
Now schema creation works again. Will look into upgrading ASAP and hope it doesn't happen again

Thanks for the help!

1